Yapay Zeka · 8 dk okuma · 3 Mayıs 2026
Mixed Precision Training Stabilizes Neural ODEs
Researchers demonstrate a framework that reduces memory use by 50% and speeds up neural ODE training 2x by carefully mixing low and high precision arithmetic.
Kaynak: arxiv/cs.LG · Elena Celledoni, Brynjulf Owren, Lars Ruthotto, Tianjiao Nicole Yang · orijinali aç ↗ ↗
Mixed precision training for neural ODEs reduces memory and computation while maintaining accuracy through selective precision management.
- — Standard mixed precision fails for neural ODEs due to accumulated roundoff errors in iterative solvers.
- — Framework uses low precision for network velocity evaluation and intermediate state storage.
- — High precision accumulation for gradients and solutions prevents numerical instability.
- — Custom dynamic adjoint scaling addresses gradient growth across time steps.
- — Achieves 50% memory reduction and up to 2x speedup on image classification and generative tasks.
- — Open-source PyTorch package (rampde) provides drop-in replacement for existing code.
- — Explicit ODE solvers paired with custom backpropagation enable the precision switching strategy.
Sık sorulanlar
- Neural ODEs solve differential equations iteratively over many time steps. Roundoff errors from low-precision arithmetic accumulate across these iterations, causing numerical instability. Standard mixed precision, which only protects weights, does not account for the error growth in the solution trajectory itself. This framework adds high-precision accumulation of gradients and solutions to mitigate that problem.