Adaptive Probabilistic ODE Solvers Without Adaptive Memory Requirements

• URL: http://arxiv.org/abs/2410.10530v2
• Date: Thu, 03 Jul 2025 12:07:20 GMT
• Title: Adaptive Probabilistic ODE Solvers Without Adaptive Memory Requirements
• Authors: Nicholas Krämer,
• Abstract summary: We develop an adaptive probabilistic solver with fixed memory demands.<n>Switching to our method eliminates memory issues for long time series.<n>We also accelerate simulations by orders of magnitude through unlocking just-in-time compilation.
• Score: 6.0735728088312175
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite substantial progress in recent years, probabilistic solvers with adaptive step sizes can still not solve memory-demanding differential equations -- unless we care only about a single point in time (which is far too restrictive; we want the whole time series). Counterintuitively, the culprit is the adaptivity itself: Its unpredictable memory demands easily exceed our machine's capabilities, making our simulations fail unexpectedly and without warning. Still, dropping adaptivity would abandon years of progress, which can't be the answer. In this work, we solve this conundrum. We develop an adaptive probabilistic solver with fixed memory demands building on recent developments in robust state estimation. Switching to our method (i) eliminates memory issues for long time series, (ii) accelerates simulations by orders of magnitude through unlocking just-in-time compilation, and (iii) makes adaptive probabilistic solvers compatible with scientific computing in JAX.

Related papers

• Gated KalmaNet: A Fading Memory Layer Through Test-Time Ridge Regression [53.48692193399171]
  Gated KalmaNet (GKA) is a layer that reduces the gap by accounting for the full past when predicting the next token.<n>We solve an online ridge regression problem at test time, with constant memory and linear compute cost in the sequence length.<n>On long-context, GKA excels at real-world RAG and LongQA tasks up to 128k tokens, achieving more than $10$% relative improvement over other fading memory baselines.
  arXiv  Detail & Related papers  (2025-11-26T03:26:37Z)
• Accelerated zero-order SGD under high-order smoothness and overparameterized regime [79.85163929026146]
  We present a novel gradient-free algorithm to solve convex optimization problems. Such problems are encountered in medicine, physics, and machine learning. We provide convergence guarantees for the proposed algorithm under both types of noise.
  arXiv  Detail & Related papers  (2024-11-21T10:26:17Z)
• Computation-Aware Kalman Filtering and Smoothing [27.55456716194024]
  We propose a probabilistic numerical inference for high-dimensional Gauss-ov models. Our algorithm leverages GPU acceleration and crucially enables a tunable trade-off between predictive cost and uncertainty.
  arXiv  Detail & Related papers  (2024-05-14T21:31:11Z)
• AdaLomo: Low-memory Optimization with Adaptive Learning Rate [59.64965955386855]
  We introduce low-memory optimization with adaptive learning rate (AdaLomo) for large language models. AdaLomo results on par with AdamW, while significantly reducing memory requirements, thereby lowering the hardware barrier to training large language models.
  arXiv  Detail & Related papers  (2023-10-16T09:04:28Z)
• Memory-and-Anticipation Transformer for Online Action Understanding [52.24561192781971]
  We propose a novel memory-anticipation-based paradigm to model an entire temporal structure, including the past, present, and future. We present Memory-and-Anticipation Transformer (MAT), a memory-anticipation-based approach, to address the online action detection and anticipation tasks.
  arXiv  Detail & Related papers  (2023-08-15T17:34:54Z)
• Constrained Optimization via Exact Augmented Lagrangian and Randomized Iterative Sketching [55.28394191394675]
  We develop an adaptive inexact Newton method for equality-constrained nonlinear, nonIBS optimization problems. We demonstrate the superior performance of our method on benchmark nonlinear problems, constrained logistic regression with data from LVM, and a PDE-constrained problem.
  arXiv  Detail & Related papers  (2023-05-28T06:33:37Z)
• Adaptive Sparse Gaussian Process [0.0]
  We propose the first adaptive sparse Gaussian Process (GP) able to address all these issues. We first reformulate a variational sparse GP algorithm to make it adaptive through a forgetting factor. We then propose updating a single inducing point of the sparse GP model together with the remaining model parameters every time a new sample arrives.
  arXiv  Detail & Related papers  (2023-02-20T21:34:36Z)
• Memory-Efficient Differentiable Programming for Quantum Optimal Control of Discrete Lattices [1.5012666537539614]
  Quantum optimal control problems are typically solved by gradient-based algorithms such as GRAPE. QOC reveals that memory requirements are a barrier for simulating large models or long time spans. We employ a nonstandard differentiable programming approach that significantly reduces the memory requirements at the cost of a reasonable amount of recomputation.
  arXiv  Detail & Related papers  (2022-10-15T20:59:23Z)
• Confident Adaptive Language Modeling [95.45272377648773]
  CALM is a framework for dynamically allocating different amounts of compute per input and generation timestep. We demonstrate the efficacy of our framework in reducing compute -- potential speedup of up to $times 3$ -- while provably maintaining high performance.
  arXiv  Detail & Related papers  (2022-07-14T17:00:19Z)
• Reducing Memory Requirements of Quantum Optimal Control [0.0]
  gradient-based algorithms such as GRAPE suffer from exponential growth in storage with increasing number of qubits and linear growth in memory requirements with increasing number of time steps. We have created a nonstandard automatic differentiation technique that can compute gradients needed by GRAPE by exploiting the fact that the inverse of a unitary matrix is its conjugate transpose. Our approach significantly reduces the memory requirements for GRAPE, at the cost of a reasonable amount of recomputation.
  arXiv  Detail & Related papers  (2022-03-23T20:42:54Z)
• Learning Under Delayed Feedback: Implicitly Adapting to Gradient Delays [0.0]
  We consider convex optimization problems, where several machines act asynchronously in parallel while sharing a common memory. We propose a robust training method for the constrained setting and derive non convergence guarantees that do not depend on prior knowledge of update delays, objective smoothness, and variance.
  arXiv  Detail & Related papers  (2021-06-23T09:36:36Z)
• Balancing Rates and Variance via Adaptive Batch-Size for Stochastic Optimization Problems [120.21685755278509]
  In this work, we seek to balance the fact that attenuating step-size is required for exact convergence with the fact that constant step-size learns faster in time up to an error. Rather than fixing the minibatch the step-size at the outset, we propose to allow parameters to evolve adaptively.
  arXiv  Detail & Related papers  (2020-07-02T16:02:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.