Learning to Solve Generative ODEs Beyond the Linear Span

• URL: http://arxiv.org/abs/2606.08672v1
• Date: Sun, 07 Jun 2026 15:22:13 GMT
• Title: Learning to Solve Generative ODEs Beyond the Linear Span
• Abstract summary: We propose SpanLift, a lightweight neural solver that augments scalar-coefficient updates with a spatial residual operator.<n>Across pixel-space diffusion, latent flow matching, and precipitation nowcasting, SpanLift achieves state-of-the-art few-step sampling.
• Score: 50.13853710831612
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion and flow generative models sample by integrating a learned ODE, but high quality still requires many sequential model evaluations. Solver learning reduces this cost by adapting scalar coefficients, timesteps, or both, while keeping the backbone model fixed. In this work, we identify a structural bottleneck in this update family: each step remains span-limited. Since the scalar-coefficient update lies in the span of buffered velocity evaluations, it can fit only the in-span component while leaving any out-of-span residual unreachable by scalar recombination alone. We propose SpanLift, a lightweight neural solver that augments scalar-coefficient updates with a spatial residual operator. SpanLift keeps a fixed base solver as an in-span prior and learns a spatial residual operator over the state and velocity buffer. The operator is trained by endpoint teacher matching, preserves the pretrained backbone, and adds no model NFEs. Empirically, the learned correction transfers across base solvers and is predominantly out-of-span. Across pixel-space diffusion, latent flow matching, and precipitation nowcasting, SpanLift achieves state-of-the-art few-step sampling. With only 3 NFE, it improves CIFAR-10 FID from 8.16 to 5.69 and ImageNet FID from 17.37 to 11.83.

Related papers

• Training Crossroads for Recurrent Vision Transformers: Recurrence, Neural ODEs, and Deep Supervision [2.1465474237525113]
  Single-block recurrent ViTs (bViT) remove this growth by repeatedly applying one shared block.<n>We find that standard ViTs remain preferable when FLOPs are the primary constraint, whereas recurrent ViTs offer a better accuracy-- parameter trade-off under memory constraints.
  arXiv  Detail & Related papers  (2026-08-05T14:06:50Z)
• Ouroboros-Spatial: Closing the Data-Model Loop for Spatial Reasoning [13.767813688729161]
  Existing approaches largely rely on large-scale, statically curated datasets.<n>Ouroboros-Spatial is a self-evolving training framework in which the model plays dual roles as a proposer and a solver.<n>Ouroboros-Spatial substantially improves Qwen3-VL-4B and Qwen3-VL-8B while using an order of fewer training examples than recent large-scale curated datasets.
  arXiv  Detail & Related papers  (2026-06-10T06:49:21Z)
• Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Models [37.194177406897964]
  We propose a closed-form formulation that computes the linear probe induced by the synthetic set with a sample-space kernel ridge solver.<n>On ImageNet-100 with four pre-trained backbones, CLP-DD substantially improves over LGM without DSA and approaches LGM with DSA at a fraction of the computational cost.<n>On ImageNet-1K, CLP-DD matches or surpasses LGM with DSA on three of four backbones while running roughly $14times$ faster and using less than one-eighth of the GPU memory.
  arXiv  Detail & Related papers  (2026-05-08T03:41:51Z)
• SLE-FNO: Single-Layer Extensions for Task-Agnostic Continual Learning in Fourier Neural Operators [0.4369550829556577]
  New experimental conditions or simulation regimes may differ significantly, requiring extrapolation and model updates without re-access to prior data.<n>This creates a need for continual learning (CL) frameworks that can adapt to distribution shifts while preventing catastrophic forgetting.<n>We introduce a new architecture-based approach (SLE-FNO) combining a Single-Layer Extension (SLE) with the Fourier Neural Operator (FNO) to support efficient CL.
  arXiv  Detail & Related papers  (2026-03-20T18:30:38Z)
• SCORE: Replacing Layer Stacking with Contractive Recurrent Depth [0.0]
  Residual connections are central to modern deep neural networks.<n>We propose SCORE (Skip-Connection ODE Recurrent Embedding), a discrete recurrent alternative to classical layer stacking.
  arXiv  Detail & Related papers  (2026-03-11T08:49:03Z)
• Network-Optimised Spiking Neural Network for Event-Driven Networking [2.5941336499463383]
  Spiking neural networks offer event-driven computation suited to time-critical networking tasks such as anomaly detection, local routing control, and congestion management at the edge.<n>We introduce Network-Optimised Spiking (NOS), a compact two-variable unit whose state encodes normalised queue occupancy and a recovery resource.<n>We provide guidance for data-driven initialisation, surrogate-gradient training with a homotopy on reset sharpness, and explicit stability checks with topology of bounds for resource constrained deployments.
  arXiv  Detail & Related papers  (2025-09-27T22:31:24Z)
• VoxelOpt: Voxel-Adaptive Message Passing for Discrete Optimization in Deformable Abdominal CT Registration [15.78340001680369]
  We propose VoxelOpt, a discrete optimization-based deformable image registration framework.<n>It combines the strengths of learning-based and iterative methods to achieve a better balance between registration accuracy and runtime.<n>In abdominal CT registration, these changes allow VoxelOpt to outperform leading iterative in both efficiency and accuracy, while matching state-of-the-art learning-based methods trained with label supervision.
  arXiv  Detail & Related papers  (2025-06-24T19:44:04Z)
• A Stable Whitening Optimizer for Efficient Neural Network Training [99.7641280234926]
  Building on the Shampoo family of algorithms, we identify and alleviate three key issues, resulting in the proposed SPlus method.<n>First, we find that naive Shampoo is prone to divergence when matrix-inverses are cached for long periods.<n>Second, we adapt a shape-aware scaling to enable learning rate transfer across network width.<n>Third, we find that high learning rates result in large parameter noise, and propose a simple iterate-averaging scheme which unblocks faster learning.
  arXiv  Detail & Related papers  (2025-06-08T18:43:31Z)
• Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Model [89.8764435351222]
  We propose a new family of unbiased estimators called WTA-CRS, for matrix production with reduced variance. Our work provides both theoretical and experimental evidence that, in the context of tuning transformers, our proposed estimators exhibit lower variance compared to existing ones.
  arXiv  Detail & Related papers  (2023-05-24T15:52:08Z)
• Improved techniques for deterministic l2 robustness [63.34032156196848]
  Training convolutional neural networks (CNNs) with a strict 1-Lipschitz constraint under the $l_2$ norm is useful for adversarial robustness, interpretable gradients and stable training. We introduce a procedure to certify robustness of 1-Lipschitz CNNs by replacing the last linear layer with a 1-hidden layer. We significantly advance the state-of-the-art for standard and provable robust accuracies on CIFAR-10 and CIFAR-100.
  arXiv  Detail & Related papers  (2022-11-15T19:10:12Z)
• Deep Equilibrium Optical Flow Estimation [80.80992684796566]
  Recent state-of-the-art (SOTA) optical flow models use finite-step recurrent update operations to emulate traditional algorithms. These RNNs impose large computation and memory overheads, and are not directly trained to model such stable estimation. We propose deep equilibrium (DEQ) flow estimators, an approach that directly solves for the flow as the infinite-level fixed point of an implicit layer.
  arXiv  Detail & Related papers  (2022-04-18T17:53:44Z)
• Robust Implicit Networks via Non-Euclidean Contractions [63.91638306025768]
  Implicit neural networks show improved accuracy and significant reduction in memory consumption. They can suffer from ill-posedness and convergence instability. This paper provides a new framework to design well-posed and robust implicit neural networks.
  arXiv  Detail & Related papers  (2021-06-06T18:05:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.