ODE-ViT: Plug & Play Attention Layer from the Generalization of the ViT as an Ordinary Differential Equation

• URL: http://arxiv.org/abs/2511.16501v1
• Date: Thu, 20 Nov 2025 16:19:41 GMT
• Title: ODE-ViT: Plug & Play Attention Layer from the Generalization of the ViT as an Ordinary Differential Equation
• Abstract summary: We introduce ODE-ViT, a Vision Transformer that satisfies the conditions for well-posed and stable dynamics.<n>Experiments on CIFAR-10 and CIFAR-100 demonstrate that ODE-ViT achieves stable, interpretable, and competitive performance with up to one order of magnitude fewer parameters.
• Score: 2.0973843981871574
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In recent years, increasingly large models have achieved outstanding performance across CV tasks. However, these models demand substantial computational resources and storage, and their growing complexity limits our understanding of how they make decisions. Most of these architectures rely on the attention mechanism within Transformer-based designs. Building upon the connection between residual neural networks and ordinary differential equations (ODEs), we introduce ODE-ViT, a Vision Transformer reformulated as an ODE system that satisfies the conditions for well-posed and stable dynamics. Experiments on CIFAR-10 and CIFAR-100 demonstrate that ODE-ViT achieves stable, interpretable, and competitive performance with up to one order of magnitude fewer parameters, surpassing prior ODE-based Transformer approaches in classification tasks. We further propose a plug-and-play teacher-student framework in which a discrete ViT guides the continuous trajectory of ODE-ViT by treating the intermediate representations of the teacher as solutions of the ODE. This strategy improves performance by more than 10% compared to training a free ODE-ViT from scratch.

Related papers

• Expanding the Chaos: Neural Operator for Stochastic (Partial) Differential Equations [65.80144621950981]
  We build on Wiener chaos expansions (WCE) to design neural operator (NO) architectures for SPDEs and SDEs.<n>We show that WCE-based neural operators provide a practical and scalable way to learn SDE/SPDE solution operators.
  arXiv  Detail & Related papers  (2026-01-03T00:59:25Z)
• Efficient Generative Transformer Operators For Million-Point PDEs [12.324265832276538]
  ECHO is a transformer-operator framework for generating million-point PDE trajectories.<n>We demonstrate state-of-the-art performance on million-point simulations featuring complex, high-frequency dynamics, and long-term horizons.
  arXiv  Detail & Related papers  (2025-12-04T16:46:48Z)
• X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model [62.21943953611646]
  Vision-Language-Action models rely on effective training across diverse robotic platforms.<n>We propose a novel Soft Prompt approach with minimally added parameters.<n>We show that our 0.9B instantiation-X-VLA-0.9B simultaneously achieves SOTA performance over a sweep of benchmarks.
  arXiv  Detail & Related papers  (2025-10-11T16:20:17Z)
• Transfer Operator Learning with Fusion Frame [0.0]
  This work presents a novel framework that enhances the transfer learning capabilities of operator learning models for solving Partial Differential Equations (PDEs) We introduce an innovative architecture that combines fusion frames with POD-DeepONet, demonstrating superior performance across various PDEs in our experimental analysis. Our framework addresses the critical challenge of transfer learning in operator learning models, paving the way for adaptable and efficient solutions across a wide range of scientific and engineering applications.
  arXiv  Detail & Related papers  (2024-08-20T00:03:23Z)
• KAN-ODEs: Kolmogorov-Arnold Network Ordinary Differential Equations for Learning Dynamical Systems and Hidden Physics [0.0]
  Kolmogorov-Arnold networks (KANs) are an alternative to multi-layer perceptrons (MLPs) This work applies KANs as the backbone of a neural ordinary differential equation (ODE) framework.
  arXiv  Detail & Related papers  (2024-07-05T00:38:49Z)
• CViT: Continuous Vision Transformer for Operator Learning [24.1795082775376]
  Continuous Vision Transformer (CViT) is a novel neural operator architecture that leverages advances in computer vision to address challenges in learning complex physical systems.<n>CViT combines a vision transformer encoder, a novel grid-based coordinate embedding, and a query-wise cross-attention mechanism to effectively capture multi-scale dependencies.<n>We demonstrate CViT's effectiveness across a diverse range of partial differential equation (PDE) systems, including fluid dynamics, climate modeling, and reaction-diffusion processes.
  arXiv  Detail & Related papers  (2024-05-22T21:13:23Z)
• On the Trajectory Regularity of ODE-based Diffusion Sampling [79.17334230868693]
  Diffusion-based generative models use differential equations to establish a smooth connection between a complex data distribution and a tractable prior distribution. In this paper, we identify several intriguing trajectory properties in the ODE-based sampling process of diffusion models.
  arXiv  Detail & Related papers  (2024-05-18T15:59:41Z)
• HSViT: Horizontally Scalable Vision Transformer [16.46308352393693]
  Vision Transformer (ViT) requires pre-training on large-scale datasets to perform well. This paper introduces a novel horizontally scalable vision transformer (HSViT) scheme. HSViT achieves up to 10% higher top-1 accuracy than state-of-the-art schemes on small datasets.
  arXiv  Detail & Related papers  (2024-04-08T04:53:29Z)
• DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training [87.90342423839876]
  We present a new auto-regressive denoising pre-training strategy, which allows for more stable and efficient pre-training on PDE data. We train our PDE foundation model with up to 0.5B parameters on 10+ PDE datasets with more than 100k trajectories.
  arXiv  Detail & Related papers  (2024-03-06T08:38:34Z)
• End-to-End Meta-Bayesian Optimisation with Transformer Neural Processes [52.818579746354665]
  This paper proposes the first end-to-end differentiable meta-BO framework that generalises neural processes to learn acquisition functions via transformer architectures. We enable this end-to-end framework with reinforcement learning (RL) to tackle the lack of labelled acquisition data.
  arXiv  Detail & Related papers  (2023-05-25T10:58:46Z)
• Global Vision Transformer Pruning with Hessian-Aware Saliency [93.33895899995224]
  This work challenges the common design philosophy of the Vision Transformer (ViT) model with uniform dimension across all the stacked blocks in a model stage. We derive a novel Hessian-based structural pruning criteria comparable across all layers and structures, with latency-aware regularization for direct latency reduction. Performing iterative pruning on the DeiT-Base model leads to a new architecture family called NViT (Novel ViT), with a novel parameter that utilizes parameters more efficiently.
  arXiv  Detail & Related papers  (2021-10-10T18:04:59Z)
• ODE Transformer: An Ordinary Differential Equation-Inspired Model for Neural Machine Translation [25.86053637998726]
  We show that a residual block of layers in Transformer can be described as a higher-order solution to ODEs. As a natural extension to Transformer, ODE Transformer is easy to implement and parameter efficient.
  arXiv  Detail & Related papers  (2021-04-06T06:13:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.