Transformer-like Inference from Optimal Control

• URL: http://arxiv.org/abs/2605.15608v1
• Date: Fri, 15 May 2026 04:42:19 GMT
• Title: Transformer-like Inference from Optimal Control
• Abstract summary: This paper derives, from first principles, inference architectures that solve the same prediction problem.<n>For two model classes, the prediction objective is reformulated as an optimal control problem.<n>Experiments reveal that when the embedding dimension is insufficient, the transformer implicitly exploits non-Markovian structure.
• Score: 0.5161531917413708
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Decoder-only transformers compute the conditional probability of the next token from a sequence of past observations. This paper derives, from first principles, inference architectures that solve the same prediction problem - and in doing so, recovers transformer-like layer operations as a consequence of optimal control theory. The framework is developed for two model classes: a nonlinear model of discrete-valued processes, directly motivated by the transformer, and a linear Gaussian model as a tractable baseline. For both model classes, the prediction objective is reformulated as an optimal control problem whose solution yields an explicit inference algorithm, the dual filter, with a layer structure that mirrors the layer structure of a decoder-only transformer. Numerical experiments provide a comparison of the optimal control to attention weights from a trained transformer. These experiments reveal that when the embedding dimension is insufficient, the transformer implicitly exploits non-Markovian structure.

Related papers

• Correction of Transformer-Based Models with Smoothing Pseudo-Projector [0.0]
  The pseudo-projector is a lightweight modification that can be integrated into existing language models and other neural networks.<n>It can be viewed as a hidden-representation corrector that reduces sensitivity to noise by suppressing directions induced by label-irrelevant input content.
  arXiv  Detail & Related papers  (2026-03-10T15:42:46Z)
• A Constrained Optimization Perspective of Unrolled Transformers [77.12297732942095]
  We introduce a constrained optimization framework for training transformers that behave like optimization descent algorithms.<n>We observe constrained transformers achieve stronger to perturbations robustness and maintain higher out-of-distribution generalization.
  arXiv  Detail & Related papers  (2026-01-24T02:12:39Z)
• Sparse Transformer Architectures via Regularized Wasserstein Proximal Operator with $L_1$ Prior [0.49193859756091124]
  We propose a sparse transformer architecture that incorporates prior information about the underlying data distribution directly into the transformer structure of the neural network.<n>We demonstrate that the sparse transformer achieves higher accuracy and faster convergence to the target distribution than classical neural ODE-based methods.
  arXiv  Detail & Related papers  (2025-10-18T05:26:13Z)
• Dual Filter: A Mathematical Framework for Inference using Transformer-like Architectures [1.9567015559455132]
  We present a framework for causal nonlinear prediction in settings where observations are generated from an underlying hidden Markov model (HMM)<n>Both the problem formulation and the proposed solution are motivated by the decoder-only transformer architecture.
  arXiv  Detail & Related papers  (2025-05-01T19:19:29Z)
• In-Context Learning of Linear Dynamical Systems with Transformers: Approximation Bounds and Depth-Separation [14.716318664357408]
  This paper investigates approximation-theoretic aspects of the in-context learning capability of the transformers in representing a family of noisy linear dynamical systems.<n>Our first theoretical result establishes an upper bound on the approximation error of multi-layer transformers with respect to an $L2$-testing loss uniformly defined across tasks.<n>Our second result establishes a non-diminishing lower bound on the approximation error for a class of single-layer linear transformers.
  arXiv  Detail & Related papers  (2025-02-12T05:40:11Z)
• Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalization [88.5582111768376]
  We study the optimization of a Transformer composed of a self-attention layer with softmax followed by a fully connected layer under gradient descent on a certain data distribution model. Our results establish a sharp condition that can distinguish between the small test error phase and the large test error regime, based on the signal-to-noise ratio in the data model.
  arXiv  Detail & Related papers  (2024-09-28T13:24:11Z)
• Learning on Transformers is Provable Low-Rank and Sparse: A One-layer Analysis [63.66763657191476]
  We show that efficient numerical training and inference algorithms as low-rank computation have impressive performance for learning Transformer-based adaption. We analyze how magnitude-based models affect generalization while improving adaption. We conclude that proper magnitude-based has a slight on the testing performance.
  arXiv  Detail & Related papers  (2024-06-24T23:00:58Z)
• EulerFormer: Sequential User Behavior Modeling with Complex Vector Attention [88.45459681677369]
  We propose a novel transformer variant with complex vector attention, named EulerFormer. It provides a unified theoretical framework to formulate both semantic difference and positional difference. It is more robust to semantic variations and possesses moresuperior theoretical properties in principle.
  arXiv  Detail & Related papers  (2024-03-26T14:18:43Z)
• In-Context Convergence of Transformers [63.04956160537308]
  We study the learning dynamics of a one-layer transformer with softmax attention trained via gradient descent. For data with imbalanced features, we show that the learning dynamics take a stage-wise convergence process.
  arXiv  Detail & Related papers  (2023-10-08T17:55:33Z)
• Uncovering mesa-optimization algorithms in Transformers [61.06055590704677]
  Some autoregressive models can learn as an input sequence is processed, without undergoing any parameter changes, and without being explicitly trained to do so. We show that standard next-token prediction error minimization gives rise to a subsidiary learning algorithm that adjusts the model as new inputs are revealed. Our findings explain in-context learning as a product of autoregressive loss minimization and inform the design of new optimization-based Transformer layers.
  arXiv  Detail & Related papers  (2023-09-11T22:42:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.