Two-Layer Linear Auto-Regressive Models Estimate Latent States

• URL: http://arxiv.org/abs/2606.12691v1
• Date: Wed, 10 Jun 2026 21:25:16 GMT
• Title: Two-Layer Linear Auto-Regressive Models Estimate Latent States
• Abstract summary: We show how two-layer auto-regressive models learn to approximate Kalman latent estimation.<n>We also show that the representations of auto-regressive models recover state estimates.
• Score: 20.029974938994663
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Auto-regressive models have emerged as powerful tools for sequential data, from language to video. Understanding how and why these models learn latent representations remains an open theoretical question. In this work, we demonstrate that when trained by empirical risk minimization on data from partially observed linear dynamical systems, two-layer linear auto-regressive models naturally learn to approximate Kalman filtering. In particular, we show that the learned hidden representation coincides, up to a similarity transformation, with the state estimates produced by the optimal (Kalman) filter, even though the model has no explicit knowledge of the underlying dynamics or state. The result follows from three main insights. First, we establish that the Kalman filter is well approximated by an auto-regressive model with bounded truncation error. Second, we show that despite non-convexity, the two-layer optimization landscape is benign, i.e., all stationary points are either strict saddles or global minima. Finally, as our main contributions, we provide finite-sample guarantees on prediction error, parameter estimation error, and latent state recovery. Numerical simulations support the theoretical results and demonstrate that the latent representations of auto-regressive models recover state estimates.

Related papers

• Joint Model and Data Sparsification via the Marginal Likelihood [53.29070892356214]
  We propose jointly learning individual feature and sample relevancies, enabling simultaneous model and data sparsification.<n>This symmetric pruning of model and data offers a natural extension that preserves conjugacy.<n> Empirical results across diverse regression tasks affirm that a joint ARD approach consistently yields both sparse and robust prediction models.
  arXiv  Detail & Related papers  (2026-05-28T13:26:53Z)
• Asymmetric Scaling Laws from Sparse Features [2.7838875470471858]
  We introduce a model for neural scaling laws under sparse activations.<n>Test loss is often dominated by rare coordinates that are never observed in the training input.<n>We show that the sparsity-induced effect persists under nonlinear activations.
  arXiv  Detail & Related papers  (2026-05-22T13:00:52Z)
• Order-Agnostic Autoregressive Modelling with Missing Data [12.971361761803891]
  We show that order-agnostic autoregressive models implicitly perform imputation under a missing completely at random mechanism.<n>We introduce the first principled framework for training them directly on incomplete datasets under general missingness mechanisms.<n>Across a suite of real-world benchmarks, our Missingness-Aware Order-Agnostic Autoregressive Model consistently outperforms established imputation baselines.
  arXiv  Detail & Related papers  (2026-05-07T14:34:08Z)
• Understanding Overparametrization in Survival Models through Interpolation [14.444096460952961]
  Recent advances in machine learning have revealed a more complex pattern, textitdouble-descent, in which test loss, after peaking near the threshold, decreases again as model capacity continues to grow.<n>This study investigates overparametrization in four representative survival models: DeepSurv, PC-Hazard, Nnet-Survival, and N-MTLR.
  arXiv  Detail & Related papers  (2025-12-13T21:23:02Z)
• GKNet: Graph Kalman Filtering and Model Inference via Model-based Deep Learning [10.609815608017065]
  Inference tasks with time series over graphs are of importance in applications such as urban water networks, economics, and networked neuroscience.<n>We propose a graph-aware state space model for graph time series, where both the latent state and the observation equation are parametric graph-induced models with a limited number of parameters that need to be learned.
  arXiv  Detail & Related papers  (2025-06-27T08:17:07Z)
• Lost in Retraining: Roaming the Parameter Space of Exponential Families Under Closed-Loop Learning [0.0]
  We study closed-loop learning for models that belong to exponential families.<n>We show that maximum likelihood of the parameters endows sufficient statistics with the martingale property.<n>We show that this outcome may be prevented if the data contains at least one data point generated from a ground truth model.
  arXiv  Detail & Related papers  (2025-06-25T17:12:22Z)
• One-for-More: Continual Diffusion Model for Anomaly Detection [63.50488826645681]
  Anomaly detection methods utilize diffusion models to generate or reconstruct normal samples when given arbitrary anomaly images.<n>Our study found that the diffusion model suffers from severe faithfulness hallucination'' and catastrophic forgetting''<n>We propose a continual diffusion model that uses gradient projection to achieve stable continual learning.
  arXiv  Detail & Related papers  (2025-02-27T07:47:27Z)
• Derivative-Free Diffusion Manifold-Constrained Gradient for Unified XAI [59.96044730204345]
  We introduce Derivative-Free Diffusion Manifold-Constrainted Gradients (FreeMCG)<n>FreeMCG serves as an improved basis for explainability of a given neural network.<n>We show that our method yields state-of-the-art results while preserving the essential properties expected of XAI tools.
  arXiv  Detail & Related papers  (2024-11-22T11:15:14Z)
• Uncertainty Representations in State-Space Layers for Deep Reinforcement Learning under Partial Observability [59.758009422067]
  We propose a standalone Kalman filter layer that performs closed-form Gaussian inference in linear state-space models.<n>Similar to efficient linear recurrent layers, the Kalman filter layer processes sequential data using a parallel scan.<n> Experiments show that Kalman filter layers excel in problems where uncertainty reasoning is key for decision-making, outperforming other stateful models.
  arXiv  Detail & Related papers  (2024-09-25T11:22:29Z)
• Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processes [52.92110730286403]
  It is commonly believed that the marginal likelihood should be reminiscent of cross-validation metrics and that both should deteriorate with larger input dimensions. We prove that by tuning hyper parameters, the performance, as measured by the marginal likelihood, improves monotonically with the input dimension. We also prove that cross-validation metrics exhibit qualitatively different behavior that is characteristic of double descent.
  arXiv  Detail & Related papers  (2022-10-14T08:09:33Z)
• Second-order regression models exhibit progressive sharpening to the edge of stability [30.92413051155244]
  We show that for quadratic objectives in two dimensions, a second-order regression model exhibits progressive sharpening towards a value that differs slightly from the edge of stability. In higher dimensions, the model generically shows similar behavior, even without the specific structure of a neural network.
  arXiv  Detail & Related papers  (2022-10-10T17:21:20Z)
• Probabilistic Modeling for Human Mesh Recovery [73.11532990173441]
  This paper focuses on the problem of 3D human reconstruction from 2D evidence. We recast the problem as learning a mapping from the input to a distribution of plausible 3D poses.
  arXiv  Detail & Related papers  (2021-08-26T17:55:11Z)
• Goal-directed Generation of Discrete Structures with Conditional Generative Models [85.51463588099556]
  We introduce a novel approach to directly optimize a reinforcement learning objective, maximizing an expected reward. We test our methodology on two tasks: generating molecules with user-defined properties and identifying short python expressions which evaluate to a given target value.
  arXiv  Detail & Related papers  (2020-10-05T20:03:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.