UnHiPPO: Uncertainty-aware Initialization for State Space Models

• URL: http://arxiv.org/abs/2506.05065v1
• Date: Thu, 05 Jun 2025 14:11:36 GMT
• Title: UnHiPPO: Uncertainty-aware Initialization for State Space Models
• Authors: Marten Lienen, Abdullah Saydemir, Stephan Günnemann,
• Abstract summary: State space models are emerging as a dominant model class for sequence problems.<n>HiPPO assumes data to be noise-free; an assumption often violated in practice.<n>We extend the HiPPO theory with measurement noise and derive an uncertainty-aware framework for state space model dynamics.
• Score: 49.843505326598596
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: State space models are emerging as a dominant model class for sequence problems with many relying on the HiPPO framework to initialize their dynamics. However, HiPPO fundamentally assumes data to be noise-free; an assumption often violated in practice. We extend the HiPPO theory with measurement noise and derive an uncertainty-aware initialization for state space model dynamics. In our analysis, we interpret HiPPO as a linear stochastic control problem where the data enters as a noise-free control signal. We then reformulate the problem so that the data become noisy outputs of a latent system and arrive at an alternative dynamics initialization that infers the posterior of this latent system from the data without increasing runtime. Our experiments show that our initialization improves the resistance of state-space models to noise both at training and inference time. Find our implementation at https://cs.cit.tum.de/daml/unhippo.

Related papers

• NoiseAR: AutoRegressing Initial Noise Prior for Diffusion Models [50.51982871889886]
  NoiseAR is a novel method for AutoRegressive Initial Noise Prior for Diffusion Models.<n>We formulate the generation of the initial noise prior's parameters as an autoregressive probabilistic modeling task over spatial patches or tokens.<n>Our experiments demonstrate that NoiseAR can generate initial noise priors that lead to improved sample quality and enhanced consistency with conditional inputs.
  arXiv  Detail & Related papers  (2025-06-02T05:32:35Z)
• TrackDiffuser: Nearly Model-Free Bayesian Filtering with Diffusion Model [23.40376181606577]
  We present TrackDiffuser, a generative framework addressing both challenges by reformulating Bayesian filtering as a conditional diffusion model.<n>Our approach implicitly learns system dynamics from data to mitigate the effects of inaccurate SSM.<n>TrackDiffuser exhibits remarkable robustness to SSM inaccuracies, offering a practical solution for real-world state estimation problems.
  arXiv  Detail & Related papers  (2025-02-08T16:21:18Z)
• Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence [11.400431211239958]
  Diffusion models have emerged as powerful tools for generative modeling.<n>We propose a control framework for fine-tuning diffusion models.<n>We show that PI-FT achieves global convergence at a linear rate.
  arXiv  Detail & Related papers  (2024-12-24T04:55:46Z)
• Inference at the data's edge: Gaussian processes for modeling and inference under model-dependency, poor overlap, and extrapolation [0.0]
  The Gaussian Process (GP) is a flexible non-linear regression approach. It provides a principled approach to handling our uncertainty over predicted (counterfactual) values. This is especially valuable under conditions of extrapolation or weak overlap.
  arXiv  Detail & Related papers  (2024-07-15T05:09:50Z)
• Impact of Noisy Supervision in Foundation Model Learning [91.56591923244943]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets.<n>We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• One More Step: A Versatile Plug-and-Play Module for Rectifying Diffusion Schedule Flaws and Enhancing Low-Frequency Controls [77.42510898755037]
  One More Step (OMS) is a compact network that incorporates an additional simple yet effective step during inference. OMS elevates image fidelity and harmonizes the dichotomy between training and inference, while preserving original model parameters. Once trained, various pre-trained diffusion models with the same latent domain can share the same OMS module.
  arXiv  Detail & Related papers  (2023-11-27T12:02:42Z)
• Sub-linear Regret in Adaptive Model Predictive Control [56.705978425244496]
  We present STT-MPC (Self-Tuning Tube-based Model Predictive Control), an online oracle that combines the certainty-equivalence principle and polytopic tubes. We analyze the regret of the algorithm, when compared to an algorithm initially aware of the system dynamics.
  arXiv  Detail & Related papers  (2023-10-07T15:07:10Z)
• Learning non-Markovian Decision-Making from State-only Sequences [57.20193609153983]
  We develop a model-based imitation of state-only sequences with non-Markov Decision Process (nMDP) We demonstrate the efficacy of the proposed method in a path planning task with non-Markovian constraints.
  arXiv  Detail & Related papers  (2023-06-27T02:26:01Z)
• Hierarchical model reduction driven by machine learning for parametric advection-diffusion-reaction problems in the presence of noisy data [0.0]
  We propose a new approach to generate a reliable reduced model for a parametric elliptic problem in the presence of noisy data. We show that directional HiPOD looses in terms of accuracy when problem data are affected by noise. We replace with Machine Learning fitting models which better discriminate relevant physical features in the data from irrelevant noise.
  arXiv  Detail & Related papers  (2022-04-01T16:02:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.