Vision-Conditioned Variational Bayesian Last Layer Dynamics Models

• URL: http://arxiv.org/abs/2601.09178v2
• Date: Fri, 16 Jan 2026 16:32:59 GMT
• Title: Vision-Conditioned Variational Bayesian Last Layer Dynamics Models
• Abstract summary: Agile control of robotic systems often requires anticipating how the environment affects system behavior.<n>Traditional modeling approaches often struggle to capture abrupt variations in system behavior.<n>We propose a vision-conditioned variational Bayesian last-layer dynamics model that leverages visual context to anticipate changes in the environment.
• Score: 2.936118677102837
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Agile control of robotic systems often requires anticipating how the environment affects system behavior. For example, a driver must perceive the road ahead to anticipate available friction and plan actions accordingly. Achieving such proactive adaptation within autonomous frameworks remains a challenge, particularly under rapidly changing conditions. Traditional modeling approaches often struggle to capture abrupt variations in system behavior, while adaptive methods are inherently reactive and may adapt too late to ensure safety. We propose a vision-conditioned variational Bayesian last-layer dynamics model that leverages visual context to anticipate changes in the environment. The model first learns nominal vehicle dynamics and is then fine-tuned with feature-wise affine transformations of latent features, enabling context-aware dynamics prediction. The resulting model is integrated into an optimal controller for vehicle racing. We validate our method on a Lexus LC500 racing through water puddles. With vision-conditioning, the system completed all 12 attempted laps under varying conditions. In contrast, all baselines without visual context consistently lost control, demonstrating the importance of proactive dynamics adaptation in high-performance applications.

Related papers

• End-to-end Conditional Diffusion for Realistic and Controllable Visual Traffic Scenario Generation [59.188006170485046]
  E2E-CDiff is an end-to-end conditional diffusion framework for controllable and realistic scenario generation.<n>Differentiable guidance regulates speed, enforces drivable-area compliance, and supports collision-avoidance or collision-seeking behaviors.<n>E2E-CDiff also performs competitively as a learning-based ego planner.
  arXiv  Detail & Related papers  (2026-07-21T02:13:25Z)
• Learning Context-Aware Neural ODE Dynamics for Adaptive Robotic Control [55.01272784312405]
  We propose a context-aware dynamics model based on neural ordinary differential equations.<n>We validate the approach across diverse robotic platforms, including a quadrotor in simulation, as well as a Sphero BOLT robot and a Fanuc manipulator in real-world experiments.
  arXiv  Detail & Related papers  (2026-06-13T20:55:46Z)
• Multistep Belief Space Dynamics Learning For Risk-Aware Control [16.923373296752533]
  We present a learning framework to predict distributional dynamics that can be optimized in real time for Model Predictive Control (MPC)<n>A rigorous ablation study is conducted on a large dataset of real world off-road driving that shows the impact of deviations from our proposed structure.<n>Our planning architecture is able to naturally regulate the speed of the vehicle based on the environment and consistently demonstrates intelligent behavior over miles of diverse terrain.
  arXiv  Detail & Related papers  (2026-05-12T18:11:12Z)
• DynFlowDrive: Flow-Based Dynamic World Modeling for Autonomous Driving [67.83404465171257]
  DynFlowDrive is a latent world model that models the transition of world states under different driving actions.<n>It learns a velocity field that describes how the scene state changes under different driving actions.<n>Building upon this, we introduce a stability-aware multi-mode trajectory selection strategy.
  arXiv  Detail & Related papers  (2026-03-20T06:19:31Z)
• Optimization-Guided Diffusion for Interactive Scene Generation [52.23368750264419]
  We present OMEGA, an optimization-guided, training-free framework that enforces structural consistency and interaction awareness during diffusion-based sampling.<n>We show that OMEGA improves generation realism, consistency, and controllability, increasing the ratio of physically and behaviorally valid scenes.<n>Our approach can also generate $5times$ more near-collision frames with a time-to-collision under three seconds.
  arXiv  Detail & Related papers  (2025-12-08T15:56:18Z)
• Vehicle Dynamics Embedded World Models for Autonomous Driving [13.012978896886134]
  We propose the Vehicle Dynamics embedded Dreamer (VDD) method, which decouples the modeling of ego-vehicle dynamics from environmental transition dynamics.<n>The proposed model significantly improves both driving performance and robustness to variations in vehicle dynamics, outperforming existing approaches.
  arXiv  Detail & Related papers  (2025-12-02T04:57:52Z)
• SEAL: Vision-Language Model-Based Safe End-to-End Cooperative Autonomous Driving with Adaptive Long-Tail Modeling [13.81210267833274]
  SEAL is a vision-based model-based framework with adaptive multimodal learning for robust cooperative autonomous driving under long-tail scenarios.<n> SEAL introduces three core innovations: (i) a prompt-driven long-tail scenario generation and evaluation pipeline that leverages foundation models to synthesize realistic long-tail conditions; (ii) a multi-scenario adaptive attention module that modulates the visual stream using scenario priors to recalibrate ambiguous or corrupted features; and (iii) a multi-task scenario-aware contrastive learning objective that improves multimodal alignment and promotes cross-scenario feature separability.
  arXiv  Detail & Related papers  (2025-06-26T06:42:03Z)
• Meta-Learning Online Dynamics Model Adaptation in Off-Road Autonomous Driving [9.515695438588855]
  High-speed off-road autonomous driving presents unique challenges due to complex, evolving terrain characteristics.<n>We propose a novel framework that combines a Kalman filter-based online adaptation scheme with meta-learned parameters to address these challenges.<n>Our results underscore the effectiveness of meta-learned dynamics model adaptation, advancing the development of reliable autonomous systems.
  arXiv  Detail & Related papers  (2025-04-23T17:51:36Z)
• MetaFollower: Adaptable Personalized Autonomous Car Following [63.90050686330677]
  We propose an adaptable personalized car-following framework - MetaFollower. We first utilize Model-Agnostic Meta-Learning (MAML) to extract common driving knowledge from various CF events. We additionally combine Long Short-Term Memory (LSTM) and Intelligent Driver Model (IDM) to reflect temporal heterogeneity with high interpretability.
  arXiv  Detail & Related papers  (2024-06-23T15:30:40Z)
• SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries [94.84458417662407]
  We introduce SAFE-SIM, a controllable closed-loop safety-critical simulation framework. Our approach yields two distinct advantages: 1) generating realistic long-tail safety-critical scenarios that closely reflect real-world conditions, and 2) providing controllable adversarial behavior for more comprehensive and interactive evaluations. We validate our framework empirically using the nuScenes and nuPlan datasets across multiple planners, demonstrating improvements in both realism and controllability.
  arXiv  Detail & Related papers  (2023-12-31T04:14:43Z)
• Learning Terrain-Aware Kinodynamic Model for Autonomous Off-Road Rally Driving With Model Predictive Path Integral Control [4.23755398158039]
  We propose a method for learning terrain-aware kinodynamic model conditioned on both proprioceptive and exteroceptive information. The proposed model generates reliable predictions of 6-degree-of-freedom motion and can even estimate contact interactions. We demonstrate the effectiveness of our approach through experiments on a simulated off-road track, showing that our proposed model-controller pair outperforms the baseline.
  arXiv  Detail & Related papers  (2023-05-01T06:09:49Z)
• Active Learning of Discrete-Time Dynamics for Uncertainty-Aware Model Predictive Control [46.81433026280051]
  We present a self-supervised learning approach that actively models the dynamics of nonlinear robotic systems. Our approach showcases high resilience and generalization capabilities by consistently adapting to unseen flight conditions.
  arXiv  Detail & Related papers  (2022-10-23T00:45:05Z)
• Isolating and Leveraging Controllable and Noncontrollable Visual Dynamics in World Models [65.97707691164558]
  We present Iso-Dream, which improves the Dream-to-Control framework in two aspects. First, by optimizing inverse dynamics, we encourage world model to learn controllable and noncontrollable sources. Second, we optimize the behavior of the agent on the decoupled latent imaginations of the world model.
  arXiv  Detail & Related papers  (2022-05-27T08:07:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.