Going with the Flow: Koopman Behavioral Models as Implicit Planners for Visuo-Motor Dexterity

• URL: http://arxiv.org/abs/2602.07413v2
• Date: Tue, 10 Feb 2026 17:20:36 GMT
• Title: Going with the Flow: Koopman Behavioral Models as Implicit Planners for Visuo-Motor Dexterity
• Abstract summary: We introduce Unified Behavioral Models (UBMs), a framework that learns to represent dexterous skills as coupled dynamical systems.<n>Koopman-UBM is a first instantiation of UBMs that learns a unified representation in which the joint flow of latent visual and proprioceptive features is governed by a structured linear system.
• Score: 1.6865201067073452
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: There has been rapid and dramatic progress in learning complex visuo-motor manipulation skills from demonstrations, thanks in part to expressive policy classes that employ diffusion- and transformer-based backbones. However, these design choices require significant data and computational resources and remain far from reliable, particularly within the context of multi-fingered dexterous manipulation. Fundamentally, they model skills as reactive mappings and rely on fixed-horizon action chunking to mitigate jitter, creating a rigid trade-off between temporal coherence and reactivity. In this work, we introduce Unified Behavioral Models (UBMs), a framework that learns to represent dexterous skills as coupled dynamical systems that capture how visual features of the environment (visual flow) and proprioceptive states of the robot (action flow) co-evolve. By capturing such behavioral dynamics, UBMs can ensure temporal coherence by construction rather than by heuristic averaging. To operationalize these models, we propose Koopman-UBM, a first instantiation of UBMs that leverages Koopman Operator theory to effectively learn a unified representation in which the joint flow of latent visual and proprioceptive features is governed by a structured linear system. We demonstrate that Koopman-UBM can be viewed as an implicit planner: given an initial condition, it computes the desired robot behavior with the resulting flow of visual features over the entire skill horizon. To enable reactivity, we introduce an online replanning strategy in which the model acts as its own runtime monitor that automatically triggers replanning when predicted and observed visual flow diverge. Across seven simulated and two real-world tasks, we demonstrate that K-UBM matches or exceeds the performance of SOTA baselines, while offering faster inference, smooth execution, robustness to occlusions, and flexible replanning.

Related papers

• Structure-Aware Graph Multi-Task Learning for Dynamic Sparse OD Demand Prediction [1.7101795903804942]
  Origin-Destination (OD) demand prediction is fundamental to intelligent transportation systems.<n>Real-world OD flows are often dynamically sparse, long-tailed, and characterized by heterogeneous zero-flow patterns.<n>We propose SAGMTL, a Structure-Aware Graph Multi-Task Learning framework for dynamic sparse OD demand prediction.
  arXiv  Detail & Related papers  (2026-06-19T01:17:48Z)
• AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding [55.535374902204296]
  AffordanceVLA is a unified framework that introduces structured affordance forecasting as a task-oriented intermediate representation.<n>We show that AffordanceVLA achieves strong performance across diverse manipulation scenarios.
  arXiv  Detail & Related papers  (2026-06-04T13:28:51Z)
• Afford-VLA: Action-Aligned Visual Planning via Internalized Affordance [108.46436073194546]
  We argue that effective planning should be local, visually grounded, internally generated, and directly aligned with action.<n>We propose Afford-VLA, a unified framework that internalizes task-conditioned affordance as an explicit visual planning interface within VLA models.
  arXiv  Detail & Related papers  (2026-05-22T20:43:47Z)
• Learning from Imperfect Demonstrations via Temporal Behavior Tree-Guided Trajectory Repair [5.4872967954342675]
  Learning robot control policies from demonstrations is a powerful paradigm, yet real-world data is often suboptimal, noisy, or otherwise imperfect.<n>We present a formal framework that leverages Temporal Behavior Trees (TBT) to repair suboptimal trajectories prior to their use in downstream policy learning.<n>We demonstrate the effectiveness of this framework on discrete grid-world navigation and continuous single and multi-agent reach-avoid tasks.
  arXiv  Detail & Related papers  (2026-04-05T18:49:45Z)
• FutureVLA: Joint Visuomotor Prediction for Vision-Language-Action Model [73.03346643967309]
  We argue that effective joint visuomotor predictive modeling requires both temporal continuity and visually-conditioned supervision decoupling.<n>FutureVLA is designed to extract joint visuomotor embeddings by first decoupling visual and motor information.<n>In the post-training stage, we employ a latent embeddings alignment strategy, enabling diverse downstream VLA models to internalize these temporal priors.
  arXiv  Detail & Related papers  (2026-03-11T12:39:55Z)
• AR-VLA: True Autoregressive Action Expert for Vision-Language-Action Models [36.00004339916959]
  We propose a standalone autoregressive (AR) Action Expert that generates actions as a continuous causal sequence.<n>Our work introduces a scalable, context-aware action generation schema that provides a robust structural foundation for training effective robotic policies.
  arXiv  Detail & Related papers  (2026-03-10T18:03:29Z)
• Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis [34.5994686982342]
  Strong semantic representations improve the convergence and generation quality of diffusion and flow models.<n>Existing approaches largely rely on external models, which require separate training, operate on misaligned objectives, and exhibit unexpected scaling behavior.<n>We introduce Self-Flow: a self-supervised flow matching paradigm that integrates representation learning within the generative framework.
  arXiv  Detail & Related papers  (2026-03-06T17:41:49Z)
• \textsc{NaVIDA}: Vision-Language Navigation with Inverse Dynamics Augmentation [50.027425808733994]
  textscNaVIDA is a unified VLN framework that couples policy learning with action-grounded visual dynamics and adaptive execution.<n>textscNaVIDA augments training with chunk-based inverse-dynamics supervision to learn causal relationship between visual changes and corresponding actions.<n>Experiments show that textscNaVIDA achieves superior navigation performance compared to state-of-the-art methods with fewer parameters.
  arXiv  Detail & Related papers  (2026-01-26T06:16:17Z)
• StaMo: Unsupervised Learning of Generalizable Robot Motion from Compact State Representation [56.996371714721995]
  We propose an unsupervised approach that learns a highly compressed two-token state representation.<n>Our representation is efficient, interpretable, and integrates seamlessly into existing VLA-based models.<n>We name our method StaMo for its ability to learn generalizable robotic Motion from compact State representation.
  arXiv  Detail & Related papers  (2025-10-06T17:37:24Z)
• Foundation Model for Skeleton-Based Human Action Understanding [56.89025287217221]
  This paper presents a Unified Skeleton-based Dense Representation Learning framework.<n>USDRL consists of a Transformer-based Dense Spatio-Temporal (DSTE), Multi-Grained Feature Decorrelation (MG-FD), and Multi-Perspective Consistency Training (MPCT)
  arXiv  Detail & Related papers  (2025-08-18T02:42:16Z)
• VITA: Vision-to-Action Flow Matching Policy [27.783132918359545]
  VITA is a noise-free and conditioning-free policy learning framework.<n>It maps visual representations to latent actions using flow matching.<n>We evaluate VITA on 8 simulation and 2 real-world tasks from ALOHA and Robomimic.
  arXiv  Detail & Related papers  (2025-07-17T15:41:57Z)
• KPFlow: An Operator Perspective on Dynamic Collapse Under Gradient Descent Training of Recurrent Networks [9.512147747894026]
  We show how a gradient flow can be decomposed into a product that involves two operators.<n>We show how their interplay gives rise to low-dimensional latent dynamics under GD.<n>For multi-task training, we show that the operators can be used to measure how objectives relevant to individual sub-tasks align.
  arXiv  Detail & Related papers  (2025-07-08T20:33:15Z)
• Active Sensing with Predictive Coding and Uncertainty Minimization [0.0]
  We present an end-to-end procedure for embodied exploration inspired by two biological computations. We first demonstrate our approach in a maze navigation task and show that it can discover the underlying transition distributions and spatial features of the environment. We show that our model builds unsupervised representations through exploration that allow it to efficiently categorize visual scenes.
  arXiv  Detail & Related papers  (2023-07-02T21:14:49Z)
• Continual Visual Reinforcement Learning with A Life-Long World Model [55.05017177980985]
  We present a new continual learning approach for visual dynamics modeling.<n>We first introduce the life-long world model, which learns task-specific latent dynamics.<n>Then, we address the value estimation challenge for previous tasks with the exploratory-conservative behavior learning approach.
  arXiv  Detail & Related papers  (2023-03-12T05:08:03Z)
• Physics-Inspired Temporal Learning of Quadrotor Dynamics for Accurate Model Predictive Trajectory Tracking [76.27433308688592]
  Accurately modeling quadrotor's system dynamics is critical for guaranteeing agile, safe, and stable navigation. We present a novel Physics-Inspired Temporal Convolutional Network (PI-TCN) approach to learning quadrotor's system dynamics purely from robot experience. Our approach combines the expressive power of sparse temporal convolutions and dense feed-forward connections to make accurate system predictions.
  arXiv  Detail & Related papers  (2022-06-07T13:51:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.