BFMTrack: Latent Sequence Optimization for Physics-Based Motion Tracking with Behavioral Foundation Models

• URL: http://arxiv.org/abs/2606.25056v1
• Date: Tue, 23 Jun 2026 18:12:22 GMT
• Title: BFMTrack: Latent Sequence Optimization for Physics-Based Motion Tracking with Behavioral Foundation Models
• Abstract summary: Behavioral Foundation Models (BFMs) organize a rich repertoire of physically plausible behaviors into a latent space, guided by a large-scale motion dataset.<n>These models excel at time-invariant tasks, such as goal-reaching and state-based reward optimization, but their latent space does not directly support time-varying objectives, such as tracking a motion sequence.<n>We propose a novel Latent Sequence Optimization (LSO) to address these shortcomings.
• Score: 7.959692704349906
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Behavioral Foundation Models (BFMs) offer a promising path toward universal physics-based character control by organizing a rich repertoire of physically plausible behaviors into a latent space, guided by a large-scale motion dataset. While these models excel at time-invariant tasks, such as goal-reaching and state-based reward optimization, their latent space does not directly support time-varying objectives, such as tracking a motion sequence. For tracking, existing heuristics rely on moving-window-averaging that fails to capture the nuances of highly dynamic motions. In this work, we propose a novel Latent Sequence Optimization (LSO) to address these shortcomings. Our approach combines simulation rollouts with a policy gradient update to optimize over a sequence of latents, extending the capabilities of BFMs toward precise motion tracking without requiring reward engineering and tuning. To guide the optimization toward smooth, coherent latent trajectories, we model the latent sequence using temporally correlated noise. We validate our approach across dense tracking, sparse keyframing, and direct deployment onto a real humanoid robot.

Related papers

• GoStop: Reinforcement Learning for Adaptive Temporal Aggregation in Event-Based Feature Tracking [71.9852231453445]
  Event cameras enable low-latency and motion-robust perception, making them well-suited for feature tracking under fast and non-linear motion.<n>Existing event-based feature tracking methods rely on fixed rules based on hand-tuning for event accumulation.<n>We introduce reinforcement learning framework to adaptively control the accumulation process for online event-based feature tracking.
  arXiv  Detail & Related papers  (2026-07-17T07:18:07Z)
• Model-Based Diffusion Optimal Control for Multi-Robot Motion Planning [5.5818653283217055]
  Multi-Robot Motion Planning in continuous environments is challenging due to the growth of the joint trajectory space and the difficulty of enforcing dynamic feasibility and hard safety constraints.<n>Recent approaches recast trajectory planning as probabilistic inference, sampling from a posterior over trajectories using diffusion models whose score functions are learned from demonstration data.<n>We introduce Model-Based Diffusion Optimal Control (MDOC), a model-based diffusion planner that efficiently produces dynamically feasible trajectories without relying on data.
  arXiv  Detail & Related papers  (2026-07-14T06:54:03Z)
• Hypergraph-State Collaborative Reasoning for Multi-Object Tracking [63.32950991964095]
  We propose a collaborative reasoning framework that enhances motion estimation through joint inference among multiple correlated objects.<n>By allowing objects with similar motion states to mutually constrain and refine each other, our framework stabilizes noisy trajectories and infers plausible motion continuity even when target is occluded.
  arXiv  Detail & Related papers  (2026-04-14T12:37:39Z)
• HAD: Combining Hierarchical Diffusion with Metric-Decoupled RL for End-to-End Driving [51.268878540511054]
  We propose HAD, an end-to-end planning framework with a Hierarchical Diffusion Policy.<n>We show that HAD achieves new state-of-the-art performance on both NAVSIM and HUGSIM.
  arXiv  Detail & Related papers  (2026-04-04T04:12:47Z)
• Optimizing Control-Friendly Trajectories with Self-Supervised Residual Learning [10.551597725130675]
  Real-world physics can only be analytically modeled with a certain level of precision for modern intricate robotic systems.<n>This paper presents a self-supervised residual learning and trajectory optimization framework to address the aforementioned challenges.
  arXiv  Detail & Related papers  (2026-01-06T06:01:58Z)
• Planning with Sketch-Guided Verification for Physics-Aware Video Generation [71.29706409814324]
  We propose SketchVerify as a training-free, sketch-verification-based planning framework for video generation.<n>Our method predicts multiple candidate motion plans and ranks them using a vision-language verifier.<n>We iteratively refine the motion plan until a satisfactory one is identified, which is then passed to the trajectory-conditioned generator for final synthesis.
  arXiv  Detail & Related papers  (2025-11-21T17:48:02Z)
• Autoregressive End-to-End Planning with Time-Invariant Spatial Alignment and Multi-Objective Policy Refinement [15.002921311530374]
  Autoregressive models are a formidable baseline for end-to-end planning in autonomous driving.<n>Their performance is constrained by atemporal misalignment, as the planner must condition future actions on past sensory data.<n>We propose a Time-Invariant Alignment (TISA) module that learns to project initial environmental features into a consistent ego-centric frame.<n>We also introduce a multi-objective post-training stage using Direct Preference Optimization (DPO) to move beyond pure imitation.
  arXiv  Detail & Related papers  (2025-09-25T09:24:45Z)
• KoopMotion: Learning Almost Divergence Free Koopman Flow Fields for Motion Planning [10.112572107877076]
  We propose a flow field-based motion planning method that drives a robot from any initial state to a desired reference trajectory.<n>KoopMotion represents motion flow fields as dynamical systems, parameterized by Koopman Operators to mimic desired trajectories.<n>Our approach is highly sample efficient in both space and time, requiring only 3% of the LASA dataset to generate dense motion plans.
  arXiv  Detail & Related papers  (2025-09-11T00:42:01Z)
• Differentiable Motion Manifold Primitives for Reactive Motion Generation under Kinodynamic Constraints [5.982922468400902]
  Differentiable Motion Manifold Primitives (DMMP) is a novel neural network architecture that encodes and generates continuous-time, differentiable trajectories.<n> Experiments on dynamic throwing with a 7-DoF robot arm demonstrate that DMMP outperforms prior methods in planning speed, task success, and constraint satisfaction.
  arXiv  Detail & Related papers  (2024-10-16T03:29:33Z)
• FLD: Fourier Latent Dynamics for Structured Motion Representation and Learning [19.491968038335944]
  We introduce a self-supervised, structured representation and generation method that extracts spatial-temporal relationships in periodic or quasi-periodic motions. Our work opens new possibilities for future advancements in general motion representation and learning algorithms.
  arXiv  Detail & Related papers  (2024-02-21T13:59:21Z)
• MotionTrack: Learning Motion Predictor for Multiple Object Tracking [68.68339102749358]
  We introduce a novel motion-based tracker, MotionTrack, centered around a learnable motion predictor. Our experimental results demonstrate that MotionTrack yields state-of-the-art performance on datasets such as Dancetrack and SportsMOT.
  arXiv  Detail & Related papers  (2023-06-05T04:24:11Z)
• Motion Transformer with Global Intention Localization and Local Movement Refinement [103.75625476231401]
  Motion TRansformer (MTR) models motion prediction as the joint optimization of global intention localization and local movement refinement. MTR achieves state-of-the-art performance on both the marginal and joint motion prediction challenges.
  arXiv  Detail & Related papers  (2022-09-27T16:23:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.