TEMPO: Learning Temporal Context for Dynamic Robot Manipulation

• URL: http://arxiv.org/abs/2609.16864v1
• Date: Tue, 15 Sep 2026 08:56:52 GMT
• Title: TEMPO: Learning Temporal Context for Dynamic Robot Manipulation
• Abstract summary: We propose TEMPO, which augments a pretrained VLA with two temporal inputs to resolve motion ambiguity and state aliasing.<n>It improves Bottle Handover success from 44% to 74% and is the only method that solves state aliasing.<n>We further release TEMPO-Bench, a benchmark of over 50k annotated frames for evaluating motion-aware robot perception.
• Score: 12.057683796484232
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Vision-language-action (VLA) models have achieved impressive performance in quasi-static manipulation, but struggle in dynamic manipulation tasks because they operate on a single observation at inference time. We identify two representational failures that underlie this limitation. The first is motion ambiguity, where a single observation does not include scene dynamics and therefore cannot anticipate the future state of moving objects. The second is state aliasing, where visually similar observations from different points in a task require different actions. We argue that these failures persist regardless of model scale and inference latency, showing that the bottleneck is missing temporal context rather than model capacity. Based on this insight, we propose TEMPO, which augments a pretrained VLA with two temporal inputs: a motion summary extracted from a frozen video foundation model to resolve motion ambiguity and a compact proprioceptive history to resolve state aliasing. TEMPO requires no modification to the backbone and adds minimal compute overhead at training or deployment. Across four dynamic manipulation tasks, it improves Bottle Handover success from 44% to 74% and is the only method that solves state aliasing. Probing and ablation studies confirm that each temporal signal independently addresses its corresponding failure. We further release TEMPO-Bench, a benchmark of over 50k annotated frames for evaluating motion-aware robot perception in both regression and multiple-choice formats. Project Website: https://tempo-robot.github.io/

Related papers

• MotionVLA: Injecting Geometric Motion into Vision-Language-Action Model [41.634737117765745]
  We introduce MotionVLA, a motion-history interface that converts a short past-only video window into compact, time-continuous trajectory-field tokens.<n>We show that MotionVLA improves long-horizon manipulation while producing smoother and more direct executions.
  arXiv  Detail & Related papers  (2026-06-06T18:20:02Z)
• Improving Viewpoint-Invariance and Temporal Consistency for Action Detection [2.0019258650495493]
  This paper introduces a novel two-stage action detection approach.<n>In the first stage, we extract motion features from augmented virtual viewpoints.<n>The second stage introduces a view-invariant, multi-scale temporal encoder.
  arXiv  Detail & Related papers  (2026-05-21T16:38:21Z)
• ST-$π$: Structured SpatioTemporal VLA for Robotic Manipulation [24.44733656837927]
  We propose a structured structuredtemporal V model for robotic manipulation.<n>Our model is guided by two key designs: 1) Stemporal VLM and 2) Stemporal action expert.<n>In addition, we propose a real-world robotic dataset with annotations for fine-tuning.
  arXiv  Detail & Related papers  (2026-04-20T06:48:47Z)
• OFlow: Injecting Object-Aware Temporal Flow Matching for Robust Robotic Manipulation [55.56490813640669]
  VLA models typically act only on the current frame, while future prediction and object-aware reasoning are often learned in separate latent spaces.<n>We propose OFlow, a framework that addresses both limitations by unifying temporal foresight and object-aware reasoning in a shared semantic latent space.<n>Our method forecasts future latents with temporal flow matching, factorizes them into object-aware representations that emphasize physically relevant cues.
  arXiv  Detail & Related papers  (2026-04-20T06:38:01Z)
• Robotic VLA Benefits from Joint Learning with Motion Image Diffusion [114.60268819583017]
  Vision-Language-Action (VLA) models have achieved remarkable progress in robotic manipulation by mapping multimodal observations and instructions directly to actions.<n>We propose joint learning with motion image diffusion, a novel strategy that enhances VLA models with motion reasoning capabilities.<n> Experiments in both simulation and real-world environments demonstrate that joint learning with motion image diffusion improves the success rate of pi-series VLAs to 97.5%.
  arXiv  Detail & Related papers  (2025-12-19T19:07:53Z)
• Splannequin: Freezing Monocular Mannequin-Challenge Footage with Dual-Detection Splatting [28.641196407703315]
  We create high-fidelity frozen 3D scenes from monocular Mannequin-Challenge (MC) videos.<n>Instead of focusing on modeling motion, our goal is to create a frozen scene while preserving subtle dynamics.<n>Our method integrates into existing dynamic Gaussian pipelines via simple loss terms, requires no architectural changes, and adds zero inference overhead.
  arXiv  Detail & Related papers  (2025-12-04T18:59:53Z)
• SAM2Grasp: Resolve Multi-modal Grasping via Prompt-conditioned Temporal Action Prediction [7.708279811172532]
  Imitation learning for robotic grasping is often plagued by the multimodal problem.<n>Standard imitation learning policies fail by averaging these distinct actions into a single, invalid action.<n>We introduce SAM2Grasp, a novel framework that reformulating the task as a uni-modal, prompt-conditioned prediction problem.
  arXiv  Detail & Related papers  (2025-12-02T10:15:00Z)
• TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models [75.42002690128486]
  TemporalBench is a new benchmark dedicated to evaluating fine-grained temporal understanding in videos. It consists of 10K video question-answer pairs, derived from 2K high-quality human annotations detailing the temporal dynamics in video clips. Results show that state-of-the-art models like GPT-4o achieve only 38.5% question answering accuracy on TemporalBench.
  arXiv  Detail & Related papers  (2024-10-14T17:59:58Z)
• Bidirectional Decoding: Improving Action Chunking via Guided Test-Time Sampling [51.38330727868982]
  We show how action chunking impacts the divergence between a learner and a demonstrator.<n>We propose Bidirectional Decoding (BID), a test-time inference algorithm that bridges action chunking with closed-loop adaptation.<n>Our method boosts the performance of two state-of-the-art generative policies across seven simulation benchmarks and two real-world tasks.
  arXiv  Detail & Related papers  (2024-08-30T15:39:34Z)
• Motion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and Tracking [52.393359791978035]
  Motion2VecSets is a 4D diffusion model for dynamic surface reconstruction from point cloud sequences. We parameterize 4D dynamics with latent sets instead of using global latent codes. For more temporally-coherent object tracking, we synchronously denoise deformation latent sets and exchange information across multiple frames.
  arXiv  Detail & Related papers  (2024-01-12T15:05:08Z)
• Exploring Motion and Appearance Information for Temporal Sentence Grounding [52.01687915910648]
  We propose a Motion-Appearance Reasoning Network (MARN) to solve temporal sentence grounding. We develop separate motion and appearance branches to learn motion-guided and appearance-guided object relations. Our proposed MARN significantly outperforms previous state-of-the-art methods by a large margin.
  arXiv  Detail & Related papers  (2022-01-03T02:44:18Z)
• Learning to Shift Attention for Motion Generation [55.61994201686024]
  One challenge of motion generation using robot learning from demonstration techniques is that human demonstrations follow a distribution with multiple modes for one task query. Previous approaches fail to capture all modes or tend to average modes of the demonstrations and thus generate invalid trajectories. We propose a motion generation model with extrapolation ability to overcome this problem.
  arXiv  Detail & Related papers  (2021-02-24T09:07:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.