Understanding and Mitigating the Video-Action Generalization Gap via Temporal Ratio

• URL: http://arxiv.org/abs/2607.08127v1
• Date: Thu, 09 Jul 2026 05:56:08 GMT
• Title: Understanding and Mitigating the Video-Action Generalization Gap via Temporal Ratio
• Abstract summary: Generative video foundation models exhibit strong compositional priors.<n>World-action models (WAMs) and video-action models (VAMs) often lose these priors after finetuning on robotic action data.<n>We propose an inference-time adaptive guidance method, which exploits this intrinsic feature attention pattern.
• Score: 42.378085641934796
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Generative video foundation models exhibit strong compositional priors, yet world-action models (WAMs) and video-action models (VAMs) often lose these priors after finetuning on robotic action data. We refer to this discrepancy as the video-action generalization gap. In this paper, we systematically investigate this gap by evaluating a comprehensive design space of VAMs, demonstrating that standard design choices yield no emergent explanation pattern. To explain this behavior, we introduce the Temporal Ratio (TR), an attention-based measure of how strongly the action head relies on future latent rollouts relative to the anchored current frame. TR has two key properties: first, a model's structural reliance on future-predictive latents, measured via TR, acts as a predictor of its compositional generalization capacity; second, it natively fluctuates based on task phase, shifting attention to future frames during planning and reverting to the present frame for precise manipulation. Finally, based on these findings, we propose an inference-time adaptive guidance method, which exploits this intrinsic feature attention pattern to dynamically amplify compositional video conditioning signals precisely when the policy relies on future rollouts. Evaluated on the LIBERO benchmark and real-world tasks, our approach mitigates the OOD-ID compositional generalization gap. More details: https://umishra.me/temporal-ratio/

Related papers

• Adaptive Multi-Granularity Temporal Modeling for Weakly Supervised Video Anomaly Detection [1.2072875728151518]
  weakly supervised video anomaly detection (WSVAD) has emerged as a critical research frontier.<n>Most existing approaches formulate WSVAD within a Multiple Instance Learning (MIL) framework that relies on rigid, hand-crafted temporal priors to supervise anomaly scoring.<n>We propose an adaptive temporal modeling framework for WSVAD that explicitly accounts for variations in video dynamics across multiple temporal granularities.
  arXiv  Detail & Related papers  (2026-09-04T12:27:43Z)
• COMET: Contrastive Motion-Enhanced Temporal Reasoning for Video Multimodal Large Language Models [15.39037309359101]
  Video multimodal large language models have advanced significantly, yet fine-grained motion-temporal understanding remains fragile.<n>We propose COMET, a temporally grounded framework that systematically strengthens video MLLMs through explicit temporal representation.<n>For optimization, COMET combines temporal prior distillation with a forward-reverse TC-GRPO stage that turns temporal order into a direct learning signal.
  arXiv  Detail & Related papers  (2026-08-21T12:28:36Z)
• Native Video-Action Pretraining for Generalizable Robot Control [77.59831077077176]
  LingBot-VA 2.0 is a video-action foundation model built from the ground up for embodiment.<n>To meet the demands of high-frequency inference, our model employs a sparse MoE backbone.<n>Real-world deployment validates LingBot-VA 2.0 as a robust foundation model.
  arXiv  Detail & Related papers  (2026-07-09T16:15:43Z)
• What Makes Video World Model Latents Action-Relevant: Prediction over Reconstruction [9.020077150911526]
  We find that action-relevant structure is driven primarily by temporal video pretraining rather than pixel reconstruction fidelity.<n>Our results identify temporal predictive structure -- not reconstruction fidelity -- as the primary ingredient underlying action-relevant video representations.
  arXiv  Detail & Related papers  (2026-06-05T04:43:02Z)
• CaC: Advancing Video Reward Models via Hierarchical Spatiotemporal Concentrating [49.94171749437024]
  We propose Concentrate and Concentrate (CaC), a coarse-to-fine anomaly reward model based on Vision-Language Models.<n>We construct the first largescale generated video anomaly dataset with per-frame bounding-box annotations, temporal anomaly windows, and fine-grained attribution labels.<n>In experiments, CaC can stably concentrate on subtle anomalies, achieving a 25.7% accuracy improvement on fine-grained anomaly benchmarks.
  arXiv  Detail & Related papers  (2026-05-12T08:08:33Z)
• HarmoWAM: Harmonizing Generalizable and Precise Manipulation via Adaptive World Action Models [58.191567345416836]
  World Action Models (WAMs) have emerged as a promising paradigm for robot control by modeling physical dynamics.<n>HarmoWAM fully leverages a world model to unify predictive and reactive control, enabling both general transit and precise manipulation.<n>We construct three training-unseen test environments across six real-world robotic tasks, covering variations in background, position, and object semantics.
  arXiv  Detail & Related papers  (2026-05-11T17:59:56Z)
• NoiseGate: Learning Per-Latent Timestep Schedules as Information Gating in World Action Models [19.272356473995245]
  World Action Models (WAMs) are policies that tie robot action generation to future-observation modeling.<n>NoiseGate is a lightweight Gating Policy Network that emits per-latent time increments during denoising.<n>NoiseGate delivers consistent gains on diverse RoboTwin random-scene manipulation tasks.
  arXiv  Detail & Related papers  (2026-05-08T14:31:52Z)
• FutureVLA: Joint Visuomotor Prediction for Vision-Language-Action Model [73.03346643967309]
  We argue that effective joint visuomotor predictive modeling requires both temporal continuity and visually-conditioned supervision decoupling.<n>FutureVLA is designed to extract joint visuomotor embeddings by first decoupling visual and motor information.<n>In the post-training stage, we employ a latent embeddings alignment strategy, enabling diverse downstream VLA models to internalize these temporal priors.
  arXiv  Detail & Related papers  (2026-03-11T12:39:55Z)
• Chain of World: World Model Thinking in Latent Motion [24.24061036481793]
  Vision-Language-Action (VLA) models often overlook the predictive and temporal-causal structure underlying visual dynamics.<n>We introduce CoWVLA, a new "Chain of World" paradigm that unifies world-model temporal reasoning with a disentangled latent motion representation.<n>CoWVLA outperforms existing world-model and latent-action approaches and achieves moderate computational efficiency.
  arXiv  Detail & Related papers  (2026-03-03T17:52:06Z)
• StepVAR: Structure-Texture Guided Pruning for Visual Autoregressive Models [98.72926158261937]
  We propose a training-free token pruning framework for Visual AutoRegressive models.<n>We employ a lightweight high-pass filter to capture local texture details, while leveraging Principal Component Analysis (PCA) to preserve global structural information.<n>To maintain valid next-scale prediction under sparse tokens, we introduce a nearest neighbor feature propagation strategy.
  arXiv  Detail & Related papers  (2026-03-02T11:35:05Z)
• Temporal Graph Pattern Machine [17.352525018007473]
  Temporal Graph Pattern Machine (TGPM) conceptualizes each interaction as an interaction patch synthesized via temporally-biased random walks.<n>TGPM consistently achieves state-of-the-art performance in both transductive and inductive link prediction.
  arXiv  Detail & Related papers  (2026-01-30T01:46:13Z)
• MAUCell: An Adaptive Multi-Attention Framework for Video Frame Prediction [0.0]
  We introduce the Multi-Attention Unit (MAUCell) which combines Generative Adrative Networks (GANs) and attention mechanisms to improve video prediction.<n>The new design system maintains equilibrium between temporal continuity and spatial accuracy to deliver reliable video prediction.
  arXiv  Detail & Related papers  (2025-01-28T14:52:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.