Action- and Language-Conditioned Video Assessment for Embodied Control

• URL: http://arxiv.org/abs/2608.08273v1
• Date: Sat, 08 Aug 2026 18:05:21 GMT
• Title: Action- and Language-Conditioned Video Assessment for Embodied Control
• Abstract summary: Vision-based embodied agents executing natural language instructions require feedback mechanisms that assess task progress over complete trajectories.<n>We propose ALVA (Action- and Language-Conditioned Video Assessment), a trajectory evaluator that conditions its assessment on visual observations, the executed action sequence, and the natural language instruction.<n>In simulated 3D household environments, ALVA exhibits a conservative assessment pattern with near-zero false-positive rates.
• Score: 25.788621525752006
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-based embodied agents executing multi-step natural language instructions require feedback mechanisms that assess task progress over complete trajectories. Conventional approaches based on final-frame matching or continuous embedding similarity may overlook intermediate transitions that are necessary for determining whether an instruction has been completed. We propose ALVA (Action- and Language-Conditioned Video Assessment), a trajectory evaluator that conditions its assessment on visual observations, the executed action sequence, and the natural language instruction. The method uses a pre-trained vision-language model (VLM) in two stages: it first summarizes frame-to-frame visual transitions conditioned on the executed actions and then assesses the generated summary with respect to the instruction to produce a discrete trajectory-level progress score. In simulated 3D household environments, ALVA exhibits a conservative assessment pattern with near-zero false-positive rates. When used as terminal feedback for closed-loop policy optimization, it provides more effective feedback than the evaluated static image and embedding-based visual baselines and reduces the performance gap to a ground-truth oracle. These results support action- and language-conditioned video assessment as an interpretable feedback mechanism for the evaluated simulated embodied-control tasks.

Related papers

• VLA-Corrector: Stage-Aware Observable State Understanding for Prompt-Based Closed-Loop Recovery of Vision-Language-Action Policies [12.698552476812855]
  Long-horizon robot manipulation with Vision-Language-Action (VLA) policies remains vulnerable to execution-time deviations.<n>We introduce a stage-aware failure verification and Prompt Recovery framework that enables closed-loop correction of a fixed VLA policy.
  arXiv  Detail & Related papers  (2026-09-06T09:50:30Z)
• FutureRTC: Real-Time Robot Execution with Anticipatory-Conditioned Action Chunking [51.97839311685636]
  FutureRTC is a plug-and-play adaptation framework that predicts execution-time observations and states for asynchronous VLA control without modifying the underlying policy.<n>We show that FutureRTC achieves robustness to inference delays, resulting in smoother trajectories, faster execution, and consistently higher task success rates.
  arXiv  Detail & Related papers  (2026-07-27T05:10:16Z)
• Action with Visual Primitives [36.230909065494345]
  Vision-Language-Action (VLA) models have emerged as a promising paradigm for generalist robotic manipulation.<n>We introduce AVP (Action with Visual Primitives), an end-to-end architecture that implements this visual-primitive-centric interface.<n>We show that AVP improves the success rate by 27.61% over pi_0.5 and outperforms other recent methods.
  arXiv  Detail & Related papers  (2026-05-21T08:52:47Z)
• Interactive Evaluation Requires a Design Science [90.62695599188204]
  Large language models (LLMs) are increasingly deployed as systems that act over time through tools, environments, users, and other agents.<n>The field has begun to build interactive benchmarks, but the resulting landscape is fragmented.<n>This position paper argues that interactive evaluation should be treated as a principled evaluation paradigm.
  arXiv  Detail & Related papers  (2026-05-18T04:03:18Z)
• AsgardBench -- Evaluating Visually Grounded Interactive Planning Under Minimal Feedback [28.254110943200683]
  AsgardBench aims to evaluate visually grounded, high-level action sequence generation and interactive planning.<n>Our evaluations show that performance drops sharply without visual input, revealing weaknesses in visual grounding and state tracking.<n>Our benchmark zeroes in on a narrower question: can a model actually use what it sees to adapt a plan when things do not go as expected?
  arXiv  Detail & Related papers  (2026-03-16T20:31:43Z)
• See, Plan, Rewind: Progress-Aware Vision-Language-Action Models for Robust Robotic Manipulation [59.07792608884117]
  We introduce See, Plan, Rewind (SPR), a progress-aware vision-language-action framework.<n>SPR operates through a continuous core cycle, Seeing the current state and upcoming milestone, Planning a trajectory towards the next 2D waypoint, and Rewinding to a recoverable state upon failure.<n>SPR achieves state-of-the-art robustness with the smallest performance drop, surpassing OpenVLA-OFT and UniVLA.
  arXiv  Detail & Related papers  (2026-03-10T07:22:51Z)
• AC^2-VLA: Action-Context-Aware Adaptive Computation in Vision-Language-Action Models for Efficient Robotic Manipulation [21.23747444669735]
  We propose Action-Context-aware Adaptive Computation for VLA models (AC2-VLA)<n>AC2-VLA adaptively performs cognition reuse across timesteps, token pruning, and selective execution of model components within a unified mechanism.<n>Experiments on robotic manipulation benchmarks show that AC2-VLA achieves up to a 1.79times speedup while reducing FLOPs to 29.4% of the dense baseline.
  arXiv  Detail & Related papers  (2026-01-27T14:10:39Z)
• ReViP: Reducing False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalance [50.05984919728878]
  We present ReViP, a novel VLA framework with Vision-Proprioception Rebalance to enhance visual grounding and robustness under perturbations.<n>Specifically, we use an external VLM as a task-stage observer to extract real-time task-centric visual cues from visual observations.<n>To evaluate false completion, we propose the first False-Completion Benchmark Suite built on LIBERO with controlled settings such as Object-Drop.
  arXiv  Detail & Related papers  (2026-01-23T11:31:07Z)
• Do What You Say: Steering Vision-Language-Action Models via Runtime Reasoning-Action Alignment Verification [17.948161564138033]
  Reasoning Vision Language Action (VLA) models improve robotic instruction-following by generating step-by-step textual plans before low-level actions.<n>But even with a correct textual plan, the generated actions can still miss the intended outcomes in the plan, especially in out-of-distribution scenarios.<n>We formalize this phenomenon as a lack of embodied CoT faithfulness, and introduce a training-free, runtime policy steering method for reasoning-action alignment.
  arXiv  Detail & Related papers  (2025-10-18T00:38:45Z)
• Adapting Vision-Language Models for Evaluating World Models [24.813041196394582]
  We present UNIVERSE, a method for adapting Vision-language Evaluator for Rollouts in Simulated Environments under data and compute constraints.<n>We conduct a large-scale study comparing full, partial, and parameter-efficient finetuning across task formats, context lengths, sampling strategies, and data compositions.<n>The resulting unified evaluator matches the performance of task-specific baselines using a single checkpoint.
  arXiv  Detail & Related papers  (2025-06-22T09:53:28Z)
• VITA: Zero-Shot Value Functions via Test-Time Adaptation of Vision-Language Models [49.78447737655287]
  VITA is a zero-shot value function learning method that enhances both capabilities via test-time adaptation.<n>We demonstrate that VITA's zero-shot value estimates can be utilized for reward shaping in offline reinforcement learning.
  arXiv  Detail & Related papers  (2025-06-11T18:05:33Z)
• End-to-End Evaluation for Low-Latency Simultaneous Speech Translation [55.525125193856084]
  We propose the first framework to perform and evaluate the various aspects of low-latency speech translation under realistic conditions.<n>This includes the segmentation of the audio as well as the run-time of the different components.<n>We also compare different approaches to low-latency speech translation using this framework.
  arXiv  Detail & Related papers  (2023-08-07T09:06:20Z)
• FineDiving: A Fine-grained Dataset for Procedure-aware Action Quality Assessment [93.09267863425492]
  We argue that understanding both high-level semantics and internal temporal structures of actions in competitive sports videos is the key to making predictions accurate and interpretable. We construct a new fine-grained dataset, called FineDiving, developed on diverse diving events with detailed annotations on action procedures.
  arXiv  Detail & Related papers  (2022-04-07T17:59:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.