FactCheck: Feasibility-aware Long-term Action Anticipation with Multi-agent Collaboration

• URL: http://arxiv.org/abs/2606.14778v1
• Date: Wed, 10 Jun 2026 06:25:09 GMT
• Title: FactCheck: Feasibility-aware Long-term Action Anticipation with Multi-agent Collaboration
• Abstract summary: Long-term action anticipation aims to predict an ordered sequence of future verb-noun actions from a partially observed video.<n>We propose FactCheck, a novel multi-agent collaboration framework that improves feasibility through a closed-loop "Observe-Plan-Verify" mechanism.<n>Our work establishes a new paradigm for feasibility-aware long-term action anticipation, effectively closing the loop of action recognition, action prediction and action verification.
• Score: 29.089100270702613
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Long-term action anticipation (LTA) aims to predict an ordered sequence of future verb-noun actions from a partially observed video. While this task serves as the foundation for embodied intelligence, anticipating physically feasible long-term actions remains a critical challenge. Existing methods, which operate in an open-loop manner, often hallucinate non-existent objects, violate object affordances, or disregard object states, as they lack explicit mechanisms to verify action feasibility against the physical environment. To address this, we propose FactCheck, a novel multi-agent collaboration framework that improves feasibility through a closed-loop "Observe-Plan-Verify" mechanism. FactCheck decomposes the complex LTA task into specialized roles: an Observer that recognizes historical actions from video observations and constructs a dual-form structured memory, comprising a History Action Abstract that captures high-level human intentions and environmental status, and a History Action Graph that encodes object states and temporal dependencies; a Planner that generates draft future actions conditioned on both low-level historical actions and high-level History Action Abstract; and a Verifier that rigorously validates the draft against the History Action Graph and refines infeasible actions. Extensive experiments on the EPIC-Kitchens-55 and EGTEA Gaze+ benchmarks demonstrate that FactCheck consistently outperforms state-of-the-art methods. Our work establishes a new paradigm for feasibility-aware long-term action anticipation, effectively closing the loop of action recognition, action prediction and action verification.

Related papers

• Towards Neuro-Symbolic Procedural Reasoning for Long-Horizon Vision-Language-Action Manipulation [12.461719086222788]
  Vision-language-action (VLA) models can execute short manipulation skills, but remain brittle in long-horizon procedures.<n>We investigate a neuro-symbolic framework that combines learned VLA control with explicit task graphs and multimodal procedural memory.
  arXiv  Detail & Related papers  (2026-09-04T17:14:50Z)
• Revisiting Open-Loop Execution in Robotics: Toward Reactive, Higher-Performing Policies [21.50347781480984]
  Action chunking has emerged as a key enabler of recent progress in imitation learning for robotic manipulation.<n> executing long open-loop prefixes reduces reactivity, limiting policies' ability to correct for errors.<n>We argue that long open-loop execution primarily helps short-context policies imitate "non-Markovian demonstrations"
  arXiv  Detail & Related papers  (2026-08-16T21:49:07Z)
• VeriSpace: Spatially Grounded Action Verification for Vision-Language-Action Models [19.75611749501909]
  Vision-Ground-action (VLA) models have shown strong promise for robotic manipulation.<n>But their reliability at test time remains limited by one-shot action prediction.<n>We present VeriSpace, a 3D-aware verifier for test-time action selection in VLA systems.
  arXiv  Detail & Related papers  (2026-06-09T08:31:59Z)
• Coarse-to-Control: Action-Token Planning for Vision-Language-Action Models [101.44987067974999]
  Coarse-to-Control is a plan-execute VLA that introduces planning in the action-token space.<n>We show that action-token planning consistently improves over direct action generation.
  arXiv  Detail & Related papers  (2026-06-05T10:01:37Z)
• OFlow: Injecting Object-Aware Temporal Flow Matching for Robust Robotic Manipulation [55.56490813640669]
  VLA models typically act only on the current frame, while future prediction and object-aware reasoning are often learned in separate latent spaces.<n>We propose OFlow, a framework that addresses both limitations by unifying temporal foresight and object-aware reasoning in a shared semantic latent space.<n>Our method forecasts future latents with temporal flow matching, factorizes them into object-aware representations that emphasize physically relevant cues.
  arXiv  Detail & Related papers  (2026-04-20T06:38:01Z)
• See, Plan, Rewind: Progress-Aware Vision-Language-Action Models for Robust Robotic Manipulation [59.07792608884117]
  We introduce See, Plan, Rewind (SPR), a progress-aware vision-language-action framework.<n>SPR operates through a continuous core cycle, Seeing the current state and upcoming milestone, Planning a trajectory towards the next 2D waypoint, and Rewinding to a recoverable state upon failure.<n>SPR achieves state-of-the-art robustness with the smallest performance drop, surpassing OpenVLA-OFT and UniVLA.
  arXiv  Detail & Related papers  (2026-03-10T07:22:51Z)
• Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planning [97.29507133345766]
  We propose Fast-ThinkAct, an efficient reasoning framework that achieves compact yet performant planning through verbalizable latent reasoning.<n>Experiments across diverse embodied manipulation and reasoning benchmarks demonstrate that Fast-ThinkAct achieves strong performance with up to 89.3% reduced inference latency.
  arXiv  Detail & Related papers  (2026-01-14T18:59:59Z)
• BEAC: Imitating Complex Exploration and Task-oriented Behaviors for Invisible Object Nonprehensile Manipulation [9.453992660558256]
  This paper proposes a novel imitation learning framework called Belief Exploration-Action Cloning (BEAC)<n>Our proposed method achieved the best task performance, higher mode and action prediction accuracies, while reducing the cognitive load in a user study.
  arXiv  Detail & Related papers  (2025-03-21T02:26:14Z)
• Harnessing Temporal Causality for Advanced Temporal Action Detection [53.654457142657236]
  We introduce CausalTAD, which combines causal attention and causal Mamba to achieve state-of-the-art performance on benchmarks. We ranked 1st in the Action Recognition, Action Detection, and Audio-Based Interaction Detection tracks at the EPIC-Kitchens Challenge 2024, and 1st in the Moment Queries track at the Ego4D Challenge 2024.
  arXiv  Detail & Related papers  (2024-07-25T06:03:02Z)
• Task and Motion Planning for Execution in the Real [24.01204729304763]
  This work generates task and motion plans that include actions cannot be fully grounded at planning time. Execution combines offline planned motions and online behaviors till reaching the task goal. Forty real-robot trials and motivating demonstrations are performed to evaluate the proposed framework. Results show faster execution time, less number of actions, and more success in problems where diverse gaps arise.
  arXiv  Detail & Related papers  (2024-06-05T22:30:40Z)
• PALM: Predicting Actions through Language Models [74.10147822693791]
  We introduce PALM, an approach that tackles the task of long-term action anticipation. Our method incorporates an action recognition model to track previous action sequences and a vision-language model to articulate relevant environmental details. Our experimental results demonstrate that PALM surpasses the state-of-the-art methods in the task of long-term action anticipation.
  arXiv  Detail & Related papers  (2023-11-29T02:17:27Z)
• Expected Eligibility Traces [31.942254205913798]
  Expected traces allow to update states and actions that could have preceded the current state, even if they did not do so on this occasion. We provide a way to smoothly interpolate between instantaneous and expected traces by a mechanism similar to bootstrapping.
  arXiv  Detail & Related papers  (2020-07-03T17:46:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.