Event-Driven Proactive Assistive Manipulation with Grounded Vision-Language Planning

• URL: http://arxiv.org/abs/2603.23950v1
• Date: Wed, 25 Mar 2026 05:14:04 GMT
• Title: Event-Driven Proactive Assistive Manipulation with Grounded Vision-Language Planning
• Abstract summary: We introduce a shift from request-driven assistance to event-driven proactive assistance.<n>We propose an event-driven framework that tracks interaction progress with an event monitor.<n>We evaluate the framework on a real tabletop number-block collaboration task.
• Score: 14.022917047438375
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Assistance in collaborative manipulation is often initiated by user instructions, making high-level reasoning request-driven. In fluent human teamwork, however, partners often infer the next helpful step from the observed outcome of an action rather than waiting for instructions. Motivated by this, we introduce a shift from request-driven assistance to event-driven proactive assistance, where robot actions are initiated by workspace state transitions induced by human--object interactions rather than user-provided task instructions. To this end, we propose an event-driven framework that tracks interaction progress with an event monitor and, upon event completion, extracts stabilized pre/post snapshots that characterize the resulting state transition. Given the stabilized snapshots, the planner analyzes the implied state transition to infer a task-level goal and decide whether to intervene; if so, it generates a sequence of assistive actions. To make outputs executable and verifiable, we restrict actions to a set of action primitives and reference objects via integer IDs. We evaluate the framework on a real tabletop number-block collaboration task, demonstrating that explicit pre/post state-change evidence improves proactive completion on solvable scenes and appropriate waiting on unsolvable ones.

Related papers

• Towards Neuro-Symbolic Procedural Reasoning for Long-Horizon Vision-Language-Action Manipulation [12.461719086222788]
  Vision-language-action (VLA) models can execute short manipulation skills, but remain brittle in long-horizon procedures.<n>We investigate a neuro-symbolic framework that combines learned VLA control with explicit task graphs and multimodal procedural memory.
  arXiv  Detail & Related papers  (2026-09-04T17:14:50Z)
• Intent-Driven Situation Tracking for User-Centric Multi-Turn Agents [4.138547191663224]
  We propose Intent-Driven Situation States (IDSS), a training-free framework that maintains an explicit situation state alongside the dialogue.<n>IDSS parses tool returns into provenance-aware entities and attributes, tracks user intents, required variables, constraints, and execution status, and propagates new facts to task constraints to update action executability.
  arXiv  Detail & Related papers  (2026-08-16T14:14:07Z)
• AutoIntervene: Calibrated Intervention for Action-Chunking Imitation Learning Policies [4.786936073180585]
  Action-chunking visuomotor policies learn from demonstrations and improve temporal consistency.<n>We present AutoIntervene, an online framework that selectively transfers control between an action-chunking policy and an operator during deployment.
  arXiv  Detail & Related papers  (2026-08-07T10:14:29Z)
• Melding LLM and temporal logic for reliable human-swarm collaboration in complex scenarios [17.56189537207367]
  We introduce a neuro-symbolic framework for long-horizon human-swarm collaboration.<n>We formalize mission goals and operational rules as temporal logic formulas and admissible task orderings as task automata.<n>An event-triggered interaction protocol limits operator involvement to sparse, high-level confirmation and guidance.
  arXiv  Detail & Related papers  (2026-05-08T15:32:14Z)
• Learning Semantic-Geometric Task Graph-Representations from Human Demonstrations [16.68801520494275]
  We introduce a semantic-geometric task graph-representation that encodes object identities, inter-object relations, and their temporal geometric evolution from human demonstrations.<n>We show that semantic-geometric task graph-representations are particularly beneficial for tasks with high action and object variability.
  arXiv  Detail & Related papers  (2026-01-16T17:35:00Z)
• AnyPos: Automated Task-Agnostic Actions for Bimanual Manipulation [24.199522837278128]
  We present a new notion of task-agnostic action paradigm that decouples action execution from task-specific conditioning.<n>ATARA is a scalable self-supervised framework that accelerates collection by over $ 30times $ compared to human teleoperation.<n>We propose AnyPos, an inverse dynamics model equipped with Arm-Decoupled Estimation and a Direction-Aware Decoder.
  arXiv  Detail & Related papers  (2025-07-17T03:48:57Z)
• Towards Bridging the Gap between High-Level Reasoning and Execution on Robots [2.6107298043931206]
  When reasoning about actions, e.g., by means of task planning or agent programming with Golog, the robot's actions are typically modeled on an abstract level. However, when executing such an action on a robot it can no longer be seen as a primitive. In this thesis, we propose several approaches towards closing this gap.
  arXiv  Detail & Related papers  (2023-12-30T12:26:12Z)
• Unified Task and Motion Planning using Object-centric Abstractions of Motion Constraints [56.283944756315066]
  We propose an alternative TAMP approach that unifies task and motion planning into a single search. Our approach is based on an object-centric abstraction of motion constraints that permits leveraging the computational efficiency of off-the-shelf AI search to yield physically feasible plans.
  arXiv  Detail & Related papers  (2023-12-29T14:00:20Z)
• ThinkBot: Embodied Instruction Following with Thought Chain Reasoning [66.09880459084901]
  Embodied Instruction Following (EIF) requires agents to complete human instruction by interacting objects in complicated surrounding environments. We propose ThinkBot that reasons the thought chain in human instruction to recover the missing action descriptions. Our ThinkBot outperforms the state-of-the-art EIF methods by a sizable margin in both success rate and execution efficiency.
  arXiv  Detail & Related papers  (2023-12-12T08:30:09Z)
• Code Models are Zero-shot Precondition Reasoners [83.8561159080672]
  We use code representations to reason about action preconditions for sequential decision making tasks. We propose a precondition-aware action sampling strategy that ensures actions predicted by a policy are consistent with preconditions.
  arXiv  Detail & Related papers  (2023-11-16T06:19:27Z)
• Proactive Human-Robot Interaction using Visuo-Lingual Transformers [0.0]
  Humans possess the innate ability to extract latent visuo-lingual cues to infer context through human interaction. We propose a learning-based method that uses visual cues from the scene, lingual commands from a user and knowledge of prior object-object interaction to identify and proactively predict the underlying goal the user intends to achieve.
  arXiv  Detail & Related papers  (2023-10-04T00:50:21Z)
• Event-Guided Procedure Planning from Instructional Videos with Text Supervision [31.82121743586165]
  We focus on the task of procedure planning from instructional videos with text supervision. A critical challenge of this task is the large semantic gap between observed visual states and unobserved intermediate actions. We propose a novel event-guided paradigm, which first infers events from the observed states and then plans out actions based on both the states and predicted events.
  arXiv  Detail & Related papers  (2023-08-17T09:43:28Z)
• A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution [54.385344986265714]
  We propose a persistent spatial semantic representation method to bridge the gap between language and robot actions. We evaluate our approach on the ALFRED benchmark and achieve state-of-the-art results, despite completely avoiding the commonly used step-by-step instructions.
  arXiv  Detail & Related papers  (2021-07-12T17:47:19Z)
• Instance-Aware Predictive Navigation in Multi-Agent Environments [93.15055834395304]
  We propose an Instance-Aware Predictive Control (IPC) approach, which forecasts interactions between agents as well as future scene structures. We adopt a novel multi-instance event prediction module to estimate the possible interaction among agents in the ego-centric view. We design a sequential action sampling strategy to better leverage predicted states on both scene-level and instance-level.
  arXiv  Detail & Related papers  (2021-01-14T22:21:25Z)
• Inferring Temporal Compositions of Actions Using Probabilistic Automata [61.09176771931052]
  We propose to express temporal compositions of actions as semantic regular expressions and derive an inference framework using probabilistic automata. Our approach is different from existing works that either predict long-range complex activities as unordered sets of atomic actions, or retrieve videos using natural language sentences.
  arXiv  Detail & Related papers  (2020-04-28T00:15:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.