Plan, Watch, Recover: A Benchmark and Architectures for Proactive Procedural Assistance

• URL: http://arxiv.org/abs/2606.04970v1
• Date: Wed, 03 Jun 2026 14:52:03 GMT
• Title: Plan, Watch, Recover: A Benchmark and Architectures for Proactive Procedural Assistance
• Abstract summary: We release textbfEgoProactive, a wearable-egocentric dataset for proactive procedural assistance with explicit Out-of-Plan (OOP) annotations and recovery steps.<n>We propose a textbfdecoupled planner--interaction architecture specialized for procedural state, visual cues, and recovery injection.<n>In extensive experiments, our trained Llama-4 system substantially improves objective intervention quality over strong proprietary baselines.
• Score: 49.46086226939232
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We envision a proactive multi-modal assistant system which gives users real-time step-by-step guidance on a procedural task, autonomously deciding \textit{when} to interrupt, and \textit{how} to coach. However, progress is limited by the absence of large-scale, cross-domain benchmarks that reflect realistic conditions, particularly the common case in which users deviate from the expected step sequence. We address this gap with four contributions: \textbf{(1)}~we release \textbf{EgoProactive}, a large-scale wearable-egocentric dataset for proactive procedural assistance with explicit Out-of-Plan (OOP) annotations and recovery steps; \textbf{(2)}~we augment five established benchmarks (Ego4D, EPIC-KITCHENS, EgoExo4D, HoloAssist, HowTo100M) into \textbf{Pro\textsuperscript{2}Bench} under a unified proactive-guidance schema; \textbf{(3)}~we propose a \textbf{decoupled planner--interaction architecture} specialized for procedural state, visual cues, and recovery injection; \textbf{(4)}~we introduce a post-training recipe that transfers across model families, validated by cross-backbone replication on Llama~4 and Qwen-3.6-VL. In extensive experiments, our trained Llama-4 system substantially improves objective intervention quality over strong proprietary baselines (Claude Opus~4.6, Gemini~3.1~Pro, GPT~5.2) and open-weight baselines (Qwen3~VL~235B) baselines across all six datasets. Oracle-plan experiments further show that, when plan quality is controlled, the trained duplex model produces high-quality guidance and large gains on Out-of-Plan recovery.

Related papers

• SLAI T-Rex: Full-Parameter Post-training of the DeepSeek-V4 Family on Ascend SuperPOD [91.71050339479261]
  This report presents an end-to-end optimization practice on the Ascend NPU SuperPOD.<n>We develop a hierarchical optimization framework spanning model-level parallelism, computation-communication orchestration, and low-level kernel execution.<n>The resulting system achieves 34.22% Model FLOPs Utilization (MFU) with a 2.93x improvement over the open-source baseline recipe.
  arXiv  Detail & Related papers  (2026-07-22T13:49:17Z)
• Cortex: A Bidirectionally Aligned Embodied Agent Framework for Long-horizon Manipulation [40.0334846364089]
  Vision-Language-Action (VLA) models struggle with long-horizon tasks due to their Markovian nature-relying solely on current observations.<n>We introduce Cortex, a framework that conveys executable and tractable subtask plans from high-level VLM to low-level VLA.<n>This enables automatic annotation of over 4k hours of open-source video data and generation of 30 hours of simulation data.
  arXiv  Detail & Related papers  (2026-07-06T17:55:05Z)
• Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models [63.96252564303143]
  Embodied-R1.5 is a unified Embodied Foundation Model (EFM) that integrates comprehensive embodied reasoning capabilities.<n>We build a large-scale data system of over 15B tokens, and design a multi-task balanced RL recipe.<n>With only 8B parameters, Embodied-R1.5 SOTA on 16 out of 24 embodied VLM benchmarks, surpassing leading models like Gemini-Robotics-ER-1.5 and GPT-5.4.
  arXiv  Detail & Related papers  (2026-06-09T18:07:50Z)
• TIER: Trajectory-Invariant Execution Rewards for Multi-Step Tool Composition [62.56752617853322]
  Outcome-based rewards provide only sparse feedback, while trajectory-supervised rewards depend on annotated reference solutions.<n>We propose TIER: Trajectory-Invariant Execution Rewards, a reward framework that derives supervision directly from function schemas and runtime execution.
  arXiv  Detail & Related papers  (2026-05-16T03:47:26Z)
• PIVOT: Bridging Planning and Execution in LLM Agents via Trajectory Refinement [22.285969725581044]
  Large language model (LLM)-based agents generate seemingly coherent plans that fail upon execution due to infeasible actions, constraint violations, and compounding errors over extended horizons.<n>PIVOT addresses this plan-execution misalignment through a self-supervised framework that treats trajectories as optimizable objects iteratively refined via environment interaction.
  arXiv  Detail & Related papers  (2026-05-11T20:43:20Z)
• Every Step Counts: Step-Level Credit Assignment for Tool-Integrated Text-to-SQL [13.130554329857496]
  We propose FineStep, a novel framework for step-level credit assignment in tool-augmented Text-to-augmented Text.<n>We show that FineStep achieves state-of-the-art performance and reduces redundant tool interactions, with a 3.25% average EX gain over GRPO at the 4B scale.
  arXiv  Detail & Related papers  (2026-05-06T10:10:31Z)
• APEX-EM: Non-Parametric Online Learning for Autonomous Agents via Structured Procedural-Episodic Experience Replay [7.370176470430802]
  LLM-based autonomous agents lack persistent procedural memory.<n>We present APEX-EM, a non-parametric online learning framework that accumulates, retrieves, and reuses structured procedural plans.
  arXiv  Detail & Related papers  (2026-03-31T00:24:56Z)
• MagicAgent: Towards Generalized Agent Planning [73.21129030631421]
  We present textbfMagicAgent, a series of foundation models specifically designed for generalized agent planning.<n>We introduce a lightweight and scalable synthetic data framework that generates high-quality trajectories across diverse planning tasks.<n>We show that MagicAgent-32B and MagicAgent-30B-A3B achieve superior performance across diverse open-source benchmarks.
  arXiv  Detail & Related papers  (2026-02-22T01:39:16Z)
• Tool-Aware Planning in Contact Center AI: Evaluating LLMs through Lineage-Guided Query Decomposition [2.8180871881371456]
  We present a domain-grounded framework and benchmark for tool-aware plan generation in contact centers.<n>Our contributions are threefold: (i) a reference-based plan evaluation framework operating in two modes - a metric-wise evaluator and a one-shot evaluator, and (ii) a data methodology that iteratively refines plans via an evaluator->optimizer loop.
  arXiv  Detail & Related papers  (2026-02-16T17:36:05Z)
• ProRAG: Process-Supervised Reinforcement Learning for Retrieval-Augmented Generation [54.071574153853994]
  ProRAG is a process-supervised reinforcement learning framework designed to integrate learned step-level supervision into the online optimization loop.<n>Our framework consists of four stages: (1) Supervised Policy Warmup to initialize the model with a structured reasoning format; (2) construction of an MCTS-based Process Reward Model (PRM) to quantify intermediate reasoning quality; (3) PRM-Guided Reasoning Refinement to align the policy with fine-grained process preferences; and (4) Process-Supervised Reinforcement Learning with a dual-granularity advantage mechanism.
  arXiv  Detail & Related papers  (2026-01-29T16:04:59Z)
• Experiences from Benchmarking Vision-Language-Action Models for Robotic Manipulation [22.40720239761228]
  This paper reports our textbfempirical experiences from benchmarking four representative VLAs.<n>We establish a textbfstandardized evaluation framework that measures performance along three key dimensions.<n>We find practical trade-offs among VLA model architectures in balancing precision, generalization, and deployment cost.
  arXiv  Detail & Related papers  (2025-11-14T13:35:30Z)
• SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modelling [58.05959902776133]
  We introduce Single-Pass.<n>with Reference-Guided Evaluation (SPARE), a novel structured framework that enables efficient per-step annotation.<n>We demonstrate SPARE's effectiveness across four diverse datasets spanning mathematical reasoning (GSM8K, MATH), multi-hop question answering (MuSiQue-Ans), and spatial reasoning (SpaRP)<n>On ProcessBench, SPARE demonstrates data-efficient out-of-distribution generalization, using only $sim$16% of training samples compared to human-labeled and other synthetically trained baselines.
  arXiv  Detail & Related papers  (2025-06-18T14:37:59Z)
• Learning to Decompose: Hypothetical Question Decomposition Based on Comparable Texts [65.84370471189676]
  We look at large-scale intermediate pre-training of decomposition-based transformers using distant supervision from comparable texts. We show that with such intermediate pre-training, developing robust decomposition-based models for a diverse range of tasks becomes more feasible.
  arXiv  Detail & Related papers  (2022-10-30T15:38:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.