GuideMe: Multi-Domain Task Guidance and Intervention in Streaming Video

• URL: http://arxiv.org/abs/2607.02991v1
• Date: Fri, 03 Jul 2026 06:00:04 GMT
• Title: GuideMe: Multi-Domain Task Guidance and Intervention in Streaming Video
• Abstract summary: We construct GuideMe, the first multi-domain benchmark for streaming video that supports training and evaluation of MLLMs for closed-loop interactive task guidance.<n>It comprises 2,458 videos spanning 223.7 hours across diverse domains (eg, cooking, object manipulation, daily-life guidance, and fitness), with 47,775 interaction samples covering next-step instructions, completion feedback, error detection, and corrective guidance.
• Score: 54.048366682378706
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While multimodal Large Language Models (MLLMs) excel at offline video understanding, an interesting question of how far they are from serving as a real-time procedural coach remains unknown. Such a role typically requires an MLLM to continuously monitor the execution, detect mistakes, and provide corrective guidance in a closed-loop interaction. In this paper, we construct GuideMe, the first multi-domain benchmark for streaming video that supports training and evaluation of MLLMs for closed-loop interactive task guidance. It comprises 2,458 videos spanning 223.7 hours across diverse domains (\eg, cooking, object manipulation, daily-life guidance, and fitness), with 47,775 interaction samples covering next-step instructions, completion feedback, error detection, and corrective guidance. To evaluate existing models on GuideMe, we design a three-component assessment framework to measure the capabilities of representative MLLMs, which consists of temporal-semantic bipartite matching for sequence-level alignment, behavioral classification for intervention timing, and LLM-as-a-Judge for content quality. Extensive experiments highlight a critical performance asymmetry: despite excelling at providing instructions, existing MLLMs consistently fail to identify execution errors and respond with corrective feedback. Code and data are released at https://fawnliu.github.io/project/guideme.

Related papers

• Benchmarking Visual State Tracking in Multimodal Video Understanding [67.73383974166462]
  We introduce Visual STAte Tracking benchmark (VSTAT) to diagnose visual state tracking in Multimodal Large Language Models (MLLMs)<n>VSTAT consists of 834 clips drawn from both synthetic and real-world videos, paired with 1,500 questions that cannot be answered from any single frame or short segment.<n>We find that MLLMs reason and track correctly in text, but fail at visually perceiving the events they need to track.
  arXiv  Detail & Related papers  (2026-06-02T17:12:05Z)
• Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance? [27.55597863240135]
  We evaluate state-of-the-art multi-modal Large Language Models (LLM) on the Qualcomm Interactive Cooking benchmark.<n>We introduce LiveMamba, a streaming multi-modal LLM designed for interactive instructional guidance.<n>This work provides the first dedicated benchmark and a strong baseline for developing and evaluating on live, situated coaching.
  arXiv  Detail & Related papers  (2025-11-27T00:54:35Z)
• Controlling Multimodal LLMs via Reward-guided Decoding [17.5544679985101]
  We study the adaptation of Multimodal Large Language Models (MLLMs) through controlled decoding.<n>Our method involves building reward models for visual grounding and using them to guide the MLLM's decoding process.<n>We evaluate our method on standard object hallucination benchmarks, showing that it provides significant controllability over MLLM inference.
  arXiv  Detail & Related papers  (2025-08-15T17:29:06Z)
• A Survey on Video Temporal Grounding with Multimodal Large Language Model [107.24431595873808]
  Recent advancement in temporal grounding (VTG) has significantly enhanced fine-grained video understanding.<n>With superior multimodal comprehension and reasoning abilities, VTG approaches based on MLLMs (VTG-MLLMs) are gradually surpassing traditional fine-tuned methods.<n>Despite extensive surveys on general video-language understanding, comprehensive reviews specifically addressing VTG-MLLMs remain scarce.
  arXiv  Detail & Related papers  (2025-08-07T08:52:11Z)
• Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verification [17.67273082468732]
  Verifiers -- functions assigning rewards to agent behavior -- have been key for AI progress in domains like math and board games.<n>We evaluate Multimodal Large Language Models (MLLMs) as verifiers of agent trajectories across web navigation, computer use, and robotic manipulation.<n>We propose Self-Grounded Verification (SGV), a lightweight method that enables more effective use of MLLMs' knowledge and reasoning.
  arXiv  Detail & Related papers  (2025-07-15T18:50:29Z)
• If an LLM Were a Character, Would It Know Its Own Story? Evaluating Lifelong Learning in LLMs [55.8331366739144]
  We introduce LIFESTATE-BENCH, a benchmark designed to assess lifelong learning in large language models (LLMs)<n>Our fact checking evaluation probes models' self-awareness, episodic memory retrieval, and relationship tracking, across both parametric and non-parametric approaches.
  arXiv  Detail & Related papers  (2025-03-30T16:50:57Z)
• Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model [63.14883657299359]
  Multi-modal Large Language Models (MLLMs) integrate visual and linguistic reasoning to address complex tasks such as image captioning and visual question answering.<n> tuning MLLMs for downstream tasks encounters two key challenges: Task-Expert, where distribution shifts between pre-training and target datasets constrain target performance, and OpenWorld Stabilization, where catastrophic forgetting erases the model general knowledge.
  arXiv  Detail & Related papers  (2025-03-06T15:29:13Z)
• LLM-AutoDiff: Auto-Differentiate Any LLM Workflow [58.56731133392544]
  We introduce LLM-AutoDiff: a novel framework for Automatic Prompt Engineering (APE)<n>LLMs-AutoDiff treats each textual input as a trainable parameter and uses a frozen backward engine to generate feedback-akin to textual gradients.<n>It consistently outperforms existing textual gradient baselines in both accuracy and training cost.
  arXiv  Detail & Related papers  (2025-01-28T03:18:48Z)
• Unveiling Uncertainty: A Deep Dive into Calibration and Performance of Multimodal Large Language Models [36.81503322875839]
  Multimodal large language models (MLLMs) combine visual and textual data for tasks such as image captioning and visual question answering.<n>This paper investigates representative MLLMs, focusing on their calibration across various scenarios.<n>We observed miscalibration in their performance, and at the same time, no significant differences in calibration across these scenarios.
  arXiv  Detail & Related papers  (2024-12-19T09:10:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.