Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance?

• URL: http://arxiv.org/abs/2511.21998v1
• Date: Thu, 27 Nov 2025 00:54:35 GMT
• Title: Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance?
• Abstract summary: We evaluate state-of-the-art multi-modal Large Language Models (LLM) on the Qualcomm Interactive Cooking benchmark.<n>We introduce LiveMamba, a streaming multi-modal LLM designed for interactive instructional guidance.<n>This work provides the first dedicated benchmark and a strong baseline for developing and evaluating on live, situated coaching.
• Score: 27.55597863240135
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Multi-modal Large Language Models (LLM) have advanced conversational abilities but struggle with providing live, interactive step-by-step guidance, a key capability for future AI assistants. Effective guidance requires not only delivering instructions but also detecting their successful execution, as well as identifying and alerting users to mistakes, all of which has to happen in real-time. This requires models that are not turn-based, but that can react asynchronously to a video stream, as well as video data showing users performing tasks including mistakes and their corrections. To this end, we introduce Qualcomm Interactive Cooking, a new benchmark and dataset built upon CaptainCook4D, which contains user mistakes during task execution. Our dataset and benchmark features densely annotated, timed instructions and feedback messages, specifically including mistake alerts precisely timestamped to their visual occurrence in the video. We evaluate state-of-the-art multi-modal LLMs on the Qualcomm Interactive Cooking benchmark and introduce LiveMamba, a streaming multi-modal LLM designed for interactive instructional guidance. This work provides the first dedicated benchmark and a strong baseline for developing and evaluating on live, situated coaching.

Related papers

• Time-Aware Assistive Navigation [17.665122992601788]
  Current language models rarely plan over whether and when to realize a real-time response to a user.<n>We introduce a large-scale benchmark for an egocentric, assistive navigation task in complex outdoor environments.<n>We show that a simple yet effective modification of the model, including direct supervision to predict the underlying reason for each instruction, yields significant performance gains.
  arXiv  Detail & Related papers  (2026-09-04T17:59:47Z)
• Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarios [61.926801508808616]
  We introduce Video-IFBench, a benchmark for evaluating instruction following in video understanding.<n>We develop an instruction taxonomy with four templates, including single-task, multi-task, selection, and nested instructions.<n>We build a semi-automatic data construction pipeline that combines MLLMs, programmatic processing, and human verification, resulting in 1.5K samples.
  arXiv  Detail & Related papers  (2026-08-26T08:35:42Z)
• GuideMe: Multi-Domain Task Guidance and Intervention in Streaming Video [54.048366682378706]
  We construct GuideMe, the first multi-domain benchmark for streaming video that supports training and evaluation of MLLMs for closed-loop interactive task guidance.<n>It comprises 2,458 videos spanning 223.7 hours across diverse domains (eg, cooking, object manipulation, daily-life guidance, and fitness), with 47,775 interaction samples covering next-step instructions, completion feedback, error detection, and corrective guidance.
  arXiv  Detail & Related papers  (2026-07-03T06:00:04Z)
• True Multimodal In-Context Learning Needs Attention to the Visual Context [69.63677595066012]
  Multimodal Large Language Models (MLLMs) have enabled Multimodal In-Context Learning (MICL)-adapting to new tasks.<n>Current MLLMs tend to neglect visual cues and over-rely on textual patterns, leading to mere text imitation rather than genuine multimodal adaptation.<n>We introduce Dynamic Attention Reallocation (DARA), an efficient fine-tuning strategy that encourages models to attend to the visual context.
  arXiv  Detail & Related papers  (2025-07-21T17:08:18Z)
• ViSpeak: Visual Instruction Feedback in Streaming Videos [50.99067964073338]
  We propose a novel task named Visual Instruction Feedback in which models should be aware of visual contents and learn to extract instructions from them.<n>We propose the ViSpeak model, which is a SOTA streaming video understanding LMM with GPT-4o-level performance on various streaming video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-03-17T03:05:31Z)
• TWIST & SCOUT: Grounding Multimodal LLM-Experts by Forget-Free Tuning [54.033346088090674]
  We introduce TWIST & SCOUT, a framework that equips pre-trained MLLMs with visual grounding ability.<n>To fine-tune the model effectively, we generate a high-quality synthetic dataset we call SCOUT.<n>This dataset provides rich supervision signals, describing a step-by-step multimodal reasoning process.
  arXiv  Detail & Related papers  (2024-10-14T13:35:47Z)
• Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models [0.0]
  We introduce a dynamic benchmarking system for conversational agents that evaluates their performance through a single, simulated, and lengthy user interaction. We context switch regularly to interleave the tasks, which constructs a realistic testing scenario in which we assess the Long-Term Memory, Continual Learning, and Information Integration capabilities of the agents.
  arXiv  Detail & Related papers  (2024-09-30T12:01:29Z)
• User-in-the-loop Evaluation of Multimodal LLMs for Activity Assistance [14.11032383241148]
  This research investigates the capability of modern multimodal reasoning models, powered by Large Language Models (LLMs), to facilitate vision-powered assistants for daily activities. Such assistants must be able to 1) encode relevant visual history from the assistant's sensors, e.g., camera, 2) forecast future actions for accomplishing the activity, and 3) replan based on the user in the loop. We find that the Socratic approach outperforms VCLMs in both offline and online settings.
  arXiv  Detail & Related papers  (2024-08-04T06:12:42Z)
• Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning [102.54669633984278]
  We propose Momentor, a Video-LLM capable of accomplishing fine-grained temporal understanding tasks. We train Momentor on Moment-10M, enabling it to perform segment-level reasoning and localization.
  arXiv  Detail & Related papers  (2024-02-18T03:04:38Z)
• Prompt Highlighter: Interactive Control for Multi-Modal LLMs [50.830448437285355]
  This study targets a critical aspect of multi-modal LLMs' (LLMs&VLMs) inference: explicit controllable text generation. We introduce a novel inference method, Prompt Highlighter, which enables users to highlight specific prompt spans to interactively control the focus during generation. We find that, during inference, guiding the models with highlighted tokens through the attention weights leads to more desired outputs.
  arXiv  Detail & Related papers  (2023-12-07T13:53:29Z)
• Look, Remember and Reason: Grounded reasoning in videos with language models [5.3445140425713245]
  Multi-temporal language models (LM) have recently shown promising performance in high-level reasoning tasks on videos. We propose training an LM end-to-end on low-level surrogate tasks, including object detection, re-identification, tracking, to endow the model with the required low-level visual capabilities. We demonstrate the effectiveness of our framework on diverse visual reasoning tasks from the ACRE, CATER, Something-Else and STAR datasets.
  arXiv  Detail & Related papers  (2023-06-30T16:31:14Z)
• MIMIC-IT: Multi-Modal In-Context Instruction Tuning [44.879418596312554]
  We present a dataset comprising 2.8 million multimodal instruction-response pairs, with 2.2 million unique instructions derived from images and videos. Using the MIMIC-IT dataset, it has been observed that Otter demonstrates remarkable proficiency in multi-modal perception, reasoning, and in-context learning. We release the MIMIC-IT dataset, instruction-response collection pipeline, benchmarks, and the Otter model.
  arXiv  Detail & Related papers  (2023-06-08T17:59:56Z)
• Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos [76.21297023629589]
  We propose a novel method for learning pairwise modality interactions in order to better exploit complementary information for each pair of modalities in videos. Our method turns out to achieve state-of-the-art performances on four standard benchmark datasets.
  arXiv  Detail & Related papers  (2020-07-28T12:40:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.