OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs

• URL: http://arxiv.org/abs/2608.21360v1
• Date: Fri, 21 Aug 2026 17:59:52 GMT
• Title: OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs
• Abstract summary: Recent omni-modal large language models ( Omni-LLMs) show great potential as real-time video assistants.<n>Unlike traditional passive video understanding, interactive assistants should actively combine visual states, user goals, and prior knowledge to provide effective help.
• Score: 25.570460595046058
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent omni-modal large language models (Omni-LLMs) show great potential as real-time video assistants, which continuously perceive environments and guide users to achieve specific goals. Unlike traditional passive video understanding, interactive assistants should actively combine visual states, user goals, and prior knowledge to provide effective help. Evaluating this is rather challenging, as the model's unpredictable response dynamically changes the user's subsequent actions, which static offline datasets cannot accommodate. To address this bottleneck, we introduce OmniAssistBench. To solve the issue of diverging interaction paths where the same user goal can be achieved through various methods, we provide models with predefined priors derived from the source video, requiring them to guide users along the exact same routes. Since real interaction videos are rare, we construct the dataset by reverse-engineering existing Internet videos. We deduce logical user goals and segment the videos into multi-turn clips to simulate continuous interactions. This rigorous pipeline required over 1000 expert person-hours to build the dataset. Results show that the proprietary Gemini-3-Pro reaches 66.4 out of the max point of 100, while the open-source Qwen3-Omni-Instruct achieves 51.2. Although current models generally understand user inputs, they frequently provide incorrect or incomplete answers. Specifically, they struggle with visual prompts (e.g., hand gestures), fail to maintain historical context during multi-turn interactions, and fail to delay response until the target event. Results indicate substantial room for improvement before models can become reliable assistants.

Related papers

• InteracVid: Building a Real Interactive Audio-Visual Response Dataset from Live-Chat Videos [7.003662856458999]
  We introduce textbfInteracVid, emphthe firstopen-source large-scale dataset that addresses this missing supervision<n>Everysample couples a preceding audio-visual context and an external stimulus with the realinteractive response that follows.<n>A ten-rater human study confirms that the extracted interactions are causal,natural, and temporally complete for both genuine and reconstructed queries.
  arXiv  Detail & Related papers  (2026-08-02T11:28:13Z)
• OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants [43.2907783462394]
  Streaming benchmark for realtime omnimodal language models evaluated through native inference over audio-visual streams.<n>We evaluate response correctness, timing, invalid outputs interruption, and context continuity using Interaction-Aware QualityTimeliness F1, Interruption Diagnostic Suite, and 368 1QnA slots for continuous task monitoring and step guidance.
  arXiv  Detail & Related papers  (2026-05-26T02:49:47Z)
• Action-Aware Generative Sequence Modeling for Short Video Recommendation [50.74467504063892]
  Short videos often contain diverse segments, and users may not hold the same attitude toward all of them.<n>Traditional binary-classification recommendation models, which treat a video as a single holistic entity, face limitations in accurately capturing such nuanced preferences.<n>We propose a novel modeling paradigm: Action-Aware Generative Sequence Network (A2Gen), which refines user actions along the temporal dimension.
  arXiv  Detail & Related papers  (2026-04-28T16:41:04Z)
• OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments [5.613789567241434]
  We introduce OmniGUI, the first step-level benchmark designed to evaluate GUI agents in omni-modal smartphone environments.<n>The dataset encompasses 709 expert-demonstrated episodes (2,579 action steps) across 29 applications.<n>Our empirical evaluation reveals that while current models exhibit competency on visually static tasks, their action prediction performance degrades significantly in environments requiring synchronous temporal and auditory signals.
  arXiv  Detail & Related papers  (2026-04-03T08:57:06Z)
• LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMs [68.35684758116453]
  This dataset comprises high-quality videos sourced from open platforms that feature rich audio-visual dynamics.<n>We rigorously evaluate the capabilities of OmniLLMs across domains, including long-term memory, temporal localization, fine-grained understanding, and multimodal perception.<n>Open-source models generally achieve accuracies below 35%, whereas the Gemini 3 Pro reaches a peak accuracy of approximately 65%.
  arXiv  Detail & Related papers  (2026-03-19T17:58:13Z)
• Do You See What I Am Pointing At? Gesture-Based Egocentric Video Question Answering [81.15678825882692]
  EgoPointVQA is a dataset and benchmark for gesture-grounded egocentric question answering.<n>Hand Intent Tokens (HINT) encodes tokens derived from 3D hand keypoints using an off-the-shelf reconstruction model.<n>We show that our model outperforms others in different backbones and model sizes.
  arXiv  Detail & Related papers  (2026-03-13T00:23:38Z)
• Learning Next Action Predictors from Human-Computer Interaction [53.41504288073184]
  LongNAP is a user model that combines parametric and in-context learning to reason over long interaction histories.<n>We label over 360K actions across one month of continuous phone usage from 20 users, amounting to 1,800 hours of screen time.<n>LongNAP significantly outperforms supervised finetuning and prompted baselines on held-out data.
  arXiv  Detail & Related papers  (2026-03-06T05:31:04Z)
• Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance? [27.55597863240135]
  We evaluate state-of-the-art multi-modal Large Language Models (LLM) on the Qualcomm Interactive Cooking benchmark.<n>We introduce LiveMamba, a streaming multi-modal LLM designed for interactive instructional guidance.<n>This work provides the first dedicated benchmark and a strong baseline for developing and evaluating on live, situated coaching.
  arXiv  Detail & Related papers  (2025-11-27T00:54:35Z)
• One Battle After Another: Probing LLMs' Limits on Multi-Turn Instruction Following with a Benchmark Evolving Framework [51.50565654314582]
  Large language models can follow users' instructions throughout a dialogue spanning multiple topics.<n>Existing benchmarks are often limited to a fixed number of turns, making them susceptible to saturation and failing to account for the user's interactive experience.<n>We propose a framework for assessing multi-turn instruction-following ability.
  arXiv  Detail & Related papers  (2025-11-05T14:39:59Z)
• Mind the Gap! Static and Interactive Evaluations of Large Audio Models [55.87220295533817]
  Large Audio Models (LAMs) are designed to power voice-native experiences.<n>This study introduces an interactive approach to evaluate LAMs and collect 7,500 LAM interactions from 484 participants.
  arXiv  Detail & Related papers  (2025-02-21T20:29:02Z)
• Strike the Balance: On-the-Fly Uncertainty based User Interactions for Long-Term Video Object Segmentation [23.417370317522106]
  We introduce a variant of video object segmentation (VOS) that bridges interactive and semi-automatic approaches. We aim to maximize the tracking duration of an object of interest, while requiring minimal user corrections to maintain tracking over an extended period. We evaluate our approach using the recently introduced LVOS dataset, which offers numerous long-term videos.
  arXiv  Detail & Related papers  (2024-07-31T21:42:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.