EgoArgus: Benchmarking VLMs as Situational Assistants for Modality-Grounded User Supports

• URL: http://arxiv.org/abs/2608.25561v1
• Date: Wed, 26 Aug 2026 09:09:30 GMT
• Title: EgoArgus: Benchmarking VLMs as Situational Assistants for Modality-Grounded User Supports
• Abstract summary: We introduce EgoArgus, a human-annotated dataset for evaluating egocentric assistants on understanding and decision tasks.<n>Our results demonstrate that it is still challenging for current VLMs as reliable egocentric assistants.
• Score: 5.919102792488079
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: VLMs are increasingly positioned as daily assistants that perceive first-person environments, follow user dialogue, and decide how to help. Existing egocentric benchmarks mainly evaluate visual understanding in isolation, leaving open whether models can arbitrate between visual evidence and user-provided language when the two are helpful, irrelevant, or conflicting. We introduce EgoArgus, a human-annotated dataset for evaluating egocentric assistants on understanding and decision tasks in five dialogue-video daily scenarios. Our results demonstrate that it is still challenging for current VLMs as reliable egocentric assistants, which requires identifying which modality is trustworthy and deciding when intervention is warranted. Deeper analysis also shows that existing modality bias mitigation methods are quite restricted to enhance performance, providing insights to aid practioners into the deployment of current VLMs as daily assistants.

Related papers

• Self-Evolving Visual Questioner [53.949845616431624]
  Vision-language models (VLMs) are typically trained as passive answerers.<n>We show that a VLM can continuously improve itself as a visual questioner without any external supervision.<n>We propose a self-evolving framework that uses a VLM itself as both a proposer and a filter to produce harder, more informative, and visual-centric questions.
  arXiv  Detail & Related papers  (2026-06-11T21:45:46Z)
• EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoning [47.853306116245484]
  EgoIntrospect is the first egocentric dataset captured in user-driven scenarios with self-annotations.<n>It consists of 180 hours of recordings from 60 subjects, with an average recording duration of 3 hours per subject.<n>We formalize a suite of tasks centered on user internal states, including affective experience, interactive intent, and cognitive memory.
  arXiv  Detail & Related papers  (2026-05-17T05:05:29Z)
• LifeEval: A Multimodal Benchmark for Assistive AI in Egocentric Daily Life Tasks [71.05217306468857]
  LifeEval is a multimodal benchmark designed to evaluate real-time, task-oriented human-AI collaboration in daily life.<n>LifeEval emphasizes three key aspects: task-oriented holistic evaluation, egocentric real-time perception from continuous first-person streams, and human-assistant collaborative interaction through natural dialogues.
  arXiv  Detail & Related papers  (2026-02-28T06:05:31Z)
• Robust Egocentric Referring Video Object Segmentation via Dual-Modal Causal Intervention [58.05340906967343]
  Egocentric Referring Video Object (Ego-RVOS) aims to segment the specific object actively involved in a human action, as described by a language query, within first-person videos.<n>Existing methods often struggle, learning spurious correlations from skewed object-action pairings in datasets.<n>We introduce Causal-REferring (CERES), a plug-in causal framework that adapts strong, pre-trained RVOSs to the egocentric domain.
  arXiv  Detail & Related papers  (2025-12-30T16:22:14Z)
• VLMs Guided Interpretable Decision Making for Autonomous Driving [39.29020915361483]
  We evaluate state-of-the-art open-source vision-language models (VLMs) on high-level decision-making tasks.<n>We propose a new approach that shifts the role of VLMs from direct decision generators to semantic enhancers.<n>Our approach achieves state-of-the-art performance, offering a promising direction for integrating VLMs into reliable and interpretable AD systems.
  arXiv  Detail & Related papers  (2025-11-17T19:57:51Z)
• In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting [12.567763863700058]
  EgoGazeVQA is an egocentric gaze-guided video question answering benchmark.<n>Our experiments reveal that existing MLLMs struggle to accurately interpret user intentions.<n>Our gaze-guided intent prompting methods significantly enhance performance.
  arXiv  Detail & Related papers  (2025-09-09T07:11:56Z)
• EgoBlind: Towards Egocentric Visual Assistance for the Blind [94.45487133965361]
  EgoBlind is the first egocentric VideoQA dataset collected from blind individuals.<n>It comprises 1,392 first-person videos from the daily lives of blind and visually impaired individuals.<n>It also features 5,311 questions directly posed or verified by the blind to reflect their in-situation needs for visual assistance.
  arXiv  Detail & Related papers  (2025-03-11T09:40:31Z)
• EgoLife: Towards Egocentric Life Assistant [60.51196061794498]
  We introduce EgoLife, a project to develop an egocentric life assistant that accompanies and enhances personal efficiency through AI-powered wearable glasses.<n>We conduct a comprehensive data collection study where six participants lived together for one week, continuously recording their daily activities using AI glasses for multimodal egocentric video capture, along with synchronized third-person-view video references.<n>This effort resulted in the EgoLife dataset, a comprehensive 300-hour egocentric, interpersonal, multiview, and multimodal daily life dataset with intensive annotation.<n>We introduce EgoLifeQA, a suite of long-context, life-oriented question-answering tasks designed to provide
  arXiv  Detail & Related papers  (2025-03-05T18:54:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.