EgoPro-Bench: Benchmarking Personalized Proactive Interaction in Egocentric Video Streams

• URL: http://arxiv.org/abs/2605.07299v1
• Date: Fri, 08 May 2026 06:10:23 GMT
• Title: EgoPro-Bench: Benchmarking Personalized Proactive Interaction in Egocentric Video Streams
• Abstract summary: EgoPro-Bench is a novel benchmark for training and evaluating proactive interaction capabilities.<n>It comprises 2,400 videos in the evaluation set and over 12,000 videos in the training set.<n>The experiments demonstrate that EgoPro-Bench substantially enhances the intention understanding capabilities of MLLMs.
• Score: 12.739319690508482
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing Multimodal Large Language Models (MLLMs) remain primarily reactive, failing to continuously perceive environments or proactively assist users. While emerging benchmarks address proactivity, they are largely confined to alert scenarios, neglect personalized context, and fail to evaluate the precise timing of human-machine interactions (HMI).In this paper, we introduce EgoPro-Bench, a novel benchmark for training and evaluating proactive interaction capabilities based on streaming egocentric videos; it comprises 2,400 videos in the evaluation set and over 12,000 videos in the training set.Unlike previous works, EgoPro-Bench leverages simulated user profiles to generate diverse user intentions and to construct high-fidelity HMI data across 12 distinct domains.Subsequently, we propose a specialized evaluation protocol and metrics, train proactive interaction models designed for efficient reasoning and low-latency interaction on streaming video data, and conduct comprehensive evaluations.Furthermore, we introduce an interaction principle termed "short thinking, better interaction", which allocates a limited token budget prior to intent recognition, thereby enhancing interaction performance.The experiments demonstrate that EgoPro-Bench substantially enhances the intention understanding capabilities of MLLMs and enables accurate identification of appropriate timings for HMI, thereby laying a solid foundation for next-generation user-centric proactive interactive agents.

Related papers

• EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents [17.727481701114556]
  We introduce EgoBench, the first interactive multimodal benchmark for tool-using agents.<n>We implement a three-stage synergistic pipeline through which each task is designed to enforce the joint application of visual perception and tool-augmented multi-hop reasoning.<n>We additionally develop a multi-agent simulated user within EgoBench to evaluate agents' interaction capabilities, which generates high-fidelity, task-aligned responses.
  arXiv  Detail & Related papers  (2026-05-27T01:28:15Z)
• VitaBench 2.0: Evaluating Personalized and Proactive Agents in Long-Term User Interactions [63.13827503828231]
  We introduce VitaBench 2.0, a benchmark for evaluating personalized and proactive agent behavior in long-term user interactions.<n>Results show that real-world personalization remains highly challenging even for state-of-the-art models.
  arXiv  Detail & Related papers  (2026-05-26T15:07:38Z)
• MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agents [54.48066948369172]
  As embodied models become powerful, humans will collaborate with multiple embodied AI agents at their workplace or home in the future.<n>Existing challenges include effectively compressing and communicating high volumes of individual sensory inputs in the form of video.<n>We first formally define a novel problem of understanding multiple long-horizon egocentric videos simultaneously collected from embodied agents.
  arXiv  Detail & Related papers  (2026-03-10T15:48:35Z)
• Proact-VL: A Proactive VideoLLM for Real-Time AI Companions [52.23988809605433]
  We instantiate AI companions through two gaming scenarios, commentator and guide, selected for automatic evaluation.<n>We present Proact-VL, a framework that shapes multimodal language models into proactive, real-time interactive agents capable of human-like environment perception and interaction.
  arXiv  Detail & Related papers  (2026-03-03T19:02:46Z)
• LifeEval: A Multimodal Benchmark for Assistive AI in Egocentric Daily Life Tasks [71.05217306468857]
  LifeEval is a multimodal benchmark designed to evaluate real-time, task-oriented human-AI collaboration in daily life.<n>LifeEval emphasizes three key aspects: task-oriented holistic evaluation, egocentric real-time perception from continuous first-person streams, and human-assistant collaborative interaction through natural dialogues.
  arXiv  Detail & Related papers  (2026-02-28T06:05:31Z)
• Robust Egocentric Referring Video Object Segmentation via Dual-Modal Causal Intervention [58.05340906967343]
  Egocentric Referring Video Object (Ego-RVOS) aims to segment the specific object actively involved in a human action, as described by a language query, within first-person videos.<n>Existing methods often struggle, learning spurious correlations from skewed object-action pairings in datasets.<n>We introduce Causal-REferring (CERES), a plug-in causal framework that adapts strong, pre-trained RVOSs to the egocentric domain.
  arXiv  Detail & Related papers  (2025-12-30T16:22:14Z)
• PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments [36.84821207878773]
  Visual reasoning in multimodal large language models (MLLMs) has primarily been studied in static, fully observable settings.<n>We introduce the Active Visual Reasoning (AVR) task, extending visual reasoning to partially observable, interactive environments.<n>We present a benchmark featuring multi-round interactive environments designed to assess both reasoning and information-gathering efficiency.
  arXiv  Detail & Related papers  (2025-10-24T02:59:00Z)
• In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting [12.567763863700058]
  EgoGazeVQA is an egocentric gaze-guided video question answering benchmark.<n>Our experiments reveal that existing MLLMs struggle to accurately interpret user intentions.<n>Our gaze-guided intent prompting methods significantly enhance performance.
  arXiv  Detail & Related papers  (2025-09-09T07:11:56Z)
• Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactions [110.43343503158306]
  This paper embeds the manual-assisted task into a vision-language-action framework, where the assistant provides services to the instructor following egocentric vision and commands.<n>Under this setting, we accomplish InterVLA, the first large-scale human-object-human interaction dataset with 11.4 hours and 1.2M frames of multimodal data.<n>We establish novel benchmarks on egocentric human motion estimation, interaction synthesis, and interaction prediction with comprehensive analysis.
  arXiv  Detail & Related papers  (2025-08-06T17:46:23Z)
• ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Models [41.35497807436858]
  We introduce ProactiveVideoQA, the first comprehensive benchmark to evaluate a system's ability to engage in proactive interaction.<n>We also propose PAUC, the first metric that accounts for the temporal dynamics of model responses.<n>These findings demonstrate that PAUC provides a more faithful assessment of user experience in proactive interaction scenarios.
  arXiv  Detail & Related papers  (2025-07-12T15:11:50Z)
• HumanVideo-MME: Benchmarking MLLMs for Human-Centric Video Understanding [120.84817886550765]
  Multimodal Large Language Models (MLLMs) have demonstrated significant advances in visual understanding tasks involving both images and videos.<n>Existing human-centric benchmarks predominantly emphasize video generation quality and action recognition, while overlooking essential perceptual and cognitive abilities required in human-centered scenarios.<n>We propose a rigorously curated benchmark designed to provide a more holistic evaluation of MLLMs in human-centric video understanding.
  arXiv  Detail & Related papers  (2025-07-07T11:52:24Z)
• Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialogues [54.81155589931697]
  Collaborative Instance object Navigation (CoIN) is a new task setting where the agent actively resolve uncertainties about the target instance.<n>We propose a novel training-free method, Agent-user Interaction with UncerTainty Awareness (AIUTA)<n>First, upon object detection, a Self-Questioner model initiates a self-dialogue within the agent to obtain a complete and accurate observation description.<n>An Interaction Trigger module determines whether to ask a question to the human, continue or halt navigation.
  arXiv  Detail & Related papers  (2024-12-02T08:16:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.