LongEgoRefer: A Benchmark for Long-Form Egocentric Video Referring Expression Comprehension

• URL: http://arxiv.org/abs/2607.02096v1
• Date: Thu, 02 Jul 2026 12:32:53 GMT
• Title: LongEgoRefer: A Benchmark for Long-Form Egocentric Video Referring Expression Comprehension
• Abstract summary: LongEgoRefer is a novel benchmark constructed from long-form videos in the Ego4D dataset.<n>It exhibits detailed linguistic descriptions, and complex human-object interactions embedded in long, dynamic egocentric narratives.<n>It defines a demanding-temporal grounding problem that requires models to identify both intrinsic when an event occurs and where the referred object appears within extended video sequences.
• Score: 41.229006294351315
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Egocentric videos capture rich and diverse human-object interactions and have emerged as a fundamental resource for understanding human activities related to objects. In this context, Video Referring Expression Comprehension (Video REC), the task of localizing the temporal and spatial extent of a referred object in video frames given a natural language query, plays a key role in linking textual descriptions to observed objects in untrimmed egocentric recordings. However, existing egocentric Video REC benchmarks primarily focus on short video clips, where some target object appears densely within frames. Such settings do not reflect real-world egocentric recordings, which are long-form, untrimmed, and characterized by sparse object occurrences and complex activity transitions. To address this limitation, we introduce LongEgoRefer, a novel and challenging benchmark constructed from long-form videos in the Ego4D dataset. LongEgoRefer contains 1,498 referring expressions with an average video duration of 45 minutes. The benchmark exhibits extreme target sparsity, detailed linguistic descriptions, and complex human-object interactions embedded in long, dynamic egocentric narratives. Consequently, it defines a demanding spatio-temporal grounding problem that requires models to identify both when an event occurs and where the referred object appears within extended video sequences. We evaluate existing Video REC approaches, including training-free baselines based on vision-language models combined with Grounded SAM2. Extensive experiments show that even advanced baselines and current state-of-the-art models struggle significantly on LongEgoRefer. These results highlight the intrinsic difficulty of long-form egocentric spatio-temporal grounding and emphasize the need for more robust video understanding models.

Related papers

• Graph it first! Enabling Reasoning on Long-form Egocentric Videos through Scene Graphs [12.707862116540914]
  We introduce a framework for egocentric video question answering (VQA) that overcomes these input constraints through Egocentric Scene Graphs (EgoSGs)<n>By representing videos as compact, text-based scene graphs, our method preserves the essential visual and temporal information of the original video in a symbolic form.<n>On HD-EPIC VQA, our method achieves state-of-the-art results, outperforming strong video-based baselines on multiple models.
  arXiv  Detail & Related papers  (2026-06-24T13:55:45Z)
• EgoGraph: Temporal Knowledge Graph for Egocentric Video Understanding [11.51428438970598]
  EgoGraph is a training-free and dynamic knowledge-graph construction framework that explicitly encodes long-term, cross-entity dependencies in egocentric video streams.<n>We develop a temporal relational modeling strategy that captures temporal dependencies across entities and accumulates stable long-term memory over multiple days, enabling complex temporal reasoning.
  arXiv  Detail & Related papers  (2026-02-27T06:20:58Z)
• Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Data [100.5266292850922]
  Strefer is a synthetic data generation framework designed to equip Video Large Models with referring and reasoning capabilities.<n>Strefer produces diverse instruction-generation data using a data engine that pseudo-annotates temporally dense, fine-grained video metadata.<n>Our approach enhances the ability of Video LLMs to interpret to spatial and temporal references, fostering more versatile, space-time-aware reasoning essential for real-world AI companions.
  arXiv  Detail & Related papers  (2025-09-03T17:33:20Z)
• Fine-grained Spatiotemporal Grounding on Egocentric Videos [13.319346673043286]
  We introduce EgoMask, the first pixel-level benchmark for fine-temporal grounding in egocentric videos.<n>EgoMask is constructed by our proposed automatic annotation pipeline, which annotates referring expressions and object masks.<n>We also create EgoMask-Train, a large-scale training dataset to facilitate model development.
  arXiv  Detail & Related papers  (2025-08-01T10:53:27Z)
• Infinite Video Understanding [50.78256932424239]
  We argue that framing Infinite Video Understanding as a blue-sky research objective provides a vital north star for the multimedia.<n>We outline the core challenges and key research directions towards achieving this transformative capability.
  arXiv  Detail & Related papers  (2025-07-11T23:07:04Z)
• Long-RVOS: A Comprehensive Benchmark for Long-term Referring Video Object Segmentation [51.2732688481343]
  We introduce Longtextbf-RVOS, a large-scale benchmark for long-term referring object segmentation.<n>Long-RVOS contains 2,000+ videos of an average duration exceeding 60 seconds, covering a variety of objects.<n>Unlike previous benchmarks that rely solely on the per-frame spatial evaluation, we introduce two metrics to assess the temporal andtemporal consistency.
  arXiv  Detail & Related papers  (2025-05-19T04:52:31Z)
• X-LeBench: A Benchmark for Extremely Long Egocentric Video Understanding [25.85614872348223]
  Long-form egocentric video understanding provides rich contextual information and insights into long-term human behaviors.<n>Existing benchmark datasets primarily focus on single, short-duration videos or moderately long videos up to dozens of minutes.<n>We introduce X-LeBench, a novel benchmark dataset specifically crafted for evaluating tasks on extremely long egocentric video recordings.
  arXiv  Detail & Related papers  (2025-01-12T15:07:03Z)
• VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLM [81.15525024145697]
  Video Large Language Models (Video LLMs) have recently exhibited remarkable capabilities in general video understanding.<n>However, they mainly focus on holistic comprehension and struggle with capturing fine-grained spatial and temporal details.<n>We introduce the VideoRefer Suite to empower Video LLM for finer-level spatial-temporal video understanding.
  arXiv  Detail & Related papers  (2024-12-31T18:56:46Z)
• SALOVA: Segment-Augmented Long Video Assistant for Targeted Retrieval and Routing in Long-Form Video Analysis [52.050036778325094]
  We introduce SALOVA: Segment-Augmented Video Assistant, a novel video-LLM framework designed to enhance the comprehension of lengthy video content.<n>We present a high-quality collection of 87.8K long videos, each densely captioned at the segment level to enable models to capture scene continuity and maintain rich context.<n>Our framework mitigates the limitations of current video-LMMs by allowing for precise identification and retrieval of relevant video segments in response to queries.
  arXiv  Detail & Related papers  (2024-11-25T08:04:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.