X-LeBench: A Benchmark for Extremely Long Egocentric Video Understanding

• URL: http://arxiv.org/abs/2501.06835v1
• Date: Sun, 12 Jan 2025 15:07:03 GMT
• Title: X-LeBench: A Benchmark for Extremely Long Egocentric Video Understanding
• Authors: Wenqi Zhou, Kai Cao, Hao Zheng, Xinyi Zheng, Miao Liu, Per Ola Kristensson, Walterio Mayol-Cuevas, Fan Zhang, Weizhe Lin, Junxiao Shen,
• Abstract summary: Long-form egocentric video understanding provides rich contextual information and insights into long-term human behaviors. Existing benchmark datasets primarily focus on single, short-duration videos or moderately long videos up to dozens of minutes. We introduce X-LeBench, a novel benchmark dataset specifically crafted for evaluating tasks on extremely long egocentric video recordings.
• Score: 25.85614872348223
• License:
• Abstract: Long-form egocentric video understanding provides rich contextual information and unique insights into long-term human behaviors, holding significant potential for applications in embodied intelligence, long-term activity analysis, and personalized assistive technologies. However, existing benchmark datasets primarily focus on single, short-duration videos or moderately long videos up to dozens of minutes, leaving a substantial gap in evaluating extensive, ultra-long egocentric video recordings. To address this, we introduce X-LeBench, a novel benchmark dataset specifically crafted for evaluating tasks on extremely long egocentric video recordings. Leveraging the advanced text processing capabilities of large language models (LLMs), X-LeBench develops a life-logging simulation pipeline that produces realistic, coherent daily plans aligned with real-world video data. This approach enables the flexible integration of synthetic daily plans with real-world footage from Ego4D-a massive-scale egocentric video dataset covers a wide range of daily life scenarios-resulting in 432 simulated video life logs that mirror realistic daily activities in contextually rich scenarios. The video life-log durations span from 23 minutes to 16.4 hours. The evaluation of several baseline systems and multimodal large language models (MLLMs) reveals their poor performance across the board, highlighting the inherent challenges of long-form egocentric video understanding and underscoring the need for more advanced models.

Related papers

• HLV-1K: A Large-scale Hour-Long Video Benchmark for Time-Specific Long Video Understanding [52.696422425058245]
  We build a large-scale hour-long long video benchmark, HLV-1K, designed to evaluate long video understanding models. HLV-1K comprises 1009 hour-long videos with 14,847 high-quality question answering (QA) and multi-choice question asnwering (MCQA) We evaluate our benchmark using existing state-of-the-art methods and demonstrate its value for testing deep long video understanding capabilities at different levels and for various tasks.
  arXiv  Detail & Related papers  (2025-01-03T05:32:37Z)
• VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLM [81.15525024145697]
  Video Large Language Models (Video LLMs) have recently exhibited remarkable capabilities in general video understanding. However, they mainly focus on holistic comprehension and struggle with capturing fine-grained spatial and temporal details. We introduce the VideoRefer Suite to empower Video LLM for finer-level spatial-temporal video understanding.
  arXiv  Detail & Related papers  (2024-12-31T18:56:46Z)
• SCBench: A Sports Commentary Benchmark for Video LLMs [19.13963551534595]
  We develop a benchmark for sports video commentary generation for Video Large Language Models (Video LLMs) $textbfSCBench$ is a six-dimensional metric specifically designed for our task, upon which we propose a GPT-based evaluation method. Our results found InternVL-Chat-2 achieves the best performance with 5.44, surpassing the second-best by 1.04.
  arXiv  Detail & Related papers  (2024-12-23T15:13:56Z)
• SALOVA: Segment-Augmented Long Video Assistant for Targeted Retrieval and Routing in Long-Form Video Analysis [52.050036778325094]
  We introduce SALOVA: Segment-Augmented Video Assistant, a novel video-LLM framework designed to enhance the comprehension of lengthy video content. We present a high-quality collection of 87.8K long videos, each densely captioned at the segment level to enable models to capture scene continuity and maintain rich context. Our framework mitigates the limitations of current video-LMMs by allowing for precise identification and retrieval of relevant video segments in response to queries.
  arXiv  Detail & Related papers  (2024-11-25T08:04:47Z)
• LVBench: An Extreme Long Video Understanding Benchmark [38.839913137854104]
  We introduce LVBench, a benchmark specifically designed for long video understanding. Our dataset comprises publicly sourced videos and encompasses a diverse set of tasks aimed at long video comprehension and information extraction.
  arXiv  Detail & Related papers  (2024-06-12T09:36:52Z)
• CinePile: A Long Video Question Answering Dataset and Benchmark [55.30860239555001]
  We present a novel dataset and benchmark, CinePile, specifically designed for authentic long-form video understanding. Our comprehensive dataset comprises 305,000 multiple-choice questions (MCQs), covering various visual and multimodal aspects. We fine-tuned open-source Video-LLMs on the training split and evaluated both open-source and proprietary video-centric LLMs on the test split of our dataset.
  arXiv  Detail & Related papers  (2024-05-14T17:59:02Z)
• Scaling Up Video Summarization Pretraining with Large Language Models [73.74662411006426]
  We introduce an automated and scalable pipeline for generating a large-scale video summarization dataset. We analyze the limitations of existing approaches and propose a new video summarization model that effectively addresses them. Our work also presents a new benchmark dataset that contains 1200 long videos each with high-quality summaries annotated by professionals.
  arXiv  Detail & Related papers  (2024-04-04T11:59:06Z)
• LongVLM: Efficient Long Video Understanding via Large Language Models [55.813206751150716]
  LongVLM is a simple yet powerful VideoLLM for long video understanding. We encode video representations that incorporate both local and global information. Our model produces more precise responses for long video understanding.
  arXiv  Detail & Related papers  (2024-04-04T11:33:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.