EgoBlind: Towards Egocentric Visual Assistance for the Blind People

• URL: http://arxiv.org/abs/2503.08221v1
• Date: Tue, 11 Mar 2025 09:40:31 GMT
• Title: EgoBlind: Towards Egocentric Visual Assistance for the Blind People
• Authors: Junbin Xiao, Nanxin Huang, Hao Qiu, Zhulin Tao, Xun Yang, Richang Hong, Meng Wang, Angela Yao,
• Abstract summary: EgoBlind is the first egocentric VideoQA dataset collected from blind individuals.<n>It comprises 1,210 videos that record the daily lives of real blind users from a first-person perspective.<n>It also features 4,927 questions directly posed or generated by blind individuals to reflect their needs for visual assistance.
• Score: 69.6161191190939
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We present EgoBlind, the first egocentric VideoQA dataset collected from blind individuals to evaluate the assistive capabilities of contemporary multimodal large language models (MLLMs). EgoBlind comprises 1,210 videos that record the daily lives of real blind users from a first-person perspective. It also features 4,927 questions directly posed or generated and verified by blind individuals to reflect their needs for visual assistance under various scenarios. We provide each question with an average of 3 reference answers to alleviate subjective evaluation. Using EgoBlind, we comprehensively evaluate 15 leading MLLMs and find that all models struggle, with the best performers achieving accuracy around 56\%, far behind human performance of 87.4\%. To guide future advancements, we identify and summarize major limitations of existing MLLMs in egocentric visual assistance for the blind and provide heuristic suggestions for improvement. With these efforts, we hope EgoBlind can serve as a valuable foundation for developing more effective AI assistants to enhance the independence of the blind individuals' lives.

Related papers

• GuideDog: A Real-World Egocentric Multimodal Dataset for Blind and Low-Vision Accessibility-Aware Guidance [18.467461615621872]
  Mobility remains a significant challenge for the 2.2 billion people worldwide affected by blindness and low vision (BLV) We introduce GuideDog, a novel accessibility-aware guide dataset containing 22K image-description pairs. We also develop GuideDogQA, a subset of 818 samples featuring multiple-choice questions designed to evaluate fine-grained visual perception capabilities.
  arXiv  Detail & Related papers  (2025-03-17T05:43:40Z)
• Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understanding [69.96199605596138]
  Current MLLMs primarily focus on third-person (exocentric) vision, overlooking the unique aspects of first-person (egocentric) videos. We propose learning the mapping between exocentric and egocentric domains to enhance egocentric video understanding. We introduce Ego-ExoClip, a pre-training dataset comprising 1.1M synchronized ego-exo clip-text pairs.
  arXiv  Detail & Related papers  (2025-03-12T08:10:33Z)
• EgoLife: Towards Egocentric Life Assistant [60.51196061794498]
  We introduce EgoLife, a project to develop an egocentric life assistant that accompanies and enhances personal efficiency through AI-powered wearable glasses.<n>We conduct a comprehensive data collection study where six participants lived together for one week, continuously recording their daily activities using AI glasses for multimodal egocentric video capture, along with synchronized third-person-view video references.<n>This effort resulted in the EgoLife dataset, a comprehensive 300-hour egocentric, interpersonal, multiview, and multimodal daily life dataset with intensive annotation.<n>We introduce EgoLifeQA, a suite of long-context, life-oriented question-answering tasks designed to provide
  arXiv  Detail & Related papers  (2025-03-05T18:54:16Z)
• VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment [55.7956150385255]
  We investigate the efficacy of AI feedback to scale supervision for aligning vision-language models. We introduce VLFeedback, the first large-scale vision-language feedback dataset. We train Silkie, an LVLM fine-tuned via direct preference optimization on VLFeedback.
  arXiv  Detail & Related papers  (2024-10-12T07:56:47Z)
• MM-Ego: Towards Building Egocentric Multimodal LLMs [72.47344411599322]
  This research aims to explore building a multimodal foundation model for egocentric video understanding. We develop a data engine that efficiently generates 7M high-quality QA samples for egocentric videos ranging from 30 seconds to one hour long, based on human-annotated data. We contribute a challenging egocentric QA benchmark with 629 videos and 7,026 questions to evaluate the models' ability in recognizing and memorizing visual details across videos of varying lengths.
  arXiv  Detail & Related papers  (2024-10-09T17:59:59Z)
• MACAROON: Training Vision-Language Models To Be Your Engaged Partners [95.32771929749514]
  Large vision-language models (LVLMs) generate detailed responses even when questions are ambiguous or unlabeled. In this study, we aim to shift LVLMs from passive answer providers to proactive engaged partners. We introduce MACAROON, self-iMaginAtion for ContrAstive pReference OptimizatiON, which instructs LVLMs to autonomously generate contrastive response pairs for unlabeled questions.
  arXiv  Detail & Related papers  (2024-06-20T09:27:33Z)
• EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models [21.410065053609877]
  Vision-language models (VLMs) have recently shown promising results in traditional downstream tasks. EgoThink is a novel visual question-answering benchmark that encompasses six core capabilities with twelve detailed dimensions.
  arXiv  Detail & Related papers  (2023-11-27T07:44:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.