Urban Risk-Aware Navigation via VQA-Based Event Maps for People with Low Vision

• URL: http://arxiv.org/abs/2605.11782v1
• Date: Tue, 12 May 2026 08:50:13 GMT
• Title: Urban Risk-Aware Navigation via VQA-Based Event Maps for People with Low Vision
• Abstract summary: Visual impairment affects hundreds of millions of people worldwide, severely limiting their ability to navigate urban environments safely and independently.<n>We propose an event map framework based on visual question answering that leverages Vision-Language Models (VLMs) for pedestrian scene description and hazard identification across diverse real-world environments.<n>We benchmark four VQA architectures -ViLT, LLaVA, InstructBLIP, and Qwen-VL and find that generative Multimodal Large Language Models (MLLMs) substantially outperform classification-based approaches.
• Score: 0.5371337604556311
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual impairment affects hundreds of millions of people worldwide, severely limiting their ability to navigate urban environments safely and independently. While wearable assistive devices offer a promising platform for real-time hazard detection, existing approaches rely on task-specific vision pipelines that lack flexibility and generalizability. In this work, we propose an event map framework based on visual question answering that leverages Vision-Language Models (VLMs) for pedestrian scene description and hazard identification across diverse real-world environments, using a three-level hierarchical query structure to enable fine-grained scene understanding without task-specific retraining. Model responses are aggregated into a weighted risk scoring system that maps street segments into four discrete safety categories, producing navigable risk-aware event maps for route planning. To support evaluation and future research, we introduce a geographically diverse dataset spanning 20 cities across six continents, comprising over 800 annotated images and 18,000 answered questions. We benchmark four VQA architectures -ViLT, LLaVA, InstructBLIP, and Qwen-VL- and find that generative Multimodal Large Language Models (MLLMs) substantially outperform classification-based approaches, with Qwen-VL achieving the best overall balance of precision and recall. These results demonstrate the viability of MLLMs as a flexible and generalizable foundation for assistive navigation systems for visually impaired people.

Related papers

• mmWalk: Towards Multi-modal Multi-view Walking Assistance [44.184803877778556]
  mmWalk is a simulated multi-modal dataset that integrates multi-view sensor and accessibility-oriented features for outdoor safe navigation.<n>Our dataset comprises 120 manually controlled, scenario-categorized walking trajectories with 62k synchronized frames.<n>We generate mmWalkVQA, a VQA benchmark with over 69k visual question-answer triplets across 9 categories tailored for safe and informed walking assistance.
  arXiv  Detail & Related papers  (2025-10-13T15:25:52Z)
• Multi-Agent Visual-Language Reasoning for Comprehensive Highway Scene Understanding [5.830619388189558]
  This paper introduces a multi-agent framework for comprehensive highway scene understanding.<n>A large generic vision-language model (VLM) is contextualized with domain knowledge to generate task-specific chain-of-thought prompts.<n>The framework simultaneously addresses weather classification, pavement wetness assessment, and traffic congestion detection.
  arXiv  Detail & Related papers  (2025-08-24T03:55:24Z)
• SafePLUG: Empowering Multimodal LLMs with Pixel-Level Insight and Temporal Grounding for Traffic Accident Understanding [13.157191393518595]
  Multimodal large language models (MLLMs) have achieved remarkable progress across a range of vision-language tasks.<n>We propose SafePLUG, a novel framework that empowers MLLMs with both Pixel-Level Understanding and temporal Grounding for comprehensive traffic accident analysis.<n>We curate a new dataset containing multimodal question-answer pairs centered on diverse accident scenarios, with detailed pixel-level annotations and temporal event boundaries.
  arXiv  Detail & Related papers  (2025-08-09T00:25:24Z)
• Learning to Drive Anywhere with Model-Based Reannotation [49.80796496905606]
  We develop a framework for generalizable visual navigation policies for robots.<n>We leverage passively collected data, including crowd-sourced teleoperation data and unlabeled YouTube videos.<n>This relabeled data is then distilled into LogoNav, a long-horizon navigation policy conditioned on visual goals or GPS waypoints.
  arXiv  Detail & Related papers  (2025-05-08T18:43:39Z)
• ChatBEV: A Visual Language Model that Understands BEV Maps [58.3005092762598]
  We introduce ChatBEV-QA, a novel BEV VQA benchmark containing over 137k questions.<n>This benchmark is constructed using a novel data collection pipeline that generates scalable and informative VQA data for BEV maps.<n>We propose a language-driven traffic scene generation pipeline, where ChatBEV facilitates map understanding and text-aligned navigation guidance.
  arXiv  Detail & Related papers  (2025-03-18T06:12:38Z)
• NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Models [11.184459657989914]
  We introduce NuPlanQA-Eval, a multi-view, multi-modal evaluation benchmark for driving scene understanding.<n>We also propose NuPlanQA-1M, a large-scale dataset comprising 1M real-world visual question-answering (VQA) pairs.<n>Our evaluation results reveal key challenges that existing MLLMs face in driving scene-specific perception and spatial reasoning from ego-centric perspectives.
  arXiv  Detail & Related papers  (2025-03-17T03:12:39Z)
• GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Tasks [84.86699025256705]
  We present GEOBench-VLM, a benchmark specifically designed to evaluate Vision-Language Models (VLMs) on geospatial tasks.<n>Our benchmark features over 10,000 manually verified instructions and spanning diverse visual conditions, object types, and scales.<n>We evaluate several state-of-the-art VLMs to assess performance on geospatial-specific challenges.
  arXiv  Detail & Related papers  (2024-11-28T18:59:56Z)
• DivScene: Towards Open-Vocabulary Object Navigation with Large Vision Language Models in Diverse Scenes [76.24687327731031]
  We first study the challenge of open-vocabulary object navigation by introducing DivScene.<n>Our dataset provides a much greater diversity of target objects and scene types than existing datasets.<n>We fine-tuned LVLMs to predict the next action with CoT explanations.
  arXiv  Detail & Related papers  (2024-10-03T17:49:28Z)
• SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Model [77.86593720792986]
  We propose a Safety Preference Alignment dataset for Vision Language Models named SPA-VL.<n> SPA-VL covers 6 harmfulness domains, 13 categories, and 53 subcategories, and contains 100,788 samples of the quadruple (question, image, chosen response, rejected response)<n>Experiments indicate that models trained with alignment techniques on the SPA-VL dataset exhibit substantial improvements in harmlessness and helpfulness while maintaining core capabilities.
  arXiv  Detail & Related papers  (2024-06-17T18:57:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.