Mapillary Vistas Validation for Fine-Grained Traffic Signs: A Benchmark Revealing Vision-Language Model Limitations

• URL: http://arxiv.org/abs/2508.02047v1
• Date: Mon, 04 Aug 2025 04:29:06 GMT
• Title: Mapillary Vistas Validation for Fine-Grained Traffic Signs: A Benchmark Revealing Vision-Language Model Limitations
• Abstract summary: We present a new validation set for traffic signs derived from the Mapillary dataset called Mapillary Vistas Validation for Traffic Signs (MVV)<n>The dataset includes pixel-level instance masks and has been manually annotated by expert annotators to ensure label fidelity.<n>We benchmark several state-of-the-art VLMs against the self-supervised DINOv2 model on this dataset and show that DINOv2 consistently outperforms all VLM baselines.
• Score: 5.159407277301709
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Obtaining high-quality fine-grained annotations for traffic signs is critical for accurate and safe decision-making in autonomous driving. Widely used datasets, such as Mapillary, often provide only coarse-grained labels - without distinguishing semantically important types such as stop signs or speed limit signs. To this end, we present a new validation set for traffic signs derived from the Mapillary dataset called Mapillary Vistas Validation for Traffic Signs (MVV), where we decompose composite traffic signs into granular, semantically meaningful categories. The dataset includes pixel-level instance masks and has been manually annotated by expert annotators to ensure label fidelity. Further, we benchmark several state-of-the-art VLMs against the self-supervised DINOv2 model on this dataset and show that DINOv2 consistently outperforms all VLM baselines-not only on traffic sign recognition, but also on heavily represented categories like vehicles and humans. Our analysis reveals significant limitations in current vision-language models for fine-grained visual understanding and establishes DINOv2 as a strong baseline for dense semantic matching in autonomous driving scenarios. This dataset and evaluation framework pave the way for more reliable, interpretable, and scalable perception systems. Code and data are available at: https://github.com/nec-labs-ma/relabeling

Related papers

• Multimodal Reasoning with LLM for Encrypted Traffic Interpretation: A Benchmark [12.257114438374975]
  This paper proposes an end-to-end traffic-language representation framework (mmTraffic), a multimodal reasoning architecture bridging physical traffic encoding and semantic interpretation.<n>By a perception-centered traffic encoder and a cognition-centered LLM generator, mmTraffic achieves refined traffic interpretation with guaranteed category prediction.
  arXiv  Detail & Related papers  (2026-04-09T11:56:28Z)
• Traffic Sign Recognition in Autonomous Driving: Dataset, Benchmark, and Field Experiment [15.782194207723975]
  Traffic Sign Recognition (TSR) is a core perception capability for autonomous driving.<n>We present TS-1M, a large-scale and globally diverse traffic sign dataset.<n>We conduct a unified benchmark across three representative learning paradigms.
  arXiv  Detail & Related papers  (2026-03-24T10:11:27Z)
• OSM-based Domain Adaptation for Remote Sensing VLMs [49.91326341200221]
  Vision-Language Models (VLMs) adapted to remote sensing rely heavily on domain-specific image-text supervision.<n>We propose OSMDA: a self-contained domain adaptation framework that eliminates this dependency.<n>We conduct exhaustive evaluations spanning 10 benchmarks across image-text-to-text tasks.
  arXiv  Detail & Related papers  (2026-03-12T11:08:30Z)
• Navigating in the Dark: A Multimodal Framework and Dataset for Nighttime Traffic Sign Recognition [9.151417330446591]
  INTSD is a large-scale dataset of street-level night-time images of traffic signboards collected across diverse regions of India.<n>We propose LENS-Net, which integrates an adaptive image enhancement detector for joint illumination correction and sign localization.<n>Our method surpasses existing frameworks, with ablation studies confirming the effectiveness of its key components.
  arXiv  Detail & Related papers  (2025-11-21T12:04:53Z)
• VISAT: Benchmarking Adversarial and Distribution Shift Robustness in Traffic Sign Recognition with Visual Attributes [18.89029074477177]
  We present a novel open dataset and benchmarking suite for evaluating model robustness in the task of traffic sign recognition with the presence of visual attributes.<n>Our dataset introduces two benchmarks that respectively emphasize robustness against adversarial attacks and distribution shifts.<n>Our experiments focus on two major backbones, ResNet-152 and ViT-B/32, and compare the performance between base and MTL models.
  arXiv  Detail & Related papers  (2025-10-29T18:31:21Z)
• SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models [63.71984266104757]
  We propose SafeAuto, a framework that enhances MLLM-based autonomous driving by incorporating both unstructured and structured knowledge.<n>To explicitly integrate safety knowledge, we develop a reasoning component that translates traffic rules into first-order logic.<n>Our Multimodal Retrieval-Augmented Generation model leverages video, control signals, and environmental attributes to learn from past driving experiences.
  arXiv  Detail & Related papers  (2025-02-28T21:53:47Z)
• Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach [56.55633052479446]
  Web-scale visual entity recognition presents significant challenges due to the lack of clean, large-scale training data. We propose a novel methodology to curate such a dataset, leveraging a multimodal large language model (LLM) for label verification, metadata generation, and rationale explanation. Experiments demonstrate that models trained on this automatically curated data achieve state-of-the-art performance on web-scale visual entity recognition tasks.
  arXiv  Detail & Related papers  (2024-10-31T06:55:24Z)
• Cross-domain Few-shot In-context Learning for Enhancing Traffic Sign Recognition [49.20086587208214]
  We propose a cross-domain few-shot in-context learning method based on the MLLM for enhancing traffic sign recognition. By using description texts, our method reduces the cross-domain differences between template and real traffic signs. Our approach requires only simple and uniform textual indications, without the need for large-scale traffic sign images and labels.
  arXiv  Detail & Related papers  (2024-07-08T10:51:03Z)
• MUSES: The Multi-Sensor Semantic Perception Dataset for Driving under Uncertainty [46.369657697892634]
  We introduce MUSES, the MUlti-SEnsor Semantic perception dataset for driving in adverse conditions under increased uncertainty. The dataset integrates a frame camera, a lidar, a radar, an event camera, and an IMU/GNSS sensor. MUSES proves both effective for training and challenging for evaluating models under diverse visual conditions.
  arXiv  Detail & Related papers  (2024-01-23T13:43:17Z)
• Fusing Pseudo Labels with Weak Supervision for Dynamic Traffic Scenarios [0.0]
  We introduce a weakly-supervised label unification pipeline that amalgamates pseudo labels from object detection models trained on heterogeneous datasets. Our pipeline engenders a unified label space through the amalgamation of labels from disparate datasets, rectifying bias and enhancing generalization. We retrain a solitary object detection model using the merged label space, culminating in a resilient model proficient in dynamic traffic scenarios.
  arXiv  Detail & Related papers  (2023-08-30T11:33:07Z)
• OpenLane-V2: A Topology Reasoning Benchmark for Unified 3D HD Mapping [84.65114565766596]
  We present OpenLane-V2, the first dataset on topology reasoning for traffic scene structure. OpenLane-V2 consists of 2,000 annotated road scenes that describe traffic elements and their correlation to the lanes. We evaluate various state-of-the-art methods, and present their quantitative and qualitative results on OpenLane-V2 to indicate future avenues for investigating topology reasoning in traffic scenes.
  arXiv  Detail & Related papers  (2023-04-20T16:31:22Z)
• Traffic Scene Parsing through the TSP6K Dataset [109.69836680564616]
  We introduce a specialized traffic monitoring dataset, termed TSP6K, with high-quality pixel-level and instance-level annotations. The dataset captures more crowded traffic scenes with several times more traffic participants than the existing driving scenes. We propose a detail refining decoder for scene parsing, which recovers the details of different semantic regions in traffic scenes.
  arXiv  Detail & Related papers  (2023-03-06T02:05:14Z)
• ConMAE: Contour Guided MAE for Unsupervised Vehicle Re-Identification [8.950873153831735]
  This work designs a Contour Guided Masked Autoencoder for Unsupervised Vehicle Re-Identification (ConMAE) Considering that Masked Autoencoder (MAE) has shown excellent performance in self-supervised learning, this work designs a Contour Guided Masked Autoencoder for Unsupervised Vehicle Re-Identification (ConMAE)
  arXiv  Detail & Related papers  (2023-02-11T12:10:25Z)
• TrafficCAM: A Versatile Dataset for Traffic Flow Segmentation [9.744937939618161]
  Existing traffic flow datasets have two major limitations. They feature a limited number of classes, usually limited to one type of vehicle, and the scarcity of unlabelled data. We introduce a new benchmark traffic flow image dataset called TrafficCAM.
  arXiv  Detail & Related papers  (2022-11-17T16:14:38Z)
• Pluggable Weakly-Supervised Cross-View Learning for Accurate Vehicle Re-Identification [53.6218051770131]
  Cross-view consistent feature representation is key for accurate vehicle ReID. Existing approaches resort to supervised cross-view learning using extensive extra viewpoints annotations. We present a pluggable Weakly-supervised Cross-View Learning (WCVL) module for vehicle ReID.
  arXiv  Detail & Related papers  (2021-03-09T11:51:09Z)
• Towards Accurate Vehicle Behaviour Classification With Multi-Relational Graph Convolutional Networks [22.022759283770377]
  We propose a pipeline for understanding vehicle behaviour from a monocular image sequence or video. A temporal sequence of such encodings is fed to a recurrent network to label vehicle behaviours. The proposed framework can classify a variety of vehicle behaviours to high fidelity on datasets that are diverse.
  arXiv  Detail & Related papers  (2020-02-03T14:34:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.