Multi-modality Anomaly Segmentation on the Road

• URL: http://arxiv.org/abs/2503.17712v1
• Date: Sat, 22 Mar 2025 09:55:42 GMT
• Title: Multi-modality Anomaly Segmentation on the Road
• Abstract summary: Current uni-modal anomaly segmentation frameworks tend to produce high anomaly scores for non-anomalous regions in images.<n>We develop a multi-modal uncertainty-based anomaly segmentation framework, named MMRAS+, for autonomous driving systems.<n>Indeed, MMRAS+ is the first multi-modal anomaly segmentation solution for autonomous driving.
• Score: 39.96773593485109
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Semantic segmentation allows autonomous driving cars to understand the surroundings of the vehicle comprehensively. However, it is also crucial for the model to detect obstacles that may jeopardize the safety of autonomous driving systems. Based on our experiments, we find that current uni-modal anomaly segmentation frameworks tend to produce high anomaly scores for non-anomalous regions in images. Motivated by this empirical finding, we develop a multi-modal uncertainty-based anomaly segmentation framework, named MMRAS+, for autonomous driving systems. MMRAS+ effectively reduces the high anomaly outputs of non-anomalous classes by introducing text-modal using the CLIP text encoder. Indeed, MMRAS+ is the first multi-modal anomaly segmentation solution for autonomous driving. Moreover, we develop an ensemble module to further boost the anomaly segmentation performance. Experiments on RoadAnomaly, SMIYC, and Fishyscapes validation datasets demonstrate the superior performance of our method. The code is available in https://github.com/HengGao12/MMRAS_plus.

Related papers

• Road-Aware Anomaly Segmentation with Query-Guided Polygons and CLIP in Autonomous Driving [0.5371337604556311]
  We present a lightweight, postprocessing, road-aware anomaly segmentation framework that requires no retraining, no OOD data, and no auxiliary supervision.<n>Our approach builds on a mask transformer-based segmentation network by exploiting query-level mask confidence.<n> Evaluation on three public benchmarks-Fishyscapes, SMIYC, and RoadAnomaly-shows consistently strong performance.
  arXiv  Detail & Related papers  (2026-07-05T13:43:31Z)
• OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models [69.2503510410147]
  We present a unified autonomous driving framework built upon a pretrained VLM.<n>We demonstrate that pretrained VLM attention exhibits strong transferability beyond pure language modeling.<n>Experiments on end-to-end autonomous driving benchmarks demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-04-20T07:50:00Z)
• Unsupervised Learning for Detection of Rare Driving Scenarios [1.5399429731150376]
  This research explores an unsupervised learning framework for detecting rare and extreme driving scenarios.<n>We leverage the recently proposed Deep Isolation Forest (DIF) to identify non-linear and complex anomalies.<n>Our results demonstrate that the proposed approach effectively identifies rare and hazardous driving scenarios.
  arXiv  Detail & Related papers  (2025-12-29T16:35:20Z)
• Lane-Wise Highway Anomaly Detection [8.086502588472783]
  This paper proposes a scalable and interpretable framework for lane-wise highway traffic anomaly detection.<n>Unlike traditional sensor-dependent methods, our approach uses AI-powered vision models to extract lane-specific features.<n>Our framework outperforms state-of-the-art methods in precision, recall, and F1-score.
  arXiv  Detail & Related papers  (2025-05-05T12:32:23Z)
• Segmenting Objectiveness and Task-awareness Unknown Region for Autonomous Driving [46.70405993442064]
  We propose a novel framework termed Segmenting Objectiveness and Task-Awareness (SOTA) for autonomous driving scenes. SOTA enhances the segmentation of objectiveness through a Semantic Fusion Block (SFB) and filters anomalies irrelevant to road navigation tasks.
  arXiv  Detail & Related papers  (2025-04-27T10:08:54Z)
• DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understanding [76.3876070043663]
  We propose DriveLMM-o1, a dataset and benchmark designed to advance step-wise visual reasoning for autonomous driving.<n>Our benchmark features over 18k VQA examples in the training set and more than 4k in the test set, covering diverse questions on perception, prediction, and planning.<n>Our model achieves a +7.49% gain in final answer accuracy, along with a 3.62% improvement in reasoning score over the previous best open-source model.
  arXiv  Detail & Related papers  (2025-03-13T17:59:01Z)
• SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models [63.71984266104757]
  Multimodal Large Language Models (MLLMs) can process both visual and textual data.<n>We propose SafeAuto, a novel framework that enhances MLLM-based autonomous driving systems by incorporating both unstructured and structured knowledge.
  arXiv  Detail & Related papers  (2025-02-28T21:53:47Z)
• Driver Assistance System Based on Multimodal Data Hazard Detection [0.0]
  This paper proposes a multimodal driver assistance detection system.<n>It integrates road condition video, driver facial video, and audio data to enhance incident recognition accuracy.
  arXiv  Detail & Related papers  (2025-02-05T09:02:39Z)
• M2DA: Multi-Modal Fusion Transformer Incorporating Driver Attention for Autonomous Driving [11.36165122994834]
  We propose a Multi-Modal fusion transformer incorporating Driver Attention (M2DA) for autonomous driving. By incorporating driver attention, we empower the human-like scene understanding ability to autonomous vehicles to identify crucial areas precisely and ensure safety.
  arXiv  Detail & Related papers  (2024-03-19T08:54:52Z)
• Leveraging Driver Field-of-View for Multimodal Ego-Trajectory Prediction [69.29802752614677]
  RouteFormer is a novel ego-trajectory prediction network combining GPS data, environmental context, and the driver's field-of-view. To tackle data scarcity and enhance diversity, we introduce GEM, a dataset of urban driving scenarios enriched with synchronized driver field-of-view and gaze data.
  arXiv  Detail & Related papers  (2023-12-13T23:06:30Z)
• Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models [114.69732301904419]
  We present an approach to apply end-to-end open-set (any environment/scene) autonomous driving that is capable of providing driving decisions from representations queryable by image and text. Our approach demonstrates unparalleled results in diverse tests while achieving significantly greater robustness in out-of-distribution situations.
  arXiv  Detail & Related papers  (2023-10-26T17:56:35Z)
• Multi-task UNet architecture for end-to-end autonomous driving [0.0]
  We propose an end-to-end driving model that integrates a multi-task UNet (MTUNet) architecture and control algorithms in a pipeline of data flow from a front camera through this model to driving decisions. It provides quantitative measures to evaluate the holistic, dynamic, and real-time performance of end-to-end driving systems and thus the safety and interpretability of MTUNet.
  arXiv  Detail & Related papers  (2021-12-16T15:35:15Z)
• Dual-Modality Vehicle Anomaly Detection via Bilateral Trajectory Tracing [42.03797195839054]
  We propose a dual-modality modularized methodology for the robust detection of abnormal vehicles. For the vehicle detection and tracking module, we adopted YOLOv5 and multi-scale tracking to localize the anomalies. Experiments conducted on the Track 4 testset of the NVIDIA 2021 AI City Challenge yielded a result of 0.9302 F1-Score and 3.4039 root mean square error (RMSE)
  arXiv  Detail & Related papers  (2021-06-09T12:04:25Z)
• A Driving Behavior Recognition Model with Bi-LSTM and Multi-Scale CNN [59.57221522897815]
  We propose a neural network model based on trajectories information for driving behavior recognition. We evaluate the proposed model on the public BLVD dataset, achieving a satisfying performance.
  arXiv  Detail & Related papers  (2021-03-01T06:47:29Z)
• Detecting 32 Pedestrian Attributes for Autonomous Vehicles [103.87351701138554]
  In this paper, we address the problem of jointly detecting pedestrians and recognizing 32 pedestrian attributes. We introduce a Multi-Task Learning (MTL) model relying on a composite field framework, which achieves both goals in an efficient way. We show competitive detection and attribute recognition results, as well as a more stable MTL training.
  arXiv  Detail & Related papers  (2020-12-04T15:10:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.