Perception Characteristics Distance: Measuring Stability and Robustness of Perception System in Dynamic Conditions under a Certain Decision Rule

• URL: http://arxiv.org/abs/2506.09217v1
• Date: Tue, 10 Jun 2025 20:22:09 GMT
• Title: Perception Characteristics Distance: Measuring Stability and Robustness of Perception System in Dynamic Conditions under a Certain Decision Rule
• Authors: Boyu Jiang, Liang Shi, Zhengzhi Lin, Loren Stowe, Feng Guo,
• Abstract summary: We introduce the Perception Distance Characteristics (PCD) -- a novel evaluation metric that quantifies the farthest distance at which an object can be reliably detected.<n>We present the SensorRainFall dataset, collected on the Virginia Smart Road using a sensor-equipped vehicle.
• Score: 5.513141175056931
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The performance of perception systems in autonomous driving systems (ADS) is strongly influenced by object distance, scene dynamics, and environmental conditions such as weather. AI-based perception outputs are inherently stochastic, with variability driven by these external factors, while traditional evaluation metrics remain static and event-independent, failing to capture fluctuations in confidence over time. In this work, we introduce the Perception Characteristics Distance (PCD) -- a novel evaluation metric that quantifies the farthest distance at which an object can be reliably detected, incorporating uncertainty in model outputs. To support this, we present the SensorRainFall dataset, collected on the Virginia Smart Road using a sensor-equipped vehicle (cameras, radar, LiDAR) under controlled daylight-clear and daylight-rain scenarios, with precise ground-truth distances to the target objects. Statistical analysis reveals the presence of change points in the variance of detection confidence score with distance. By averaging the PCD values across a range of detection quality thresholds and probabilistic thresholds, we compute the mean PCD (mPCD), which captures the overall perception characteristics of a system with respect to detection distance. Applying state-of-the-art perception models shows that mPCD captures meaningful reliability differences under varying weather conditions -- differences that static metrics overlook. PCD provides a principled, distribution-aware measure of perception performance, supporting safer and more robust ADS operation, while the SensorRainFall dataset offers a valuable benchmark for evaluation. The SensorRainFall dataset is publicly available at https://www.kaggle.com/datasets/datadrivenwheels/sensorrainfall, and the evaluation code is open-sourced at https://github.com/datadrivenwheels/PCD_Python.

Related papers

• Experimental Evaluation of Road-Crossing Decisions by Autonomous Wheelchairs against Environmental Factors [42.90509901417468]
  We focus on the fine-tuning of tracking performance and on its experimental evaluation against outdoor environmental factors. We show that the approach can be adopted to evaluate video tracking and event detection robustness against outdoor environmental factors.
  arXiv  Detail & Related papers  (2024-05-27T08:43:26Z)
• OOSTraj: Out-of-Sight Trajectory Prediction With Vision-Positioning Denoising [49.86409475232849]
  Trajectory prediction is fundamental in computer vision and autonomous driving. Existing approaches in this field often assume precise and complete observational data. We present a novel method for out-of-sight trajectory prediction that leverages a vision-positioning technique.
  arXiv  Detail & Related papers  (2024-04-02T18:30:29Z)
• Environment-independent mmWave Fall Detection with Interacting Multiple Model [1.9358739203360094]
  mmWave radar is a promising candidate technology for its privacy-preserving and non-contact manner. FADE is a practical fall detection radar system with enhanced accuracy and robustness in real-world scenarios.
  arXiv  Detail & Related papers  (2023-11-15T07:49:46Z)
• DARTH: Holistic Test-time Adaptation for Multiple Object Tracking [87.72019733473562]
  Multiple object tracking (MOT) is a fundamental component of perception systems for autonomous driving. Despite the urge of safety in driving systems, no solution to the MOT adaptation problem to domain shift in test-time conditions has ever been proposed. We introduce DARTH, a holistic test-time adaptation framework for MOT.
  arXiv  Detail & Related papers  (2023-10-03T10:10:42Z)
• On the Importance of Accurate Geometry Data for Dense 3D Vision Tasks [61.74608497496841]
  Training on inaccurate or corrupt data induces model bias and hampers generalisation capabilities. This paper investigates the effect of sensor errors for the dense 3D vision tasks of depth estimation and reconstruction.
  arXiv  Detail & Related papers  (2023-03-26T22:32:44Z)
• A Quality Index Metric and Method for Online Self-Assessment of Autonomous Vehicles Sensory Perception [164.93739293097605]
  We propose a novel evaluation metric, named as the detection quality index (DQI), which assesses the performance of camera-based object detection algorithms. We have developed a superpixel-based attention network (SPA-NET) that utilizes raw image pixels and superpixels as input to predict the proposed DQI evaluation metric.
  arXiv  Detail & Related papers  (2022-03-04T22:16:50Z)
• Lidar Light Scattering Augmentation (LISA): Physics-based Simulation of Adverse Weather Conditions for 3D Object Detection [60.89616629421904]
  Lidar-based object detectors are critical parts of the 3D perception pipeline in autonomous navigation systems such as self-driving cars. They are sensitive to adverse weather conditions such as rain, snow and fog due to reduced signal-to-noise ratio (SNR) and signal-to-background ratio (SBR)
  arXiv  Detail & Related papers  (2021-07-14T21:10:47Z)
• On the Role of Sensor Fusion for Object Detection in Future Vehicular Networks [25.838878314196375]
  We evaluate how using a combination of different sensors affects the detection of the environment in which the vehicles move and operate. The final objective is to identify the optimal setup that would minimize the amount of data to be distributed over the channel.
  arXiv  Detail & Related papers  (2021-04-23T18:58:37Z)
• SSTN: Self-Supervised Domain Adaptation Thermal Object Detection for Autonomous Driving [6.810856082577402]
  We have proposed a deep neural network Self Supervised Thermal Network (SSTN) for learning the feature embedding to maximize the information between visible and infrared spectrum domain by contrastive learning. The proposed method is extensively evaluated on the two publicly available datasets: the FLIR-ADAS dataset and the KAIST Multi-Spectral dataset.
  arXiv  Detail & Related papers  (2021-03-04T16:42:49Z)
• Towards robust sensing for Autonomous Vehicles: An adversarial perspective [82.83630604517249]
  It is of primary importance that the resulting decisions are robust to perturbations. Adversarial perturbations are purposefully crafted alterations of the environment or of the sensory measurements. A careful evaluation of the vulnerabilities of their sensing system(s) is necessary in order to build and deploy safer systems.
  arXiv  Detail & Related papers  (2020-07-14T05:25:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.