On the calibration of underrepresented classes in LiDAR-based semantic segmentation

• URL: http://arxiv.org/abs/2210.06811v1
• Date: Thu, 13 Oct 2022 07:49:24 GMT
• Title: On the calibration of underrepresented classes in LiDAR-based semantic segmentation
• Authors: Mariella Dreissig and Florian Piewak and Joschka Boedecker
• Abstract summary: This work focuses on a class-wise evaluation of several models' confidence performance for LiDAR-based semantic segmentation. We compare the calibration abilities of three semantic segmentation models with different architectural concepts. By identifying and describing the dependency between the predictive performance of a class and the respective calibration quality we aim to facilitate the model selection and refinement for safety-critical applications.
• Score: 7.100396757261104
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The calibration of deep learning-based perception models plays a crucial role in their reliability. Our work focuses on a class-wise evaluation of several model's confidence performance for LiDAR-based semantic segmentation with the aim of providing insights into the calibration of underrepresented classes. Those classes often include VRUs and are thus of particular interest for safety reasons. With the help of a metric based on sparsification curves we compare the calibration abilities of three semantic segmentation models with different architectural concepts, each in a in deterministic and a probabilistic version. By identifying and describing the dependency between the predictive performance of a class and the respective calibration quality we aim to facilitate the model selection and refinement for safety-critical applications.

Related papers

• Calibrating Large Language Models with Sample Consistency [76.23956851098598]
  We explore the potential of deriving confidence from the distribution of multiple randomly sampled model generations, via three measures of consistency. Results show that consistency-based calibration methods outperform existing post-hoc approaches. We offer practical guidance on choosing suitable consistency metrics for calibration, tailored to the characteristics of various LMs.
  arXiv  Detail & Related papers  (2024-02-21T16:15:20Z)
• On the Calibration of Large Language Models and Alignment [63.605099174744865]
  Confidence calibration serves as a crucial tool for gauging the reliability of deep models. We conduct a systematic examination of the calibration of aligned language models throughout the entire construction process. Our work sheds light on whether popular LLMs are well-calibrated and how the training process influences model calibration.
  arXiv  Detail & Related papers  (2023-11-22T08:57:55Z)
• On the Calibration of Uncertainty Estimation in LiDAR-based Semantic Segmentation [7.100396757261104]
  We propose a metric to measure the confidence calibration quality of a semantic segmentation model with respect to individual classes. We additionally suggest a double use for the method to automatically find label problems to improve the quality of hand- or auto-annotated datasets.
  arXiv  Detail & Related papers  (2023-08-04T10:59:24Z)
• Enabling Calibration In The Zero-Shot Inference of Large Vision-Language Models [58.720142291102135]
  We measure calibration across relevant variables like prompt, dataset, and architecture, and find that zero-shot inference with CLIP is miscalibrated. A single learned temperature generalizes for each specific CLIP model across inference dataset and prompt choice.
  arXiv  Detail & Related papers  (2023-03-11T17:14:04Z)
• Benchmarking the Robustness of LiDAR Semantic Segmentation Models [78.6597530416523]
  In this paper, we aim to comprehensively analyze the robustness of LiDAR semantic segmentation models under various corruptions. We propose a new benchmark called SemanticKITTI-C, which features 16 out-of-domain LiDAR corruptions in three groups, namely adverse weather, measurement noise and cross-device discrepancy. We design a robust LiDAR segmentation model (RLSeg) which greatly boosts the robustness with simple but effective modifications.
  arXiv  Detail & Related papers  (2023-01-03T06:47:31Z)
• On Calibrating Semantic Segmentation Models: Analyses and An Algorithm [51.85289816613351]
  We study the problem of semantic segmentation calibration. Model capacity, crop size, multi-scale testing, and prediction correctness have impact on calibration. We propose a simple, unifying, and effective approach, namely selective scaling.
  arXiv  Detail & Related papers  (2022-12-22T22:05:16Z)
• Variable-Based Calibration for Machine Learning Classifiers [11.9995808096481]
  We introduce the notion of variable-based calibration to characterize calibration properties of a model. We find that models with near-perfect expected calibration error can exhibit significant miscalibration as a function of features of the data.
  arXiv  Detail & Related papers  (2022-09-30T00:49:31Z)
• Using Representation Expressiveness and Learnability to Evaluate Self-Supervised Learning Methods [61.49061000562676]
  We introduce Cluster Learnability (CL) to assess learnability. CL is measured in terms of the performance of a KNN trained to predict labels obtained by clustering the representations with K-means. We find that CL better correlates with in-distribution model performance than other competing recent evaluation schemes.
  arXiv  Detail & Related papers  (2022-06-02T19:05:13Z)
• Post-hoc Models for Performance Estimation of Machine Learning Inference [22.977047604404884]
  Estimating how well a machine learning model performs during inference is critical in a variety of scenarios. We systematically generalize performance estimation to a diverse set of metrics and scenarios. We find that proposed post-hoc models consistently outperform the standard confidence baselines.
  arXiv  Detail & Related papers  (2021-10-06T02:20:37Z)
• Semi-Supervised Disentanglement of Class-Related and Class-Independent Factors in VAE [4.533408938245526]
  We propose a framework capable of disentangling class-related and class-independent factors of variation in data. Our framework employs an attention mechanism in its latent space in order to improve the process of extracting class-related factors from data. Experiments show that our framework disentangles class-related and class-independent factors of variation and learns interpretable features.
  arXiv  Detail & Related papers  (2021-02-01T15:05:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.