Uncertainty-sensitive Activity Recognition: a Reliability Benchmark and the CARING Models

• URL: http://arxiv.org/abs/2101.00468v1
• Date: Sat, 2 Jan 2021 15:41:21 GMT
• Title: Uncertainty-sensitive Activity Recognition: a Reliability Benchmark and the CARING Models
• Authors: Alina Roitberg, Monica Haurilet, Manuel Martinez and Rainer Stiefelhagen
• Abstract summary: We present the first study of how welthe confidence values of modern action recognition architectures indeed reflect the probability of the correct outcome. We introduce a new approach which learns to transform the model output into realistic confidence estimates through an additional calibration network.
• Score: 37.60817779613977
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Beyond assigning the correct class, an activity recognition model should also be able to determine, how certain it is in its predictions. We present the first study of how welthe confidence values of modern action recognition architectures indeed reflect the probability of the correct outcome and propose a learning-based approach for improving it. First, we extend two popular action recognition datasets with a reliability benchmark in form of the expected calibration error and reliability diagrams. Since our evaluation highlights that confidence values of standard action recognition architectures do not represent the uncertainty well, we introduce a new approach which learns to transform the model output into realistic confidence estimates through an additional calibration network. The main idea of our Calibrated Action Recognition with Input Guidance (CARING) model is to learn an optimal scaling parameter depending on the video representation. We compare our model with the native action recognition networks and the temperature scaling approach - a wide spread calibration method utilized in image classification. While temperature scaling alone drastically improves the reliability of the confidence values, our CARING method consistently leads to the best uncertainty estimates in all benchmark settings.

Related papers

• Selective Learning: Towards Robust Calibration with Dynamic Regularization [79.92633587914659]
  Miscalibration in deep learning refers to there is a discrepancy between the predicted confidence and performance. We introduce Dynamic Regularization (DReg) which aims to learn what should be learned during training thereby circumventing the confidence adjusting trade-off.
  arXiv  Detail & Related papers  (2024-02-13T11:25:20Z)
• Uncertainty Estimation for Safety-critical Scene Segmentation via Fine-grained Reward Maximization [12.79542334840646]
  Uncertainty estimation plays an important role for future reliable deployment of deep segmentation models in safety-critical scenarios. We propose a novel fine-grained reward (FGRM) framework to address uncertainty estimation. Our method outperforms state-of-the-art methods by a clear margin on all the calibration metrics of uncertainty estimation.
  arXiv  Detail & Related papers  (2023-11-05T17:43:37Z)
• Towards Calibrated Robust Fine-Tuning of Vision-Language Models [97.19901765814431]
  This work proposes a robust fine-tuning method that improves both OOD accuracy and confidence calibration simultaneously in vision language models. We show that both OOD classification and OOD calibration errors have a shared upper bound consisting of two terms of ID data. Based on this insight, we design a novel framework that conducts fine-tuning with a constrained multimodal contrastive loss enforcing a larger smallest singular value.
  arXiv  Detail & Related papers  (2023-11-03T05:41:25Z)
• Two Sides of Miscalibration: Identifying Over and Under-Confidence Prediction for Network Calibration [1.192436948211501]
  Proper confidence calibration of deep neural networks is essential for reliable predictions in safety-critical tasks. Miscalibration can lead to model over-confidence and/or under-confidence. We introduce a novel metric, a miscalibration score, to identify the overall and class-wise calibration status. We use the class-wise miscalibration score as a proxy to design a calibration technique that can tackle both over and under-confidence.
  arXiv  Detail & Related papers  (2023-08-06T17:59:14Z)
• Trust, but Verify: Using Self-Supervised Probing to Improve Trustworthiness [29.320691367586004]
  We introduce a new approach of self-supervised probing, which enables us to check and mitigate the overconfidence issue for a trained model. We provide a simple yet effective framework, which can be flexibly applied to existing trustworthiness-related methods in a plug-and-play manner.
  arXiv  Detail & Related papers  (2023-02-06T08:57:20Z)
• Confidence-Calibrated Face and Kinship Verification [8.570969129199467]
  We introduce an effective confidence measure that allows verification models to convert a similarity score into a confidence score for any given face pair. We also propose a confidence-calibrated approach, termed Angular Scaling (ASC), which is easy to implement and can be readily applied to existing verification models. To the best of our knowledge, our work presents the first comprehensive confidence-calibrated solution for modern face and kinship verification tasks.
  arXiv  Detail & Related papers  (2022-10-25T10:43:46Z)
• Reliability-Aware Prediction via Uncertainty Learning for Person Image Retrieval [51.83967175585896]
  UAL aims at providing reliability-aware predictions by considering data uncertainty and model uncertainty simultaneously. Data uncertainty captures the noise" inherent in the sample, while model uncertainty depicts the model's confidence in the sample's prediction.
  arXiv  Detail & Related papers  (2022-10-24T17:53:20Z)
• Is my Driver Observation Model Overconfident? Input-guided Calibration Networks for Reliable and Interpretable Confidence Estimates [23.449073032842076]
  Driver observation models are rarely deployed under perfect conditions. We show that raw neural network-based approaches tend to significantly overestimate their prediction quality. We introduce Calibrated Action Recognition with Input Guidance (CARING)-a novel approach leveraging an additional neural network to learn scaling the confidences depending on the video representation.
  arXiv  Detail & Related papers  (2022-04-10T12:43:58Z)
• Inducing Predictive Uncertainty Estimation for Face Recognition [102.58180557181643]
  We propose a method for generating image quality training data automatically from'mated-pairs' of face images. We use the generated data to train a lightweight Predictive Confidence Network, termed as PCNet, for estimating the confidence score of a face image.
  arXiv  Detail & Related papers  (2020-09-01T17:52:00Z)
• Binary Classification from Positive Data with Skewed Confidence [85.18941440826309]
  Positive-confidence (Pconf) classification is a promising weakly-supervised learning method. In practice, the confidence may be skewed by bias arising in an annotation process. We introduce the parameterized model of the skewed confidence, and propose the method for selecting the hyper parameter.
  arXiv  Detail & Related papers  (2020-01-29T00:04:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.