A Bayesian Evaluation Framework for Subjectively Annotated Visual Recognition Tasks

• URL: http://arxiv.org/abs/2007.06711v2
• Date: Wed, 1 Sep 2021 22:05:56 GMT
• Title: A Bayesian Evaluation Framework for Subjectively Annotated Visual Recognition Tasks
• Authors: Derek S. Prijatelj (1), Mel McCurrie (2), Walter J. Scheirer (1) ((1) University of Notre Dame, Notre Dame, USA, (2) Perceptive Automata, Boston, USA)
• Abstract summary: We propose a framework for evaluating the uncertainty that comes from the predictor's internal structure. The framework is successfully applied to four image classification tasks that use subjective human judgements.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: An interesting development in automatic visual recognition has been the emergence of tasks where it is not possible to assign objective labels to images, yet still feasible to collect annotations that reflect human judgements about them. Machine learning-based predictors for these tasks rely on supervised training that models the behavior of the annotators, i.e., what would the average person's judgement be for an image? A key open question for this type of work, especially for applications where inconsistency with human behavior can lead to ethical lapses, is how to evaluate the epistemic uncertainty of trained predictors, i.e., the uncertainty that comes from the predictor's model. We propose a Bayesian framework for evaluating black box predictors in this regime, agnostic to the predictor's internal structure. The framework specifies how to estimate the epistemic uncertainty that comes from the predictor with respect to human labels by approximating a conditional distribution and producing a credible interval for the predictions and their measures of performance. The framework is successfully applied to four image classification tasks that use subjective human judgements: facial beauty assessment, social attribute assignment, apparent age estimation, and ambiguous scene labeling.

Related papers

• Interpreting Predictive Probabilities: Model Confidence or Human Label Variation? [27.226997687210044]
  We identify two main perspectives that drive starkly different evaluation protocols. We discuss their merits and limitations, and take the position that both are crucial for trustworthy and fair NLP systems. We recommend tools and highlight exciting directions towards models with disentangled representations of uncertainty about predictions and uncertainty about human labels.
  arXiv  Detail & Related papers  (2024-02-25T15:00:13Z)
• Modeling the Uncertainty with Maximum Discrepant Students for Semi-supervised 2D Pose Estimation [57.17120203327993]
  We propose a framework to estimate the quality of pseudo-labels in semi-supervised pose estimation tasks. Our method improves the performance of semi-supervised pose estimation on three datasets.
  arXiv  Detail & Related papers  (2023-11-03T08:11:06Z)
• Causal Unsupervised Semantic Segmentation [60.178274138753174]
  Unsupervised semantic segmentation aims to achieve high-quality semantic grouping without human-labeled annotations. We propose a novel framework, CAusal Unsupervised Semantic sEgmentation (CAUSE), which leverages insights from causal inference.
  arXiv  Detail & Related papers  (2023-10-11T10:54:44Z)
• Gender Biases in Automatic Evaluation Metrics for Image Captioning [87.15170977240643]
  We conduct a systematic study of gender biases in model-based evaluation metrics for image captioning tasks. We demonstrate the negative consequences of using these biased metrics, including the inability to differentiate between biased and unbiased generations. We present a simple and effective way to mitigate the metric bias without hurting the correlations with human judgments.
  arXiv  Detail & Related papers  (2023-05-24T04:27:40Z)
• What Should I Know? Using Meta-gradient Descent for Predictive Feature Discovery in a Single Stream of Experience [63.75363908696257]
  computational reinforcement learning seeks to construct an agent's perception of the world through predictions of future sensations. An open challenge in this line of work is determining from the infinitely many predictions that the agent could possibly make which predictions might best support decision-making. We introduce a meta-gradient descent process by which an agent learns what predictions to make, 2) the estimates for its chosen predictions, and 3) how to use those estimates to generate policies that maximize future reward.
  arXiv  Detail & Related papers  (2022-06-13T21:31:06Z)
• See Yourself in Others: Attending Multiple Tasks for Own Failure Detection [28.787334666116518]
  We propose an attention-based failure detection approach by exploiting the correlations among multiple tasks. The proposed framework infers task failures by evaluating the individual prediction, across multiple visual perception tasks for different regions in an image. Our proposed framework generates more accurate estimations of the prediction error for the different task's predictions.
  arXiv  Detail & Related papers  (2021-10-06T07:42:57Z)
• Learning Uncertainty For Safety-Oriented Semantic Segmentation In Autonomous Driving [77.39239190539871]
  We show how uncertainty estimation can be leveraged to enable safety critical image segmentation in autonomous driving. We introduce a new uncertainty measure based on disagreeing predictions as measured by a dissimilarity function. We show experimentally that our proposed approach is much less computationally intensive at inference time than competing methods.
  arXiv  Detail & Related papers  (2021-05-28T09:23:05Z)
• Self-Supervision by Prediction for Object Discovery in Videos [62.87145010885044]
  In this paper, we use the prediction task as self-supervision and build a novel object-centric model for image sequence representation. Our framework can be trained without the help of any manual annotation or pretrained network. Initial experiments confirm that the proposed pipeline is a promising step towards object-centric video prediction.
  arXiv  Detail & Related papers  (2021-03-09T19:14:33Z)
• Probabilistic Deep Learning for Instance Segmentation [9.62543698736491]
  We propose a generic method to obtain model-inherent uncertainty estimates within proposal-free instance segmentation models. We evaluate our method on the BBBC010 C. elegans dataset, where it yields competitive performance.
  arXiv  Detail & Related papers  (2020-08-24T19:51:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.