Transitional Uncertainty with Layered Intermediate Predictions

• URL: http://arxiv.org/abs/2405.17494v2
• Date: Sat, 1 Jun 2024 18:55:33 GMT
• Title: Transitional Uncertainty with Layered Intermediate Predictions
• Authors: Ryan Benkert, Mohit Prabhushankar, Ghassan AlRegib,
• Abstract summary: We discuss feature engineering for single-pass uncertainty estimation. We propose Transitional Uncertainty with Layered Intermediate Predictions (T) as a simple approach to address the shortcomings of current single-pass estimators.
• Score: 14.11559987180237
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we discuss feature engineering for single-pass uncertainty estimation. For accurate uncertainty estimates, neural networks must extract differences in the feature space that quantify uncertainty. This could be achieved by current single-pass approaches that maintain feature distances between data points as they traverse the network. While initial results are promising, maintaining feature distances within the network representations frequently inhibits information compression and opposes the learning objective. We study this effect theoretically and empirically to arrive at a simple conclusion: preserving feature distances in the output is beneficial when the preserved features contribute to learning the label distribution and act in opposition otherwise. We then propose Transitional Uncertainty with Layered Intermediate Predictions (TULIP) as a simple approach to address the shortcomings of current single-pass estimators. Specifically, we implement feature preservation by extracting features from intermediate representations before information is collapsed by subsequent layers. We refer to the underlying preservation mechanism as transitional feature preservation. We show that TULIP matches or outperforms current single-pass methods on standard benchmarks and in practical settings where these methods are less reliable (imbalances, complex architectures, medical modalities).

Related papers

• Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximation [47.08004958269403]
  We prove a weaker convergence condition for bootstrapped value estimation in certain cases, even with off-policy data. We extend our results to learning with truncated trajectories, showing that convergence is achievable for all tasks with minor modifications.
  arXiv  Detail & Related papers  (2024-05-31T17:36:16Z)
• QGait: Toward Accurate Quantization for Gait Recognition with Binarized Input [17.017127559393398]
  We propose a differentiable soft quantizer, which better simulates the gradient of the round function during backpropagation. This enables the network to learn from subtle input perturbations. We further refine the training strategy to ensure convergence while simulating quantization errors.
  arXiv  Detail & Related papers  (2024-05-22T17:34:18Z)
• Variation Due to Regularization Tractably Recovers Bayesian Deep Learning [44.16006844888796]
  We propose an uncertainty quantification method for large networks based on variation due to regularization. We show that regularization variation (RegVar) provides rigorous uncertainty estimates that, in the infinitesimal limit, exactly recover the Laplace approximation in Bayesian deep learning. Our experiments across multiple datasets show that RegVar not only identifies uncertain predictions effectively but also provides insights into the stability of learned representations.
  arXiv  Detail & Related papers  (2024-03-15T20:47:39Z)
• Leveraging Unlabeled Data for 3D Medical Image Segmentation through Self-Supervised Contrastive Learning [3.7395287262521717]
  Current 3D semi-supervised segmentation methods face significant challenges such as limited consideration of contextual information. We introduce two distinctworks designed to explore and exploit the discrepancies between them, ultimately correcting the erroneous prediction results. We employ a self-supervised contrastive learning paradigm to distinguish between reliable and unreliable predictions.
  arXiv  Detail & Related papers  (2023-11-21T14:03:16Z)
• ZigZag: Universal Sampling-free Uncertainty Estimation Through Two-Step Inference [54.17205151960878]
  We introduce a sampling-free approach that is generic and easy to deploy. We produce reliable uncertainty estimates on par with state-of-the-art methods at a significantly lower computational cost.
  arXiv  Detail & Related papers  (2022-11-21T13:23:09Z)
• Weakly Supervised Representation Learning with Sparse Perturbations [82.39171485023276]
  We show that if one has weak supervision from observations generated by sparse perturbations of the latent variables, identification is achievable under unknown continuous latent distributions. We propose a natural estimation procedure based on this theory and illustrate it on low-dimensional synthetic and image-based experiments.
  arXiv  Detail & Related papers  (2022-06-02T15:30:07Z)
• Domain-Adjusted Regression or: ERM May Already Learn Features Sufficient for Out-of-Distribution Generalization [52.7137956951533]
  We argue that devising simpler methods for learning predictors on existing features is a promising direction for future research. We introduce Domain-Adjusted Regression (DARE), a convex objective for learning a linear predictor that is provably robust under a new model of distribution shift. Under a natural model, we prove that the DARE solution is the minimax-optimal predictor for a constrained set of test distributions.
  arXiv  Detail & Related papers  (2022-02-14T16:42:16Z)
• Meta Learning Low Rank Covariance Factors for Energy-Based Deterministic Uncertainty [58.144520501201995]
  Bi-Lipschitz regularization of neural network layers preserve relative distances between data instances in the feature spaces of each layer. With the use of an attentive set encoder, we propose to meta learn either diagonal or diagonal plus low-rank factors to efficiently construct task specific covariance matrices. We also propose an inference procedure which utilizes scaled energy to achieve a final predictive distribution.
  arXiv  Detail & Related papers  (2021-10-12T22:04:19Z)
• Learning Uncertainty For Safety-Oriented Semantic Segmentation In Autonomous Driving [77.39239190539871]
  We show how uncertainty estimation can be leveraged to enable safety critical image segmentation in autonomous driving. We introduce a new uncertainty measure based on disagreeing predictions as measured by a dissimilarity function. We show experimentally that our proposed approach is much less computationally intensive at inference time than competing methods.
  arXiv  Detail & Related papers  (2021-05-28T09:23:05Z)
• Uncertainty Surrogates for Deep Learning [17.868995105624023]
  We introduce a novel way of estimating prediction uncertainty in deep networks through the use of uncertainty surrogates. These surrogates are features of the penultimate layer of a deep network that are forced to match predefined patterns. We show how our approach can be used for estimating uncertainty in prediction and out-of-distribution detection.
  arXiv  Detail & Related papers  (2021-04-16T14:50:28Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.