Uncertainty Estimation in Instance Segmentation of Affordances via Bayesian Visual Transformers

• URL: http://arxiv.org/abs/2605.03614v1
• Date: Tue, 05 May 2026 10:40:16 GMT
• Title: Uncertainty Estimation in Instance Segmentation of Affordances via Bayesian Visual Transformers
• Abstract summary: We present a model for instance segmentation of affordances by adopting sample-based and ensembles approaches for uncertainty estimation.<n>We propose a novel measure called Probability-based Mask Quality, which enables a comprehensive analysis of semantic and spatial variations in a probabilistic instance segmentation model.
• Score: 4.348006516438626
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Visual affordances identify regions in an image with potential interactions, offering a novel paradigm for scene understanding. Recognizing affordances allows autonomous robots to act more naturally, could enhance human-robot interactions, enrich augmented reality systems, and benefit prosthetic vision devices. Accurate and localized prediction of affordance regions, rather than general saliency maps is crucial for these applications. We present a model for instance segmentation of affordances by adopting sample-based and ensembles approaches for uncertainty estimation. We extend an attention-based architecture for our novel task, showing with detailed ablation experiments the effects of each component. By comparing the distribution of these different detections, we extract pixel-wise epistemic and aleatoric variances at both the semantic and spatial levels. In addition, we propose a novel measure called Probability-based Mask Quality, which enables a comprehensive analysis of semantic and spatial variations in a probabilistic instance segmentation model. Our results show that the global consensus of multiple sub-networks of Bayesian models improve deterministic networks due to a better mask refinement and generalization. This fact, joined with the more powerful features extracted by attention-based mechanisms, represent an improvement of +7.4 p.p on the $F_β^w$ score in the challenging IIT-Aff dataset. Bayesian models are also better calibrated, producing less overconfident probabilities and with a better uncertainty estimation. Qualitative results show that aleatoric variance appears in the contour of the objects, while the epistemic variance is observed in visual challenging pixels, adding interpretability to the neural network.

Related papers

• Counterfactual Explanations in Medical Imaging: Exploring SPN-Guided Latent Space Manipulation [2.9810923705287524]
  In medical image analysis, deep learning models have demonstrated remarkable performance.<n>Deep generative models such as variational autoencoders (VAEs) exhibit significant generative power.<n>Probability models like sum-product networks (SPNs) efficiently represent complex joint probability distributions.
  arXiv  Detail & Related papers  (2025-07-25T15:19:32Z)
• Reducing Semantic Ambiguity In Domain Adaptive Semantic Segmentation Via Probabilistic Prototypical Pixel Contrast [7.092718945468069]
  Domain adaptation aims to reduce the model degradation on the target domain caused by the domain shift between the source and target domains. Probabilistic proto-typical pixel contrast (PPPC) is a universal adaptation framework that models each pixel embedding as a probability. PPPC not only helps to address ambiguity at the pixel level, yielding discriminative representations but also significant improvements in both synthetic-to-real and day-to-night adaptation tasks.
  arXiv  Detail & Related papers  (2024-09-27T08:25:03Z)
• Gaussian Mixture Models for Affordance Learning using Bayesian Networks [50.18477618198277]
  Affordances are fundamental descriptors of relationships between actions, objects and effects. This paper approaches the problem of an embodied agent exploring the world and learning these affordances autonomously from its sensory experiences.
  arXiv  Detail & Related papers  (2024-02-08T22:05:45Z)
• Probabilistic MIMO U-Net: Efficient and Accurate Uncertainty Estimation for Pixel-wise Regression [1.4528189330418977]
  Uncertainty estimation in machine learning is paramount for enhancing the reliability and interpretability of predictive models. We present an adaptation of the Multiple-Input Multiple-Output (MIMO) framework for pixel-wise regression tasks.
  arXiv  Detail & Related papers  (2023-08-14T22:08:28Z)
• Adaptive Face Recognition Using Adversarial Information Network [57.29464116557734]
  Face recognition models often degenerate when training data are different from testing data. We propose a novel adversarial information network (AIN) to address it.
  arXiv  Detail & Related papers  (2023-05-23T02:14:11Z)
• Bayesian Deep Learning for Affordance Segmentation in images [3.15834651147911]
  We present a novel Bayesian deep network to detect affordances in images. We quantify the distribution of the aleatoric and epistemic variance at the spatial level. Our results outperform the state-of-the-art of deterministic networks.
  arXiv  Detail & Related papers  (2023-03-02T00:01:13Z)
• Bayesian Attention Belief Networks [59.183311769616466]
  Attention-based neural networks have achieved state-of-the-art results on a wide range of tasks. This paper introduces Bayesian attention belief networks, which construct a decoder network by modeling unnormalized attention weights. We show that our method outperforms deterministic attention and state-of-the-art attention in accuracy, uncertainty estimation, generalization across domains, and adversarial attacks.
  arXiv  Detail & Related papers  (2021-06-09T17:46:22Z)
• Adversarial Feature Augmentation and Normalization for Visual Recognition [109.6834687220478]
  Recent advances in computer vision take advantage of adversarial data augmentation to ameliorate the generalization ability of classification models. Here, we present an effective and efficient alternative that advocates adversarial augmentation on intermediate feature embeddings. We validate the proposed approach across diverse visual recognition tasks with representative backbone networks.
  arXiv  Detail & Related papers  (2021-03-22T20:36:34Z)
• A generalizable saliency map-based interpretation of model outcome [1.14219428942199]
  We propose a non-intrusive interpretability technique that uses the input and output of the model to generate a saliency map. Experiments show that our interpretability method can reconstruct the salient part of the input with a classification accuracy of 89%.
  arXiv  Detail & Related papers  (2020-06-16T20:34:42Z)
• Diversity inducing Information Bottleneck in Model Ensembles [73.80615604822435]
  In this paper, we target the problem of generating effective ensembles of neural networks by encouraging diversity in prediction. We explicitly optimize a diversity inducing adversarial loss for learning latent variables and thereby obtain diversity in the output predictions necessary for modeling multi-modal data. Compared to the most competitive baselines, we show significant improvements in classification accuracy, under a shift in the data distribution.
  arXiv  Detail & Related papers  (2020-03-10T03:10:41Z)
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization [56.69671152009899]
  We show that deep ensembles provide an effective mechanism for approximate Bayesian marginalization. We also propose a related approach that further improves the predictive distribution by marginalizing within basins of attraction.
  arXiv  Detail & Related papers  (2020-02-20T15:13:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.