Right Regions, Wrong Labels: Semantic Label Flips in Segmentation under Correlation Shift

• URL: http://arxiv.org/abs/2604.13326v1
• Date: Tue, 14 Apr 2026 22:15:17 GMT
• Title: Right Regions, Wrong Labels: Semantic Label Flips in Segmentation under Correlation Shift
• Abstract summary: We show that a model may achieve reasonable overlap while assigning the wrong semantic label.<n>We quantify it with a simple diagnostic (Flip) that counts how often ground truth foreground pixels are assigned the wrong foreground identity.
• Score: 36.96633404761302
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The robustness of machine learning models can be compromised by spurious correlations between non-causal features in the input data and target labels. A common way to test for such correlations is to train on data where the label is strongly tied to some non-causal cue, then evaluate on examples where that tie no longer holds. This idea is well established for classification tasks, but for semantic segmentation the specific failure modes are not well understood. We show that a model may achieve reasonable overlap while assigning the wrong semantic label, swapping one plausible foreground class for another, even when object boundaries are largely correct. We focus on this semantic label-flip behaviour and quantify it with a simple diagnostic (Flip) that counts how often ground truth foreground pixels are assigned the wrong foreground identity while remaining predicted as foreground. In a setting where category and scene are correlated during training, increasing the correlation consistently widens the gap between common and rare test conditions and increases these within-object label swaps on counterfactual groups. Overall, our results motivate assessing segmentation robustness under distribution shift beyond overlap by decomposing foreground errors into correct pixels, flipped-identity pixels, and missed-to-background pixels. We also propose an entropy-based, ground truth label-free `flip-risk' score, which is computed from foreground identity uncertainty, and show that it can flag flip-prone cases at inference time. Code is available at https://github.com/acharaakshit/label-flips.

Related papers

• Align and Segment: Unsupervised Learning for Building Segmentation From Misaligned Labels [15.14055526507309]
  Supervised learning for image segmentation typically requires spatially aligned image and label sets.<n> misaligned labels can significantly deteriorate the performance of the learned models.<n>We propose a novel approach for training on misaligned labels, where we simultaneously learn the label alignment.
  arXiv  Detail & Related papers  (2026-07-12T17:07:55Z)
• Segmentation with Noisy Labels via Spatially Correlated Distributions [3.522950356329991]
  We propose an approximate Bayesian estimation based on a probabilistic model that assumes training data includes label errors.<n>We show that the proposed method achieves performance comparable to training with clean labels under moderate noise levels.
  arXiv  Detail & Related papers  (2025-04-21T01:50:10Z)
• Using Unreliable Pseudo-Labels for Label-Efficient Semantic Segmentation [78.56076985502291]
  We argue that every pixel matters to the model training, even those unreliable and ambiguous pixels. We separate reliable and unreliable pixels via the entropy of predictions, push each unreliable pixel to a category-wise queue that consists of negative keys. Considering the training evolution, we adaptively adjust the threshold for the reliable-unreliable partition.
  arXiv  Detail & Related papers  (2023-06-04T09:40:25Z)
• All Points Matter: Entropy-Regularized Distribution Alignment for Weakly-supervised 3D Segmentation [67.30502812804271]
  Pseudo-labels are widely employed in weakly supervised 3D segmentation tasks where only sparse ground-truth labels are available for learning. We propose a novel learning strategy to regularize the generated pseudo-labels and effectively narrow the gaps between pseudo-labels and model predictions.
  arXiv  Detail & Related papers  (2023-05-25T08:19:31Z)
• Label Inference Attack against Split Learning under Regression Setting [24.287752556622312]
  We study the leakage in the scenario of the regression model, where the private labels are continuous numbers. We propose a novel learning-based attack that integrates gradient information and extra learning regularization objectives.
  arXiv  Detail & Related papers  (2023-01-18T03:17:24Z)
• Dist-PU: Positive-Unlabeled Learning from a Label Distribution Perspective [89.5370481649529]
  We propose a label distribution perspective for PU learning in this paper. Motivated by this, we propose to pursue the label distribution consistency between predicted and ground-truth label distributions. Experiments on three benchmark datasets validate the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2022-12-06T07:38:29Z)
• Automated Detection of Label Errors in Semantic Segmentation Datasets via Deep Learning and Uncertainty Quantification [5.279257531335345]
  We for the first time present a method for detecting label errors in semantic segmentation datasets with pixel-wise labels. Our approach is able to detect the vast majority of label errors while controlling the number of false label error detections.
  arXiv  Detail & Related papers  (2022-07-13T10:25:23Z)
• A Closer Look at Self-training for Zero-Label Semantic Segmentation [53.4488444382874]
  Being able to segment unseen classes not observed during training is an important technical challenge in deep learning. Prior zero-label semantic segmentation works approach this task by learning visual-semantic embeddings or generative models. We propose a consistency regularizer to filter out noisy pseudo-labels by taking the intersections of the pseudo-labels generated from different augmentations of the same image.
  arXiv  Detail & Related papers  (2021-04-21T14:34:33Z)
• Measuring Model Biases in the Absence of Ground Truth [2.802021236064919]
  We introduce a new framing to the measurement of fairness and bias that does not rely on ground truth labels. Instead, we treat the model predictions for a given image as a set of labels, analogous to a 'bag of words' approach used in Natural Language Processing (NLP) We demonstrate how the statistical properties (especially normalization) of the different association metrics can lead to different sets of labels detected as having "gender bias"
  arXiv  Detail & Related papers  (2021-03-05T01:23:22Z)
• Debiased Contrastive Learning [64.98602526764599]
  We develop a debiased contrastive objective that corrects for the sampling of same-label datapoints. Empirically, the proposed objective consistently outperforms the state-of-the-art for representation learning in vision, language, and reinforcement learning benchmarks.
  arXiv  Detail & Related papers  (2020-07-01T04:25:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.