Generalize or Detect? Towards Robust Semantic Segmentation Under Multiple Distribution Shifts

• URL: http://arxiv.org/abs/2411.03829v1
• Date: Wed, 06 Nov 2024 11:03:02 GMT
• Title: Generalize or Detect? Towards Robust Semantic Segmentation Under Multiple Distribution Shifts
• Authors: Zhitong Gao, Bingnan Li, Mathieu Salzmann, Xuming He,
• Abstract summary: In open-world scenarios, where both novel classes and domains may exist, an ideal segmentation model should detect anomaly classes for safety. Existing methods often struggle to distinguish between domain-level and semantic-level distribution shifts.
• Score: 56.57141696245328
• License:
• Abstract: In open-world scenarios, where both novel classes and domains may exist, an ideal segmentation model should detect anomaly classes for safety and generalize to new domains. However, existing methods often struggle to distinguish between domain-level and semantic-level distribution shifts, leading to poor out-of-distribution (OOD) detection or domain generalization performance. In this work, we aim to equip the model to generalize effectively to covariate-shift regions while precisely identifying semantic-shift regions. To achieve this, we design a novel generative augmentation method to produce coherent images that incorporate both anomaly (or novel) objects and various covariate shifts at both image and object levels. Furthermore, we introduce a training strategy that recalibrates uncertainty specifically for semantic shifts and enhances the feature extractor to align features associated with domain shifts. We validate the effectiveness of our method across benchmarks featuring both semantic and domain shifts. Our method achieves state-of-the-art performance across all benchmarks for both OOD detection and domain generalization. Code is available at https://github.com/gaozhitong/MultiShiftSeg.

Related papers

• Feature-Space Semantic Invariance: Enhanced OOD Detection for Open-Set Domain Generalization [10.38552112657656]
  We propose a unified framework for open-set domain generalization by introducing Feature-space Semantic Invariance (FSI) FSI maintains semantic consistency across different domains within the feature space, enabling more accurate detection of OOD instances in unseen domains. We also adopt a generative model to produce synthetic data with novel domain styles or class labels, enhancing model robustness.
  arXiv  Detail & Related papers  (2024-11-11T21:51:45Z)
• Activate and Reject: Towards Safe Domain Generalization under Category Shift [71.95548187205736]
  We study a practical problem of Domain Generalization under Category Shift (DGCS) It aims to simultaneously detect unknown-class samples and classify known-class samples in the target domains. Compared to prior DG works, we face two new challenges: 1) how to learn the concept of unknown'' during training with only source known-class samples, and 2) how to adapt the source-trained model to unseen environments.
  arXiv  Detail & Related papers  (2023-10-07T07:53:12Z)
• ATTA: Anomaly-aware Test-Time Adaptation for Out-of-Distribution Detection in Segmentation [22.084967085509387]
  We propose a dual-level OOD detection framework to handle domain shift and semantic shift jointly. The first level distinguishes whether domain shift exists in the image by leveraging global low-level features. The second level identifies pixels with semantic shift by utilizing dense high-level feature maps.
  arXiv  Detail & Related papers  (2023-09-12T06:49:56Z)
• CLIP the Gap: A Single Domain Generalization Approach for Object Detection [60.20931827772482]
  Single Domain Generalization tackles the problem of training a model on a single source domain so that it generalizes to any unseen target domain. We propose to leverage a pre-trained vision-language model to introduce semantic domain concepts via textual prompts. We achieve this via a semantic augmentation strategy acting on the features extracted by the detector backbone, as well as a text-based classification loss.
  arXiv  Detail & Related papers  (2023-01-13T12:01:18Z)
• Adaptive Domain Generalization via Online Disagreement Minimization [17.215683606365445]
  Domain Generalization aims to safely transfer a model to unseen target domains. AdaODM adaptively modifies the source model at test time for different target domains. Results show AdaODM stably improves the generalization capacity on unseen domains.
  arXiv  Detail & Related papers  (2022-08-03T11:51:11Z)
• Domain Generalisation for Object Detection under Covariate and Concept Shift [10.32461766065764]
  Domain generalisation aims to promote the learning of domain-invariant features while suppressing domain-specific features. An approach to domain generalisation for object detection is proposed, the first such approach applicable to any object detection architecture.
  arXiv  Detail & Related papers  (2022-03-10T11:14:18Z)
• Attention-based Cross-Layer Domain Alignment for Unsupervised Domain Adaptation [14.65316832227658]
  Unsupervised domain adaptation (UDA) aims to learn transferable knowledge from a labeled source domain and adapts a trained model to an unlabeled target domain. One prevailing strategy is to minimize the distribution discrepancy by aligning their semantic features extracted by deep models.
  arXiv  Detail & Related papers  (2022-02-27T08:36:12Z)
• AFAN: Augmented Feature Alignment Network for Cross-Domain Object Detection [90.18752912204778]
  Unsupervised domain adaptation for object detection is a challenging problem with many real-world applications. We propose a novel augmented feature alignment network (AFAN) which integrates intermediate domain image generation and domain-adversarial training. Our approach significantly outperforms the state-of-the-art methods on standard benchmarks for both similar and dissimilar domain adaptations.
  arXiv  Detail & Related papers  (2021-06-10T05:01:20Z)
• Cross-domain Detection via Graph-induced Prototype Alignment [114.8952035552862]
  We propose a Graph-induced Prototype Alignment (GPA) framework to seek for category-level domain alignment. In addition, in order to alleviate the negative effect of class-imbalance on domain adaptation, we design a Class-reweighted Contrastive Loss. Our approach outperforms existing methods with a remarkable margin.
  arXiv  Detail & Related papers  (2020-03-28T17:46:55Z)
• Cross-domain Object Detection through Coarse-to-Fine Feature Adaptation [62.29076080124199]
  This paper proposes a novel coarse-to-fine feature adaptation approach to cross-domain object detection. At the coarse-grained stage, foreground regions are extracted by adopting the attention mechanism, and aligned according to their marginal distributions. At the fine-grained stage, we conduct conditional distribution alignment of foregrounds by minimizing the distance of global prototypes with the same category but from different domains.
  arXiv  Detail & Related papers  (2020-03-23T13:40:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.