ODD: Overlap-aware Estimation of Model Performance under Distribution Shift

• URL: http://arxiv.org/abs/2506.14978v1
• Date: Tue, 17 Jun 2025 21:05:42 GMT
• Title: ODD: Overlap-aware Estimation of Model Performance under Distribution Shift
• Authors: Aayush Mishra, Anqi Liu,
• Abstract summary: Prior work uses disagreement discrepancy (DIS2) to derive practical error bounds under distribution shifts.<n>We devise Overlap-aware Disagreement Discrepancy (ODD)<n>Our ODD-based bound uses domain-classifiers to estimate domain-overlap and better predicts target performance than DIS2.
• Score: 8.569585481097839
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reliable and accurate estimation of the error of an ML model in unseen test domains is an important problem for safe intelligent systems. Prior work uses disagreement discrepancy (DIS^2) to derive practical error bounds under distribution shifts. It optimizes for a maximally disagreeing classifier on the target domain to bound the error of a given source classifier. Although this approach offers a reliable and competitively accurate estimate of the target error, we identify a problem in this approach which causes the disagreement discrepancy objective to compete in the overlapping region between source and target domains. With an intuitive assumption that the target disagreement should be no more than the source disagreement in the overlapping region due to high enough support, we devise Overlap-aware Disagreement Discrepancy (ODD). Maximizing ODD only requires disagreement in the non-overlapping target domain, removing the competition. Our ODD-based bound uses domain-classifiers to estimate domain-overlap and better predicts target performance than DIS^2. We conduct experiments on a wide array of benchmarks to show that our method improves the overall performance-estimation error while remaining valid and reliable. Our code and results are available on GitHub.

Related papers

• Make the U in UDA Matter: Invariant Consistency Learning for Unsupervised Domain Adaptation [86.61336696914447]
  We dub our approach "Invariant CONsistency learning" (ICON) We propose to make the U in Unsupervised DA matter by giving equal status to the two domains. ICON achieves the state-of-the-art performance on the classic UDA benchmarks: Office-Home and VisDA-2017, and outperforms all the conventional methods on the challenging WILDS 2.0 benchmark.
  arXiv  Detail & Related papers  (2023-09-22T09:43:32Z)
• Divide and Contrast: Source-free Domain Adaptation via Adaptive Contrastive Learning [122.62311703151215]
  Divide and Contrast (DaC) aims to connect the good ends of both worlds while bypassing their limitations. DaC divides the target data into source-like and target-specific samples, where either group of samples is treated with tailored goals. We further align the source-like domain with the target-specific samples using a memory bank-based Maximum Mean Discrepancy (MMD) loss to reduce the distribution mismatch.
  arXiv  Detail & Related papers  (2022-11-12T09:21:49Z)
• Distributionally Robust Domain Adaptation [12.02023514105999]
  Domain Adaptation (DA) has recently received significant attention due to its potential to adapt a learning model across source and target domains with mismatched distributions. In this paper, we propose DRDA, a distributionally robust domain adaptation method.
  arXiv  Detail & Related papers  (2022-10-30T17:29:22Z)
• Domain-Specific Risk Minimization for Out-of-Distribution Generalization [104.17683265084757]
  We first establish a generalization bound that explicitly considers the adaptivity gap. We propose effective gap estimation methods for guiding the selection of a better hypothesis for the target. The other method is minimizing the gap directly by adapting model parameters using online target samples.
  arXiv  Detail & Related papers  (2022-08-18T06:42:49Z)
• Learning Unbiased Transferability for Domain Adaptation by Uncertainty Modeling [107.24387363079629]
  Domain adaptation aims to transfer knowledge from a labeled source domain to an unlabeled or a less labeled but related target domain. Due to the imbalance between the amount of annotated data in the source and target domains, only the target distribution is aligned to the source domain. We propose a non-intrusive Unbiased Transferability Estimation Plug-in (UTEP) by modeling the uncertainty of a discriminator in adversarial-based DA methods to optimize unbiased transfer.
  arXiv  Detail & Related papers  (2022-06-02T21:58:54Z)
• Source-Free Domain Adaptation via Distribution Estimation [106.48277721860036]
  Domain Adaptation aims to transfer the knowledge learned from a labeled source domain to an unlabeled target domain whose data distributions are different. Recently, Source-Free Domain Adaptation (SFDA) has drawn much attention, which tries to tackle domain adaptation problem without using source data. In this work, we propose a novel framework called SFDA-DE to address SFDA task via source Distribution Estimation.
  arXiv  Detail & Related papers  (2022-04-24T12:22:19Z)
• Boosting Unsupervised Domain Adaptation with Soft Pseudo-label and Curriculum Learning [19.903568227077763]
  Unsupervised domain adaptation (UDA) improves classification performance on an unlabeled target domain by leveraging data from a fully labeled source domain. We propose a model-agnostic two-stage learning framework, which greatly reduces flawed model predictions using soft pseudo-label strategy. At the second stage, we propose a curriculum learning strategy to adaptively control the weighting between losses from the two domains.
  arXiv  Detail & Related papers  (2021-12-03T14:47:32Z)
• Unsupervised domain adaptation with non-stochastic missing data [0.6608945629704323]
  We consider unsupervised domain adaptation (UDA) for classification problems in the presence of missing data in the unlabelled target domain. Imputation is performed in a domain-invariant latent space and leverages indirect supervision from a complete source domain. We show the benefits of jointly performing adaptation, classification and imputation on datasets.
  arXiv  Detail & Related papers  (2021-09-16T06:37:07Z)
• Instance Level Affinity-Based Transfer for Unsupervised Domain Adaptation [74.71931918541748]
  We propose an instance affinity based criterion for source to target transfer during adaptation, called ILA-DA. We first propose a reliable and efficient method to extract similar and dissimilar samples across source and target, and utilize a multi-sample contrastive loss to drive the domain alignment process. We verify the effectiveness of ILA-DA by observing consistent improvements in accuracy over popular domain adaptation approaches on a variety of benchmark datasets.
  arXiv  Detail & Related papers  (2021-04-03T01:33:14Z)
• Robustified Domain Adaptation [13.14535125302501]
  Unsupervised domain adaptation (UDA) is widely used to transfer knowledge from a labeled source domain to an unlabeled target domain. The inevitable domain distribution deviation in UDA is a critical barrier to model robustness on the target domain. We propose a novel Class-consistent Unsupervised Domain Adaptation (CURDA) framework for training robust UDA models.
  arXiv  Detail & Related papers  (2020-11-18T22:21:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.