PseudoCal: A Source-Free Approach to Unsupervised Uncertainty Calibration in Domain Adaptation

• URL: http://arxiv.org/abs/2307.07489v1
• Date: Fri, 14 Jul 2023 17:21:41 GMT
• Title: PseudoCal: A Source-Free Approach to Unsupervised Uncertainty Calibration in Domain Adaptation
• Authors: Dapeng Hu, Jian Liang, Xinchao Wang, Chuan-Sheng Foo
• Abstract summary: Unsupervised domain adaptation (UDA) has witnessed remarkable advancements in improving the accuracy of models for unlabeled target domains. The calibration of predictive uncertainty in the target domain, a crucial aspect of the safe deployment of UDA models, has received limited attention. We propose PseudoCal, a source-free calibration method that exclusively relies on unlabeled target data.
• Score: 87.69789891809562
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Unsupervised domain adaptation (UDA) has witnessed remarkable advancements in improving the accuracy of models for unlabeled target domains. However, the calibration of predictive uncertainty in the target domain, a crucial aspect of the safe deployment of UDA models, has received limited attention. The conventional in-domain calibration method, \textit{temperature scaling} (TempScal), encounters challenges due to domain distribution shifts and the absence of labeled target domain data. Recent approaches have employed importance-weighting techniques to estimate the target-optimal temperature based on re-weighted labeled source data. Nonetheless, these methods require source data and suffer from unreliable density estimates under severe domain shifts, rendering them unsuitable for source-free UDA settings. To overcome these limitations, we propose PseudoCal, a source-free calibration method that exclusively relies on unlabeled target data. Unlike previous approaches that treat UDA calibration as a \textit{covariate shift} problem, we consider it as an unsupervised calibration problem specific to the target domain. Motivated by the factorization of the negative log-likelihood (NLL) objective in TempScal, we generate a labeled pseudo-target set that captures the structure of the real target. By doing so, we transform the unsupervised calibration problem into a supervised one, enabling us to effectively address it using widely-used in-domain methods like TempScal. Finally, we thoroughly evaluate the calibration performance of PseudoCal by conducting extensive experiments on 10 UDA methods, considering both traditional UDA settings and recent source-free UDA scenarios. The experimental results consistently demonstrate the superior performance of PseudoCal, exhibiting significantly reduced calibration error compared to existing calibration methods.

Related papers

• Calibration of Network Confidence for Unsupervised Domain Adaptation Using Estimated Accuracy [13.22445242068721]
  We introduce a calibration procedure that relies on estimating the network's accuracy on the target domain. The proposed algorithm calibrates the prediction confidence directly in the target domain by minimizing the disparity between the estimated accuracy and the computed confidence.
  arXiv  Detail & Related papers  (2024-09-06T12:46:43Z)
• Source-Free Domain-Invariant Performance Prediction [68.39031800809553]
  We propose a source-free approach centred on uncertainty-based estimation, using a generative model for calibration in the absence of source data. Our experiments on benchmark object recognition datasets reveal that existing source-based methods fall short with limited source sample availability. Our approach significantly outperforms the current state-of-the-art source-free and source-based methods, affirming its effectiveness in domain-invariant performance estimation.
  arXiv  Detail & Related papers  (2024-08-05T03:18:58Z)
• Uncertainty-Guided Alignment for Unsupervised Domain Adaptation in Regression [5.437298646956505]
  Unsupervised Domain Adaptation for Regression (UDAR) aims to adapt models from a labeled source domain to an unlabeled target domain for regression tasks. Traditional feature alignment methods, successful in classification, often prove ineffective for regression due to the correlated nature of regression features. We propose Uncertainty-Guided Alignment (UGA), a novel method that integrates predictive uncertainty into the feature alignment process.
  arXiv  Detail & Related papers  (2024-01-24T14:55:02Z)
• Cal-SFDA: Source-Free Domain-adaptive Semantic Segmentation with Differentiable Expected Calibration Error [50.86671887712424]
  The prevalence of domain adaptive semantic segmentation has prompted concerns regarding source domain data leakage. To circumvent the requirement for source data, source-free domain adaptation has emerged as a viable solution. We propose a novel calibration-guided source-free domain adaptive semantic segmentation framework.
  arXiv  Detail & Related papers  (2023-08-06T03:28:34Z)
• Source-free Unsupervised Domain Adaptation for Blind Image Quality Assessment [20.28784839680503]
  Existing learning-based methods for blind image quality assessment (BIQA) are heavily dependent on large amounts of annotated training data. In this paper, we take the first step towards the source-free unsupervised domain adaptation (SFUDA) in a simple yet efficient manner. We present a group of well-designed self-supervised objectives to guide the adaptation of the BN affine parameters towards the target domain.
  arXiv  Detail & Related papers  (2022-07-17T09:42:36Z)
• Learning Unbiased Transferability for Domain Adaptation by Uncertainty Modeling [107.24387363079629]
  Domain adaptation aims to transfer knowledge from a labeled source domain to an unlabeled or a less labeled but related target domain. Due to the imbalance between the amount of annotated data in the source and target domains, only the target distribution is aligned to the source domain. We propose a non-intrusive Unbiased Transferability Estimation Plug-in (UTEP) by modeling the uncertainty of a discriminator in adversarial-based DA methods to optimize unbiased transfer.
  arXiv  Detail & Related papers  (2022-06-02T21:58:54Z)
• Confidence Calibration for Domain Generalization under Covariate Shift [12.527429721643783]
  We present novel calibration solutions via domain generalization. Our core idea is to leverage multiple calibration domains to reduce the effective distribution disparity between the target and calibration domains. Compared against the state-of-the-art calibration methods designed for domain adaptation, we observe a decrease of 8.86 percentage points in expected calibration error.
  arXiv  Detail & Related papers  (2021-04-01T19:31:54Z)
• Transferable Calibration with Lower Bias and Variance in Domain Adaptation [139.4332115349543]
  Domain Adaptation (DA) enables transferring a learning machine from a labeled source domain to an unlabeled target one. How to estimate the predictive uncertainty of DA models is vital for decision-making in safety-critical scenarios. TransCal can be easily applied to recalibrate existing DA methods.
  arXiv  Detail & Related papers  (2020-07-16T11:09:36Z)
• Unsupervised Calibration under Covariate Shift [92.02278658443166]
  We introduce the problem of calibration under domain shift and propose an importance sampling based approach to address it. We evaluate and discuss the efficacy of our method on both real-world datasets and synthetic datasets.
  arXiv  Detail & Related papers  (2020-06-29T21:50:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.