Towards Accurate Model Selection in Deep Unsupervised Domain Adaptation
Abstract Overview
This paper addresses model selection for deep unsupervised domain adaptation (UDA), where target labels are unavailable and commonly used validation strategies are biased, unstable, or rely on labeled target data. The authors propose Deep Embedded Validation (DEV), which estimates target risk by performing importance weighting in the learned feature space rather than the original input space. They argue theoretically that this preserves unbiasedness while giving a tighter variance bound because adapted features reduce domain divergence, and they further reduce variance with a control variate. The method is positioned as a standardized, non-intrusive validation procedure for comparing deep UDA models under unlabeled target domains.
Novelty
The distinctive contribution is to embed adapted deep representations directly into the validation procedure for domain adaptation, instead of estimating importance weights only in the original input space. DEV combines feature-space density-ratio estimation with a control variate, yielding an unbiased target-risk estimator with bounded and empirically lower variance.
Results
Across multiple benchmarks, DEV selected models whose performance was consistently close to the Target Risk upper bound while avoiding target labels. On VisDA with MCD, DEV reached 72.42% mean accuracy versus 72.55% for Target Risk; on Office-31 with CDAN, 87.75% versus 88.16%; on Digits with GTA, 94.22% versus 94.50%; and on partial domain adaptation with PADA, 93.15% versus 93.34%. The toy experiment and VisDA ablation also indicate lower variance than IWCV and further gains from the control variate.
Key Points
- DEV estimates target risk in the adapted feature space, where domain divergence is smaller than in the raw input space, improving the stability of importance-weighted validation.
- The method uses a discriminative density-ratio estimator and a control variate based on the importance weights to preserve unbiasedness while reducing variance.
- Empirically, DEV consistently outperforms Source Risk and IWCV for model selection and selects models that are close to the Target Risk upper bound across standard and partial domain adaptation settings.