Pick up the PACE: Fast and Simple Domain Adaptation via Ensemble Pseudo-Labeling

• URL: http://arxiv.org/abs/2205.13508v1
• Date: Thu, 26 May 2022 17:20:08 GMT
• Title: Pick up the PACE: Fast and Simple Domain Adaptation via Ensemble Pseudo-Labeling
• Abstract summary: Domain Adaptation (DA) has received widespread attention from deep learning researchers in recent years because of its potential to improve test accuracy with out-of-distribution labeled data. In this work, we propose a fast and simple DA method consisting of three stages: (1) domain alignment by covariance matching, (2) pseudo-labeling, and (3) ensembling. PACE exceeds previous state-of-the-art by $textbf5 - 10 %$ on most benchmark adaptation tasks without training a neural network.
• Score: 16.14184145802016
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Domain Adaptation (DA) has received widespread attention from deep learning researchers in recent years because of its potential to improve test accuracy with out-of-distribution labeled data. Most state-of-the-art DA algorithms require an extensive amount of hyperparameter tuning and are computationally intensive due to the large batch sizes required. In this work, we propose a fast and simple DA method consisting of three stages: (1) domain alignment by covariance matching, (2) pseudo-labeling, and (3) ensembling. We call this method $\textbf{PACE}$, for $\textbf{P}$seudo-labels, $\textbf{A}$lignment of $\textbf{C}$ovariances, and $\textbf{E}$nsembles. PACE is trained on top of fixed features extracted from an ensemble of modern pretrained backbones. PACE exceeds previous state-of-the-art by $\textbf{5 - 10 \%}$ on most benchmark adaptation tasks without training a neural network. PACE reduces training time and hyperparameter tuning time by $82\%$ and $97\%$, respectively, when compared to state-of-the-art DA methods. Code is released here: https://github.com/Chris210634/PACE-Domain-Adaptation

Related papers

• Mild Over-Parameterization Benefits Asymmetric Tensor PCA [12.923414933046574]
  Asymmetric PCA (ATPCA) is prototypical model for studying the trade-offs between sample complexity, computation, and memory.<n>We focus on the setting where $overlinek geq 4$ is even and consider descent-based algorithms under a limited memory budget.
  arXiv  Detail & Related papers  (2026-04-11T13:34:48Z)
• Domain2Vec: Vectorizing Datasets to Find the Optimal Data Mixture without Training [53.07879717463279]
  textscDomain2Vec decomposes any dataset into a linear combination of several emphmeta-domains<n>textscDomain2Vec helps find the data mixture that enhances downstream task performance with minimal computational overhead.
  arXiv  Detail & Related papers  (2025-06-12T17:53:51Z)
• $k$-SVD with Gradient Descent [18.260910876411973]
  We provide a gradient-descent method with a simple, universal rule for step-size selection that provably finds $k$-SVD for a matrix of any rank $d geq 1$.<n>Our convergence analysis reveals that the gradient method has an attractive region, and within this attractive region, the method behaves like Heron's method (a.k.a. the Babylonian method)<n>Our analytic results imply that the gradient method can be enhanced by means of Nesterov's momentum-based acceleration technique.
  arXiv  Detail & Related papers  (2025-02-01T05:00:28Z)
• QWO: Speeding Up Permutation-Based Causal Discovery in LiGAMs [20.661343069864888]
  We introduce QWO, a novel approach that significantly enhances the efficiency of computing $mathcalGpi$ for a given permutation $pi$. QWO has a speed-up of $O(n2)$ ($n$ is the number of variables) compared to the state-of-the-art BIC-based method, making it highly scalable.
  arXiv  Detail & Related papers  (2024-10-30T16:10:46Z)
• Exploiting Pre-trained Models for Drug Target Affinity Prediction with Nearest Neighbors [58.661454334877256]
  Drug-Target binding Affinity (DTA) prediction is essential for drug discovery. Despite the application of deep learning methods to DTA prediction, the achieved accuracy remain suboptimal. We propose $k$NN-DTA, a non-representation embedding-based retrieval method adopted on a pre-trained DTA prediction model.
  arXiv  Detail & Related papers  (2024-07-21T15:49:05Z)
• FedDA: Faster Framework of Local Adaptive Gradient Methods via Restarted Dual Averaging [104.41634756395545]
  Federated learning (FL) is an emerging learning paradigm to tackle massively distributed data. We propose textbfFedDA, a novel framework for local adaptive gradient methods. We show that textbfFedDA-MVR is the first adaptive FL algorithm that achieves this rate.
  arXiv  Detail & Related papers  (2023-02-13T05:10:30Z)
• Communication-Efficient Adam-Type Algorithms for Distributed Data Mining [93.50424502011626]
  We propose a class of novel distributed Adam-type algorithms (emphi.e., SketchedAMSGrad) utilizing sketching. Our new algorithm achieves a fast convergence rate of $O(frac1sqrtnT + frac1(k/d)2 T)$ with the communication cost of $O(k log(d))$ at each iteration.
  arXiv  Detail & Related papers  (2022-10-14T01:42:05Z)
• Blessing of Class Diversity in Pre-training [54.335530406959435]
  We prove that when the classes of the pre-training task are sufficiently diverse, pre-training can significantly improve the sample efficiency of downstream tasks. Our proof relies on a vector-form Rademacher complexity chain rule for composite function classes and a modified self-concordance condition.
  arXiv  Detail & Related papers  (2022-09-07T20:10:12Z)
• Asynchronous Training Schemes in Distributed Learning with Time Delay [17.259708772713164]
  In the context of distributed deep learning, the issue of stale weights or gradients could result in poor algorithmic performance. In this paper, we propose a different approach to tackle the issue of stale weights or gradients. One practical variant of PC-ASGD is also proposed by adopting a condition to help with the determination of the tradeoff parameter.
  arXiv  Detail & Related papers  (2022-08-28T07:14:59Z)
• Not All Models Are Equal: Predicting Model Transferability in a Self-challenging Fisher Space [51.62131362670815]
  This paper addresses the problem of ranking the pre-trained deep neural networks and screening the most transferable ones for downstream tasks. It proposes a new transferability metric called textbfSelf-challenging textbfFisher textbfDiscriminant textbfAnalysis (textbfSFDA)
  arXiv  Detail & Related papers  (2022-07-07T01:33:25Z)
• One Ring to Bring Them All: Towards Open-Set Recognition under Domain Shift [39.31881646860522]
  We propose a novel training scheme to learn a ($n$+1)-way to predict the $n$ source classes and the unknown class. For target adaptation, we simply adopt a weighted entropy minimization to adapt the source pretrained model to the unlabeled target domain. Our method surpasses current UNDA approaches which demand source data during adaptation on several benchmarks.
  arXiv  Detail & Related papers  (2022-06-07T21:39:54Z)
• Bi-level Alignment for Cross-Domain Crowd Counting [113.78303285148041]
  Current methods rely on external data for training an auxiliary task or apply an expensive coarse-to-fine estimation. We develop a new adversarial learning based method, which is simple and efficient to apply. We evaluate our approach on five real-world crowd counting benchmarks, where we outperform existing approaches by a large margin.
  arXiv  Detail & Related papers  (2022-05-12T02:23:25Z)
• Adversarial Unsupervised Domain Adaptation with Conditional and Label Shift: Infer, Align and Iterate [47.67549731439979]
  We propose an adversarial unsupervised domain adaptation (UDA) approach with the inherent conditional and label shifts. We infer the marginal $p(y)$ and align $p(x|y)$ iteratively in the training, and precisely align the posterior $p(y|x)$ in testing.
  arXiv  Detail & Related papers  (2021-07-28T16:28:01Z)
• AdaGDA: Faster Adaptive Gradient Descent Ascent Methods for Minimax Optimization [104.96004056928474]
  We propose a class of faster adaptive gradient descent methods for non-strongly-concave minimax problems. We show that our method reaches a lower sample complexity of $O(kappa2.5epsilon-3)$ with the mini-batch size $O(kappa)$.
  arXiv  Detail & Related papers  (2021-06-30T14:47:09Z)
• Accelerated Zeroth-Order and First-Order Momentum Methods from Mini to Minimax Optimization [133.53164856723782]
  We propose a new accelerated zeroth-order momentum (AccZOM) method for black-box mini-optimization where only function values can be obtained. Meanwhile, we propose an accelerated zeroth-order momentum descent (Acc-MDA) method for black-box minimax optimization, where only function values can be obtained. In particular, our Acc-MDA can obtain a lower gradient complexity of $tildeO(kappa_y2.5epsilon-3)$ with a batch size $O(kappa_y4)$.
  arXiv  Detail & Related papers  (2020-08-18T22:19:29Z)
• A No-Free-Lunch Theorem for MultiTask Learning [19.645741778058227]
  We consider a seemingly favorable classification scenario where all tasks $P_t$ share a common optimal classifier $h*,$. We show that, even though such regimes admit minimax rates accounting for both $n$ and $N$, no adaptive algorithm exists.
  arXiv  Detail & Related papers  (2020-06-29T03:03:29Z)
• Keep it Simple: Image Statistics Matching for Domain Adaptation [0.0]
  Domain Adaptation (DA) is a technique to maintain detection accuracy when only unlabeled images are available of the target domain. Recent state-of-the-art methods try to reduce the domain gap using an adversarial training strategy. We propose to align either color histograms or mean and covariance of the source images towards the target domain. In comparison to recent methods, we achieve state-of-the-art performance using a much simpler procedure for the training.
  arXiv  Detail & Related papers  (2020-05-26T07:32:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.