Learning Conditional Invariances through Non-Commutativity

• URL: http://arxiv.org/abs/2402.11682v1
• Date: Sun, 18 Feb 2024 19:12:18 GMT
• Title: Learning Conditional Invariances through Non-Commutativity
• Authors: Abhra Chaudhuri, Serban Georgescu, Anjan Dutta
• Abstract summary: We show that a provably optimal and sample-efficient way of learning conditional invariances is by relaxing the invariance criterion to be non-commutatively directed towards the target domain. We prove that non-commutativity steers the optimization towards $Phi*_tau$ instead of $varphi*$, bringing the $mathcalH$-divergence between domains down to zero.
• Score: 4.820252317855078
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Invariance learning algorithms that conditionally filter out domain-specific random variables as distractors, do so based only on the data semantics, and not the target domain under evaluation. We show that a provably optimal and sample-efficient way of learning conditional invariances is by relaxing the invariance criterion to be non-commutatively directed towards the target domain. Under domain asymmetry, i.e., when the target domain contains semantically relevant information absent in the source, the risk of the encoder $\varphi^*$ that is optimal on average across domains is strictly lower-bounded by the risk of the target-specific optimal encoder $\Phi^*_\tau$. We prove that non-commutativity steers the optimization towards $\Phi^*_\tau$ instead of $\varphi^*$, bringing the $\mathcal{H}$-divergence between domains down to zero, leading to a stricter bound on the target risk. Both our theory and experiments demonstrate that non-commutative invariance (NCI) can leverage source domain samples to meet the sample complexity needs of learning $\Phi^*_\tau$, surpassing SOTA invariance learning algorithms for domain adaptation, at times by over $2\%$, approaching the performance of an oracle. Implementation is available at https://github.com/abhrac/nci.

Related papers

• AdaTriplet-RA: Domain Matching via Adaptive Triplet and Reinforced Attention for Unsupervised Domain Adaptation [15.905869933337101]
  Unsupervised domain adaption (UDA) is a transfer learning task where the data and annotations of the source domain are available but only have access to the unlabeled target data during training. We propose to improve the unsupervised domain adaptation task with an inter-domain sample matching scheme. We apply the widely-used and robust Triplet loss to match the inter-domain samples. To reduce the catastrophic effect of the inaccurate pseudo-labels generated during training, we propose a novel uncertainty measurement method to select reliable pseudo-labels automatically and progressively refine them.
  arXiv  Detail & Related papers  (2022-11-16T13:04:24Z)
• Divide and Contrast: Source-free Domain Adaptation via Adaptive Contrastive Learning [122.62311703151215]
  Divide and Contrast (DaC) aims to connect the good ends of both worlds while bypassing their limitations. DaC divides the target data into source-like and target-specific samples, where either group of samples is treated with tailored goals. We further align the source-like domain with the target-specific samples using a memory bank-based Maximum Mean Discrepancy (MMD) loss to reduce the distribution mismatch.
  arXiv  Detail & Related papers  (2022-11-12T09:21:49Z)
• Understanding Gradual Domain Adaptation: Improved Analysis, Optimal Path and Beyond [20.518134448156744]
  Gradual domain adaptation (GDA) assumes a path of $(T-1)$ unlabeled intermediate domains bridging the source and target. We prove a significantly improved generalization bound as $widetildeOleft(varepsilon_0+Oleft(sqrtlog(T)/nright)$, where $Delta$ is the average distributional distance between consecutive domains.
  arXiv  Detail & Related papers  (2022-04-18T07:39:23Z)
• Domain-shift adaptation via linear transformations [11.541238742226199]
  A predictor, $f_A, learned with data from a source domain (A) might not be accurate on a target domain (B) when their distributions are different. We propose an approach to project the source and target domains into a lower-dimensional, common space. We show the effectiveness of our approach in simulated data and in binary digit classification tasks, obtaining improvements up to 48% accuracy when correcting for the domain shift in the data.
  arXiv  Detail & Related papers  (2022-01-14T02:49:03Z)
• Certainty Volume Prediction for Unsupervised Domain Adaptation [35.984559137218504]
  Unsupervised domain adaptation (UDA) deals with the problem of classifying unlabeled target domain data. We propose a novel uncertainty-aware domain adaptation setup that models uncertainty as a multivariate Gaussian distribution in feature space. We evaluate our proposed pipeline on challenging UDA datasets and achieve state-of-the-art results.
  arXiv  Detail & Related papers  (2021-11-03T11:22:55Z)
• KL Guided Domain Adaptation [88.19298405363452]
  Domain adaptation is an important problem and often needed for real-world applications. A common approach in the domain adaptation literature is to learn a representation of the input that has the same distributions over the source and the target domain. We show that with a probabilistic representation network, the KL term can be estimated efficiently via minibatch samples.
  arXiv  Detail & Related papers  (2021-06-14T22:24:23Z)
• OVANet: One-vs-All Network for Universal Domain Adaptation [78.86047802107025]
  Existing methods manually set a threshold to reject unknown samples based on validation or a pre-defined ratio of unknown samples. We propose a method to learn the threshold using source samples and to adapt it to the target domain. Our idea is that a minimum inter-class distance in the source domain should be a good threshold to decide between known or unknown in the target.
  arXiv  Detail & Related papers  (2021-04-07T18:36:31Z)
• Self-training Avoids Using Spurious Features Under Domain Shift [54.794607791641745]
  In unsupervised domain adaptation, conditional entropy minimization and pseudo-labeling work even when the domain shifts are much larger than those analyzed by existing theory. We identify and analyze one particular setting where the domain shift can be large, but certain spurious features correlate with label in the source domain but are independent label in the target.
  arXiv  Detail & Related papers  (2020-06-17T17:51:42Z)
• Uncertainty-Aware Consistency Regularization for Cross-Domain Semantic Segmentation [63.75774438196315]
  Unsupervised domain adaptation (UDA) aims to adapt existing models of the source domain to a new target domain with only unlabeled data. Most existing methods suffer from noticeable negative transfer resulting from either the error-prone discriminator network or the unreasonable teacher model. We propose an uncertainty-aware consistency regularization method for cross-domain semantic segmentation.
  arXiv  Detail & Related papers  (2020-04-19T15:30:26Z)
• A Balanced and Uncertainty-aware Approach for Partial Domain Adaptation [142.31610972922067]
  This work addresses the unsupervised domain adaptation problem, especially in the case of class labels in the target domain being only a subset of those in the source domain. We build on domain adversarial learning and propose a novel domain adaptation method BA$3$US with two new techniques termed Balanced Adversarial Alignment (BAA) and Adaptive Uncertainty Suppression (AUS) Experimental results on multiple benchmarks demonstrate our BA$3$US surpasses state-of-the-arts for partial domain adaptation tasks.
  arXiv  Detail & Related papers  (2020-03-05T11:37:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.