Continuous Invariance Learning

• URL: http://arxiv.org/abs/2310.05348v2
• Date: Tue, 23 Apr 2024 03:24:08 GMT
• Title: Continuous Invariance Learning
• Authors: Yong Lin, Fan Zhou, Lu Tan, Lintao Ma, Jiameng Liu, Yansu He, Yuan Yuan, Yu Liu, James Zhang, Yujiu Yang, Hao Wang,
• Abstract summary: We show that existing invariance learning methods can fail for continuous domain problems. We propose Continuous Invariance Learning (CIL), which extracts invariant features across continuously indexed domains. CIL consistently outperforms strong baselines among all the tasks.
• Score: 37.5006565403112
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Invariance learning methods aim to learn invariant features in the hope that they generalize under distributional shifts. Although many tasks are naturally characterized by continuous domains, current invariance learning techniques generally assume categorically indexed domains. For example, auto-scaling in cloud computing often needs a CPU utilization prediction model that generalizes across different times (e.g., time of a day and date of a year), where `time' is a continuous domain index. In this paper, we start by theoretically showing that existing invariance learning methods can fail for continuous domain problems. Specifically, the naive solution of splitting continuous domains into discrete ones ignores the underlying relationship among domains, and therefore potentially leads to suboptimal performance. To address this challenge, we then propose Continuous Invariance Learning (CIL), which extracts invariant features across continuously indexed domains. CIL is a novel adversarial procedure that measures and controls the conditional independence between the labels and continuous domain indices given the extracted features. Our theoretical analysis demonstrates the superiority of CIL over existing invariance learning methods. Empirical results on both synthetic and real-world datasets (including data collected from production systems) show that CIL consistently outperforms strong baselines among all the tasks.

Related papers

• Algorithmic Fairness Generalization under Covariate and Dependence Shifts Simultaneously [28.24666589680547]
  We introduce a simple but effective approach that aims to learn a fair and invariant classifier. By augmenting various synthetic data domains through the model, we learn a fair and invariant classifier in source domains. This classifier can then be generalized to unknown target domains, maintaining both model prediction and fairness concerns.
  arXiv  Detail & Related papers  (2023-11-23T05:52:00Z)
• Multi-Domain Causal Representation Learning via Weak Distributional Invariances [27.72497122405241]
  Causal representation learning has emerged as the center of action in causal machine learning research. We show that autoencoders that incorporate such invariances can provably identify the stable set of latents from the rest across different settings.
  arXiv  Detail & Related papers  (2023-10-04T14:41:41Z)
• DIVERSIFY: A General Framework for Time Series Out-of-distribution Detection and Generalization [58.704753031608625]
  Time series is one of the most challenging modalities in machine learning research. OOD detection and generalization on time series tend to suffer due to its non-stationary property. We propose DIVERSIFY, a framework for OOD detection and generalization on dynamic distributions of time series.
  arXiv  Detail & Related papers  (2023-08-04T12:27:11Z)
• Generalized Representations Learning for Time Series Classification [28.230863650758447]
  We argue that the temporal complexity attributes to the unknown latent distributions within time series classification. We present experiments on gesture recognition, speech commands recognition, wearable stress and affect detection, and sensor-based human activity recognition.
  arXiv  Detail & Related papers  (2022-09-15T03:36:31Z)
• Domain Generalization via Selective Consistency Regularization for Time Series Classification [16.338176636365752]
  Domain generalization methods aim to learn models robust to domain shift with data from a limited number of source domains. We propose a novel representation learning methodology that selectively enforces prediction consistency between source domains.
  arXiv  Detail & Related papers  (2022-06-16T01:57:35Z)
• On Generalizing Beyond Domains in Cross-Domain Continual Learning [91.56748415975683]
  Deep neural networks often suffer from catastrophic forgetting of previously learned knowledge after learning a new task. Our proposed approach learns new tasks under domain shift with accuracy boosts up to 10% on challenging datasets such as DomainNet and OfficeHome.
  arXiv  Detail & Related papers  (2022-03-08T09:57:48Z)
• Uncertainty Modeling for Out-of-Distribution Generalization [56.957731893992495]
  We argue that the feature statistics can be properly manipulated to improve the generalization ability of deep learning models. Common methods often consider the feature statistics as deterministic values measured from the learned features. We improve the network generalization ability by modeling the uncertainty of domain shifts with synthesized feature statistics during training.
  arXiv  Detail & Related papers  (2022-02-08T16:09:12Z)
• Improving Out-of-Distribution Robustness via Selective Augmentation [61.147630193060856]
  Machine learning algorithms assume that training and test examples are drawn from the same distribution. distribution shift is a common problem in real-world applications and can cause models to perform dramatically worse at test time. We propose a mixup-based technique which learns invariant functions via selective augmentation called LISA.
  arXiv  Detail & Related papers  (2022-01-02T05:58:33Z)
• Contrastive learning of strong-mixing continuous-time stochastic processes [53.82893653745542]
  Contrastive learning is a family of self-supervised methods where a model is trained to solve a classification task constructed from unlabeled data. We show that a properly constructed contrastive learning task can be used to estimate the transition kernel for small-to-mid-range intervals in the diffusion case.
  arXiv  Detail & Related papers  (2021-03-03T23:06:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.