Enhancing Compositional Generalization via Compositional Feature Alignment

• URL: http://arxiv.org/abs/2402.02851v2
• Date: Wed, 22 May 2024 05:34:28 GMT
• Title: Enhancing Compositional Generalization via Compositional Feature Alignment
• Authors: Haoxiang Wang, Haozhe Si, Huajie Shao, Han Zhao,
• Abstract summary: We develop CG-Bench, a suite of CG benchmarks derived from existing real-world image datasets. We propose Compositional Feature Alignment (CFA), a simple two-stage finetuning technique. We conduct experiments on CG-Bench for CLIP and DINOv2, two powerful pretrained vision foundation models.
• Score: 14.289836081158615
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Real-world applications of machine learning models often confront data distribution shifts, wherein discrepancies exist between the training and test data distributions. In the common multi-domain multi-class setup, as the number of classes and domains scales up, it becomes infeasible to gather training data for every domain-class combination. This challenge naturally leads the quest for models with Compositional Generalization (CG) ability, where models can generalize to unseen domain-class combinations. To delve into the CG challenge, we develop CG-Bench, a suite of CG benchmarks derived from existing real-world image datasets, and observe that the prevalent pretraining-finetuning paradigm on foundational models, such as CLIP and DINOv2, struggles with the challenge. To address this challenge, we propose Compositional Feature Alignment (CFA), a simple two-stage finetuning technique that i) learns two orthogonal linear heads on a pretrained encoder with respect to class and domain labels, and ii) fine-tunes the encoder with the newly learned head frozen. We theoretically and empirically justify that CFA encourages compositional feature learning of pretrained models. We further conduct extensive experiments on CG-Bench for CLIP and DINOv2, two powerful pretrained vision foundation models. Experiment results show that CFA outperforms common finetuning techniques in compositional generalization, corroborating CFA's efficacy in compositional feature learning.

Related papers

• High-Performance Few-Shot Segmentation with Foundation Models: An Empirical Study [64.06777376676513]
  We develop a few-shot segmentation (FSS) framework based on foundation models. To be specific, we propose a simple approach to extract implicit knowledge from foundation models to construct coarse correspondence. Experiments on two widely used datasets demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2024-09-10T08:04:11Z)
• Layerwise complexity-matched learning yields an improved model of cortical area V2 [12.861402235256207]
  Deep neural networks trained end-to-end for object recognition approach human capabilities. We develop a self-supervised training methodology that operates independently on successive layers. We show that our model is better aligned with selectivity properties and neural activity in primate area V2.
  arXiv  Detail & Related papers  (2023-12-18T18:37:02Z)
• Consistency Regularization for Generalizable Source-free Domain Adaptation [62.654883736925456]
  Source-free domain adaptation (SFDA) aims to adapt a well-trained source model to an unlabelled target domain without accessing the source dataset. Existing SFDA methods ONLY assess their adapted models on the target training set, neglecting the data from unseen but identically distributed testing sets. We propose a consistency regularization framework to develop a more generalizable SFDA method.
  arXiv  Detail & Related papers  (2023-08-03T07:45:53Z)
• Bilevel Fast Scene Adaptation for Low-Light Image Enhancement [50.639332885989255]
  Enhancing images in low-light scenes is a challenging but widely concerned task in the computer vision. Main obstacle lies in the modeling conundrum from distribution discrepancy across different scenes. We introduce the bilevel paradigm to model the above latent correspondence. A bilevel learning framework is constructed to endow the scene-irrelevant generality of the encoder towards diverse scenes.
  arXiv  Detail & Related papers  (2023-06-02T08:16:21Z)
• CFA: Class-wise Calibrated Fair Adversarial Training [31.812287233814295]
  We propose a textbfClass-wise calibrated textbfFair textbfAdversarial training framework, named CFA, which customizes specific training configurations for each class automatically. Our proposed CFA can improve both overall robustness and fairness notably over other state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-03-25T13:05:16Z)
• Learning from Mistakes: Self-Regularizing Hierarchical Representations in Point Cloud Semantic Segmentation [15.353256018248103]
  LiDAR semantic segmentation has gained attention to accomplish fine-grained scene understanding. We present a coarse-to-fine setup that LEArns from classification mistaKes (LEAK) derived from a standard model. Our LEAK approach is very general and can be seamlessly applied on top of any segmentation architecture.
  arXiv  Detail & Related papers  (2023-01-26T14:52:30Z)
• Style-Hallucinated Dual Consistency Learning: A Unified Framework for Visual Domain Generalization [113.03189252044773]
  We propose a unified framework, Style-HAllucinated Dual consistEncy learning (SHADE), to handle domain shift in various visual tasks. Our versatile SHADE can significantly enhance the generalization in various visual recognition tasks, including image classification, semantic segmentation and object detection.
  arXiv  Detail & Related papers  (2022-12-18T11:42:51Z)
• Motor Imagery Decoding Using Ensemble Curriculum Learning and Collaborative Training [11.157243900163376]
  Multi-subject EEG datasets present several kinds of domain shifts. These domain shifts impede robust cross-subject generalization. We propose a two-stage model ensemble architecture built with multiple feature extractors. We demonstrate that our model ensembling approach combines the powers of curriculum learning and collaborative training.
  arXiv  Detail & Related papers  (2022-11-21T13:45:44Z)
• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data [78.69828864672978]
  A central challenge in training classification models in the real-world federated system is learning with non-IID data. We propose a novel and simple algorithm called Virtual Representations (CCVR), which adjusts the classifier using virtual representations sampled from an approximated ssian mixture model. Experimental results demonstrate that CCVR state-of-the-art performance on popular federated learning benchmarks including CIFAR-10, CIFAR-100, and CINIC-10.
  arXiv  Detail & Related papers  (2021-06-09T12:02:29Z)
• Source-Free Open Compound Domain Adaptation in Semantic Segmentation [99.82890571842603]
  In SF-OCDA, only the source pre-trained model and the target data are available to learn the target model. We propose the Cross-Patch Style Swap (CPSS) to diversify samples with various patch styles in the feature-level. Our method produces state-of-the-art results on the C-Driving dataset.
  arXiv  Detail & Related papers  (2021-06-07T08:38:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.