Flatten Long-Range Loss Landscapes for Cross-Domain Few-Shot Learning

• URL: http://arxiv.org/abs/2403.00567v2
• Date: Fri, 19 Apr 2024 03:17:16 GMT
• Title: Flatten Long-Range Loss Landscapes for Cross-Domain Few-Shot Learning
• Authors: Yixiong Zou, Yicong Liu, Yiman Hu, Yuhua Li, Ruixuan Li,
• Abstract summary: Cross-domain few-shot learning aims to acquire knowledge from limited training data in the target domain. We introduce a new normalization layer that replaces the original one in both CNNs and ViTs. Our method achieves performance improvements of up to 9% compared to the current best approaches on individual datasets.
• Score: 8.729669566501103
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Cross-domain few-shot learning (CDFSL) aims to acquire knowledge from limited training data in the target domain by leveraging prior knowledge transferred from source domains with abundant training samples. CDFSL faces challenges in transferring knowledge across dissimilar domains and fine-tuning models with limited training data. To address these challenges, we initially extend the analysis of loss landscapes from the parameter space to the representation space, which allows us to simultaneously interpret the transferring and fine-tuning difficulties of CDFSL models. We observe that sharp minima in the loss landscapes of the representation space result in representations that are hard to transfer and fine-tune. Moreover, existing flatness-based methods have limited generalization ability due to their short-range flatness. To enhance the transferability and facilitate fine-tuning, we introduce a simple yet effective approach to achieve long-range flattening of the minima in the loss landscape. This approach considers representations that are differently normalized as minima in the loss landscape and flattens the high-loss region in the middle by randomly sampling interpolated representations. We implement this method as a new normalization layer that replaces the original one in both CNNs and ViTs. This layer is simple and lightweight, introducing only a minimal number of additional parameters. Experimental results on 8 datasets demonstrate that our approach outperforms state-of-the-art methods in terms of average accuracy. Moreover, our method achieves performance improvements of up to 9\% compared to the current best approaches on individual datasets. Our code will be released.

Related papers

• Self-Supervised Pretraining for Aerial Road Extraction [11.311414617703308]
  We propose a self-supervised pretraining method for deep neural networks for aerial image segmentation. Our approach uses inpainting-based pretraining, where the model learns to reconstruct missing regions in aerial images. Experiments show that our pretraining significantly boosts segmentation accuracy, especially in low-data regimes.
  arXiv  Detail & Related papers  (2025-03-31T17:14:08Z)
• Unbiased Max-Min Embedding Classification for Transductive Few-Shot Learning: Clustering and Classification Are All You Need [83.10178754323955]
  Few-shot learning enables models to generalize from only a few labeled examples. We propose the Unbiased Max-Min Embedding Classification (UMMEC) Method, which addresses the key challenges in few-shot learning. Our method significantly improves classification performance with minimal labeled data, advancing the state-of-the-art in annotatedL.
  arXiv  Detail & Related papers  (2025-03-28T07:23:07Z)
• The Devil is in Low-Level Features for Cross-Domain Few-Shot Segmentation [22.443834719018795]
  Cross-Domain Few-Shot (CDFSS) is proposed to transfer the pixel-level segmentation capabilities learned from large-scale source-domain datasets to downstream target-domain datasets. We focus on a well-observed but unresolved phenomenon in CDFSS: for target domains, segmentation performance peaks at the very early epochs, and declines sharply as the source-domain training proceeds. We propose a method that includes two plug-and-play modules: one to flatten the loss landscapes for low-level features during source-domain training as a novel sharpness-aware method, and the other to directly supplement target-
  arXiv  Detail & Related papers  (2025-03-27T04:37:52Z)
• Simplicity bias and optimization threshold in two-layer ReLU networks [24.43739371803548]
  We show that despite overparametrization, networks converge toward simpler solutions rather than interpolating the training data. Our analysis relies on the so called early alignment phase, during which neurons align towards specific directions.
  arXiv  Detail & Related papers  (2024-10-03T09:58:57Z)
• Discriminative Sample-Guided and Parameter-Efficient Feature Space Adaptation for Cross-Domain Few-Shot Learning [0.0]
  Cross-domain few-shot classification presents the challenging task of learning new classes in previously unseen domains. We introduce a lightweight parameter-efficient adaptation strategy to address overfitting associated with fine-tuning a large number of parameters on small datasets. We replace the traditional nearest centroid with a discriminative sample-aware loss function, enhancing the model's sensitivity to the inter- and intra-class variances.
  arXiv  Detail & Related papers  (2024-03-07T13:49:29Z)
• Informative Data Mining for One-Shot Cross-Domain Semantic Segmentation [84.82153655786183]
  We propose a novel framework called Informative Data Mining (IDM) to enable efficient one-shot domain adaptation for semantic segmentation. IDM provides an uncertainty-based selection criterion to identify the most informative samples, which facilitates quick adaptation and reduces redundant training. Our approach outperforms existing methods and achieves a new state-of-the-art one-shot performance of 56.7%/55.4% on the GTA5/SYNTHIA to Cityscapes adaptation tasks.
  arXiv  Detail & Related papers  (2023-09-25T15:56:01Z)
• GIFD: A Generative Gradient Inversion Method with Feature Domain Optimization [52.55628139825667]
  Federated Learning (FL) has emerged as a promising distributed machine learning framework to preserve clients' privacy. Recent studies find that an attacker can invert the shared gradients and recover sensitive data against an FL system by leveraging pre-trained generative adversarial networks (GAN) as prior knowledge. We propose textbfGradient textbfInversion over textbfFeature textbfDomains (GIFD), which disassembles the GAN model and searches the feature domains of the intermediate layers.
  arXiv  Detail & Related papers  (2023-08-09T04:34:21Z)
• A Novel Cross-Perturbation for Single Domain Generalization [54.612933105967606]
  Single domain generalization aims to enhance the ability of the model to generalize to unknown domains when trained on a single source domain. The limited diversity in the training data hampers the learning of domain-invariant features, resulting in compromised generalization performance. We propose CPerb, a simple yet effective cross-perturbation method to enhance the diversity of the training data.
  arXiv  Detail & Related papers  (2023-08-02T03:16:12Z)
• Semi-Supervised Wide-Angle Portraits Correction by Multi-Scale Transformer [17.455782652441187]
  We propose a semi-supervised network for wide-angle portraits correction. Our network, named as Multi-Scale Swin-Unet (MS-Unet), is built upon the multi-scale swin transformer block (MSTB)
  arXiv  Detail & Related papers  (2021-09-14T09:40:25Z)
• Pixel-Level Cycle Association: A New Perspective for Domain Adaptive Semantic Segmentation [169.82760468633236]
  We propose to build the pixel-level cycle association between source and target pixel pairs. Our method can be trained end-to-end in one stage and introduces no additional parameters.
  arXiv  Detail & Related papers  (2020-10-31T00:11:36Z)
• Domain-invariant Similarity Activation Map Contrastive Learning for Retrieval-based Long-term Visual Localization [30.203072945001136]
  In this work, a general architecture is first formulated probabilistically to extract domain invariant feature through multi-domain image translation. And then a novel gradient-weighted similarity activation mapping loss (Grad-SAM) is incorporated for finer localization with high accuracy. Extensive experiments have been conducted to validate the effectiveness of the proposed approach on the CMUSeasons dataset. Our performance is on par with or even outperforms the state-of-the-art image-based localization baselines in medium or high precision.
  arXiv  Detail & Related papers  (2020-09-16T14:43:22Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)
• FDA: Fourier Domain Adaptation for Semantic Segmentation [82.4963423086097]
  We describe a simple method for unsupervised domain adaptation, whereby the discrepancy between the source and target distributions is reduced by swapping the low-frequency spectrum of one with the other. We illustrate the method in semantic segmentation, where densely annotated images are aplenty in one domain, but difficult to obtain in another. Our results indicate that even simple procedures can discount nuisance variability in the data that more sophisticated methods struggle to learn away.
  arXiv  Detail & Related papers  (2020-04-11T22:20:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.