Exploring Representation Invariance in Finetuning

• URL: http://arxiv.org/abs/2503.07399v3
• Date: Sun, 05 Oct 2025 01:23:17 GMT
• Title: Exploring Representation Invariance in Finetuning
• Authors: Wenqiang Zu, Shenghao Xie, Hao Chen, Zhiqiang Chen, Liwen Hu, Yuanhao Xi, Yiming Liang, Junliang Ye, Bo Lei, Tiejun Huang, Guoqi Li, Lei Ma,
• Abstract summary: Foundation models pretrained on large-scale natural images are widely adapted to various cross-domain low-resource downstream tasks.<n>We argue that such tasks can be effectively adapted without sacrificing the benefits of pretrained representations.<n>We introduce textitRepresentation Invariance FineTuning (RIFT), a regularization that maximizes the representation similarity between pretrained and finetuned models.
• Score: 51.19872959859021
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Foundation models pretrained on large-scale natural images are widely adapted to various cross-domain low-resource downstream tasks, benefiting from generalizable and transferable patterns captured by their representations. However, these representations are later found to gradually vanish during finetuning, accompanied by a degradation of model's original generalizability. In this paper, we argue that such tasks can be effectively adapted without sacrificing the benefits of pretrained representations. We approach this by introducing \textit{Representation Invariance FineTuning (RIFT)}, a regularization that maximizes the representation similarity between pretrained and finetuned models by leveraging orthogonal invariance of manifolds in a computationally efficient way. Experiments demonstrate that our method is compatible with mainstream finetuning methods, offering competitive or even enhanced performance and better preservation of the generalizability.

Related papers

• Interpretability-Aware Pruning for Efficient Medical Image Analysis [1.747623282473278]
  We introduce an interpretability-guided pruning framework that reduces model complexity while preserving both predictive performance and transparency.<n> Experiments across multiple medical image classification benchmarks demonstrate that this approach achieves high compression rates with minimal loss in accuracy.
  arXiv  Detail & Related papers  (2025-07-11T05:58:22Z)
• Boundless Across Domains: A New Paradigm of Adaptive Feature and Cross-Attention for Domain Generalization in Medical Image Segmentation [1.93061220186624]
  Domain-invariant representation learning is a powerful method for domain generalization. Previous approaches face challenges such as high computational demands, training instability, and limited effectiveness with high-dimensional data. We propose an Adaptive Feature Blending (AFB) method that generates out-of-distribution samples while exploring the in-distribution space.
  arXiv  Detail & Related papers  (2024-11-22T12:06:24Z)
• Fast constrained sampling in pre-trained diffusion models [77.21486516041391]
  Diffusion models have dominated the field of large, generative image models. We propose an algorithm for fast-constrained sampling in large pre-trained diffusion models.
  arXiv  Detail & Related papers  (2024-10-24T14:52:38Z)
• Towards Unsupervised Blind Face Restoration using Diffusion Prior [12.69610609088771]
  Blind face restoration methods have shown remarkable performance when trained on large-scale synthetic datasets with supervised learning. These datasets are often generated by simulating low-quality face images with a handcrafted image degradation pipeline. In this paper, we address this issue by using only a set of input images, with unknown degradations and without ground truth targets, to fine-tune a restoration model. Our best model also achieves the state-of-the-art results on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2024-10-06T20:38:14Z)
• Adaptive Correspondence Scoring for Unsupervised Medical Image Registration [9.294341405888158]
  Existing methods rely on image reconstruction as the primary supervision signal. We propose an adaptive framework that re-weights the error residuals with a correspondence scoring map during training. Our framework consistently outperforms other methods both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2023-12-01T01:11:22Z)
• PGDiff: Guiding Diffusion Models for Versatile Face Restoration via Partial Guidance [65.5618804029422]
  Previous works have achieved noteworthy success by limiting the solution space using explicit degradation models. We propose PGDiff by introducing partial guidance, a fresh perspective that is more adaptable to real-world degradations. Our method not only outperforms existing diffusion-prior-based approaches but also competes favorably with task-specific models.
  arXiv  Detail & Related papers  (2023-09-19T17:51:33Z)
• Supervised Contrastive Learning with Heterogeneous Similarity for Distribution Shifts [3.7819322027528113]
  We propose a new regularization using the supervised contrastive learning to prevent such overfitting and to train models that do not degrade their performance under the distribution shifts. Experiments on benchmark datasets that emulate distribution shifts, including subpopulation shift and domain generalization, demonstrate the advantage of the proposed method.
  arXiv  Detail & Related papers  (2023-04-07T01:45:09Z)
• Masked Images Are Counterfactual Samples for Robust Fine-tuning [77.82348472169335]
  Fine-tuning deep learning models can lead to a trade-off between in-distribution (ID) performance and out-of-distribution (OOD) robustness. We propose a novel fine-tuning method, which uses masked images as counterfactual samples that help improve the robustness of the fine-tuning model.
  arXiv  Detail & Related papers  (2023-03-06T11:51:28Z)
• Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective [51.70661197256033]
  We propose ARCO, a semi-supervised contrastive learning framework with stratified group theory for medical image segmentation. We first propose building ARCO through the concept of variance-reduced estimation and show that certain variance-reduction techniques are particularly beneficial in pixel/voxel-level segmentation tasks. We experimentally validate our approaches on eight benchmarks, i.e., five 2D/3D medical and three semantic segmentation datasets, with different label settings.
  arXiv  Detail & Related papers  (2023-02-03T13:50:25Z)
• ProCo: Prototype-aware Contrastive Learning for Long-tailed Medical Image Classification [12.399428395862639]
  We adopt the contrastive learning to tackle the long-tailed medical imbalance problem. The overall framework, namely as Prototype-aware Contrastive learning (ProCo), is unified as a single-stage pipeline. Our method outperforms the existing state-of-the-art methods by a large margin.
  arXiv  Detail & Related papers  (2022-09-01T02:24:16Z)
• (Certified!!) Adversarial Robustness for Free! [116.6052628829344]
  We certify 71% accuracy on ImageNet under adversarial perturbations constrained to be within a 2-norm of 0.5. We obtain these results using only pretrained diffusion models and image classifiers, without requiring any fine tuning or retraining of model parameters.
  arXiv  Detail & Related papers  (2022-06-21T17:27:27Z)
• Deblurring via Stochastic Refinement [85.42730934561101]
  We present an alternative framework for blind deblurring based on conditional diffusion models. Our method is competitive in terms of distortion metrics such as PSNR.
  arXiv  Detail & Related papers  (2021-12-05T04:36:09Z)
• A low-rank representation for unsupervised registration of medical images [10.499611180329804]
  We propose a novel approach based on a low-rank representation, i.e., Regnet-LRR, to tackle the problem of noisy data registration scenarios. We show that the low-rank representation can boost the ability and robustness of models as well as bring significant improvements in noisy data registration scenarios.
  arXiv  Detail & Related papers  (2021-05-20T07:04:10Z)
• Rank-smoothed Pairwise Learning In Perceptual Quality Assessment [26.599014990168836]
  We show that regularizing pairwise empirical probabilities with aggregated rankwise probabilities leads to a more reliable training loss. We show that training a deep image quality assessment model with our rank-smoothed loss consistently improves the accuracy of predicting human preferences.
  arXiv  Detail & Related papers  (2020-11-21T23:33:14Z)
• Posterior Differential Regularization with f-divergence for Improving Model Robustness [95.05725916287376]
  We focus on methods that regularize the model posterior difference between clean and noisy inputs. We generalize the posterior differential regularization to the family of $f$-divergences. Our experiments show that regularizing the posterior differential with $f$-divergence can result in well-improved model robustness.
  arXiv  Detail & Related papers  (2020-10-23T19:58:01Z)
• Learning Invariant Representations and Risks for Semi-supervised Domain Adaptation [109.73983088432364]
  We propose the first method that aims to simultaneously learn invariant representations and risks under the setting of semi-supervised domain adaptation (Semi-DA) We introduce the LIRR algorithm for jointly textbfLearning textbfInvariant textbfRepresentations and textbfRisks.
  arXiv  Detail & Related papers  (2020-10-09T15:42:35Z)
• Background Splitting: Finding Rare Classes in a Sea of Background [55.03789745276442]
  We focus on the real-world problem of training accurate deep models for image classification of a small number of rare categories. In these scenarios, almost all images belong to the background category in the dataset (>95% of the dataset is background) We demonstrate that both standard fine-tuning approaches and state-of-the-art approaches for training on imbalanced datasets do not produce accurate deep models in the presence of this extreme imbalance.
  arXiv  Detail & Related papers  (2020-08-28T23:05:15Z)
• Better Fine-Tuning by Reducing Representational Collapse [77.44854918334232]
  Existing approaches for fine-tuning pre-trained language models have been shown to be unstable. We present a method rooted in trust region theory that replaces previously used adversarial objectives with parametric noise. We show it is less prone to representation collapse; the pre-trained models maintain more generalizable representations every time they are fine-tuned.
  arXiv  Detail & Related papers  (2020-08-06T02:13:16Z)
• Robust model training and generalisation with Studentising flows [22.757298187704745]
  We discuss how these methods can be further improved based on insights from robust (in particular, resistant) statistics. We propose to endow flow-based models with fat-tailed latent distributions as a simple drop-in replacement for the Gaussian distribution. Experiments on several different datasets confirm the efficacy of the proposed approach.
  arXiv  Detail & Related papers  (2020-06-11T16:47:01Z)
• Convex Representation Learning for Generalized Invariance in Semi-Inner-Product Space [32.442549424823355]
  In this work we develop an algorithm for a variety of generalized representations in a semi-norms that representers in a lead, and bounds are established. This allows in representations to be learned efficiently and effectively as confirmed in our experiments along with accurate predictions.
  arXiv  Detail & Related papers  (2020-04-25T18:54:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.