Implicit Modeling for Transferability Estimation of Vision Foundation Models

• URL: http://arxiv.org/abs/2510.23145v1
• Date: Mon, 27 Oct 2025 09:21:19 GMT
• Title: Implicit Modeling for Transferability Estimation of Vision Foundation Models
• Authors: Yaoyan Zheng, Huiqun Wang, Nan Zhou, Di Huang,
• Abstract summary: Implicit Transferability Modeling (ITM) is a novel framework that implicitly models each model's intrinsic transferability.<n>ITM consistently outperforms existing methods in terms of stability, effectiveness, and efficiency.
• Score: 33.73062179456684
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Transferability estimation identifies the best pre-trained models for downstream tasks without incurring the high computational cost of full fine-tuning. This capability facilitates deployment and advances the pre-training and fine-tuning paradigm. However, existing methods often struggle to accurately assess transferability for emerging pre-trained models with diverse architectures, training strategies, and task alignments. In this work, we propose Implicit Transferability Modeling (ITM), a novel framework that implicitly models each model's intrinsic transferability, coupled with a Divide-and-Conquer Variational Approximation (DVA) strategy to efficiently approximate embedding space evolution. This design enables generalization across a broader range of models and downstream tasks. Extensive experiments on a comprehensive benchmark--spanning extensive training regimes and a wider variety of model types--demonstrate that ITM consistently outperforms existing methods in terms of stability, effectiveness, and efficiency.

Related papers

• Scaling and Transferability of Annealing Strategies in Large Language Model Training [59.443651879173025]
  We refine a predictive framework for optimizing annealing strategies under the Warmup-Steady-Decay (WSD) scheduler.<n>Our improved framework incorporates training steps, maximum learning rate, and annealing behavior, enabling more efficient optimization of learning rate schedules.<n>We validate our findings on extensive experiments using both Dense and Mixture-of-Experts (MoE) models.
  arXiv  Detail & Related papers  (2025-12-05T16:38:33Z)
• SAIL-Embedding Technical Report: Omni-modal Embedding Foundation Model [49.65930977591188]
  Multimodal embedding models aim to yield informative unified representations that empower diverse cross-modal tasks.<n>We introduce SAIL-Embedding, an omni-modal embedding foundation model that addresses these issues through tailored training strategies and architectural design.<n>Specifically, the content-aware progressive training aims to enhance the model's adaptability to diverse downstream tasks and master enriched cross-modal proficiency.<n>The collaboration-aware recommendation enhancement training further adapts multimodal representations for recommendation scenarios by distilling knowledge from sequence-to-item and ID-to-item embeddings.
  arXiv  Detail & Related papers  (2025-10-14T16:43:22Z)
• Align-Then-stEer: Adapting the Vision-Language Action Models through Unified Latent Guidance [63.33213516925946]
  We introduce textbfAlign-Then-stEer (textttATE), a novel, data-efficient, and plug-and-play adaptation framework.<n>Our work presents a general and lightweight solution that greatly enhances the practicality of deploying VLA models to new robotic platforms and tasks.
  arXiv  Detail & Related papers  (2025-09-02T07:51:59Z)
• Bounding Distributional Shifts in World Modeling through Novelty Detection [15.354352209595973]
  We use a variational autoencoder as a novelty detector to ensure that proposed action trajectories during planning do not cause the learned model to deviate from the training data distribution.<n>The proposed method improves over state-of-the-art solutions in terms of data efficiency.
  arXiv  Detail & Related papers  (2025-08-08T07:42:14Z)
• LEAD: Exploring Logit Space Evolution for Model Selection [39.30586021894318]
  We present LEAD, a finetuning-aligned approach based on the network output of logits.<n>The comprehensive experiments on 24 supervised and self-supervised pre-trained models across 10 downstream datasets demonstrate impressive performances.
  arXiv  Detail & Related papers  (2025-07-19T09:45:17Z)
• Domain Guidance: A Simple Transfer Approach for a Pre-trained Diffusion Model [62.11981915549919]
  Domain Guidance is a transfer approach that leverages pre-trained knowledge to guide the sampling process toward the target domain.<n>We demonstrate its substantial effectiveness across various transfer benchmarks, achieving over a 19.6% improvement in FID and a 23.4% improvement in FD$_textDINOv2$ compared to standard fine-tuning.
  arXiv  Detail & Related papers  (2025-04-02T09:07:55Z)
• Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging [75.93960998357812]
  Deep model merging represents an emerging research direction that combines multiple fine-tuned models to harness their capabilities across different tasks and domains.<n>Current model merging techniques focus on merging all available models simultaneously, with weight matrices-based methods being the predominant approaches.<n>We propose a training-free projection-based continual merging method that processes models sequentially.
  arXiv  Detail & Related papers  (2025-01-16T13:17:24Z)
• MITA: Bridging the Gap between Model and Data for Test-time Adaptation [68.62509948690698]
  Test-Time Adaptation (TTA) has emerged as a promising paradigm for enhancing the generalizability of models. We propose Meet-In-The-Middle based MITA, which introduces energy-based optimization to encourage mutual adaptation of the model and data from opposing directions.
  arXiv  Detail & Related papers  (2024-10-12T07:02:33Z)
• Structuring a Training Strategy to Robustify Perception Models with Realistic Image Augmentations [1.5723316845301678]
  This report introduces a novel methodology for training with augmentations to enhance model robustness and performance in such conditions. We present a comprehensive framework that includes identifying weak spots in Machine Learning models, selecting suitable augmentations, and devising effective training strategies. Experimental results demonstrate improvements in model performance, as measured by commonly used metrics such as mean Average Precision (mAP) and mean Intersection over Union (mIoU) on open-source object detection and semantic segmentation models and datasets.
  arXiv  Detail & Related papers  (2024-08-30T14:15:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.