Fantastic Features and Where to Find Them: A Probing Method to combine Features from Multiple Foundation Models

• URL: http://arxiv.org/abs/2512.01405v1
• Date: Mon, 01 Dec 2025 08:26:08 GMT
• Title: Fantastic Features and Where to Find Them: A Probing Method to combine Features from Multiple Foundation Models
• Authors: Benjamin Ramtoula, Pierre-Yves Lajoie, Paul Newman, Daniele De Martini,
• Abstract summary: We propose a probing-based adapter that integrates features from multiple models and layers.<n>ComBo does not require dataset-specific tuning or backpropagation through the backbone models.<n>Our results demonstrate that ComBo offers a practical and general-purpose framework for combining diverse representations from multiple FMs.
• Score: 14.643457726166632
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Foundation models (FMs) trained with different objectives and data learn diverse representations, making some more effective than others for specific downstream tasks. Existing adaptation strategies, such as parameter-efficient fine-tuning, focus on individual models and do not exploit the complementary strengths across models. Probing methods offer a promising alternative by extracting information from frozen models, but current techniques do not scale well with large feature sets and often rely on dataset-specific hyperparameter tuning. We propose Combined backBones (ComBo), a simple and scalable probing-based adapter that effectively integrates features from multiple models and layers. ComBo compresses activations from layers of one or more FMs into compact token-wise representations and processes them with a lightweight transformer for task-specific prediction. Crucially, ComBo does not require dataset-specific tuning or backpropagation through the backbone models. However, not all models are equally relevant for all tasks. To address this, we introduce a mechanism that leverages ComBo's joint multi-backbone probing to efficiently evaluate each backbone's task-relevance, enabling both practical model comparison and improved performance through selective adaptation. On the 19 tasks of the VTAB-1k benchmark, ComBo outperforms previous probing methods, matches or surpasses more expensive alternatives, such as distillation-based model merging, and enables efficient probing of tuned models. Our results demonstrate that ComBo offers a practical and general-purpose framework for combining diverse representations from multiple FMs.

Related papers

• TayFCS: Towards Light Feature Combination Selection for Deep Recommender Systems [44.80081613834248]
  Taylor Expansion Scorer (TayScorer) module for field-wise Taylor expansion on the base model.<n> Logistic Regression Elimination (LRE) estimates the corresponding information gain based on the model prediction performance.
  arXiv  Detail & Related papers  (2025-07-05T04:22:42Z)
• Structural Similarity-Inspired Unfolding for Lightweight Image Super-Resolution [88.20464308588889]
  We propose a Structural Similarity-Inspired Unfolding (SSIU) method for efficient image SR.<n>This method is designed through unfolding an SR optimization function constrained by structural similarity.<n>Our model outperforms current state-of-the-art models, boasting lower parameter counts and reduced memory consumption.
  arXiv  Detail & Related papers  (2025-06-13T14:29:40Z)
• Dynamic Fisher-weighted Model Merging via Bayesian Optimization [37.02810891820468]
  Existing merging approaches typically involve scaling the parameters model-wise or integrating parameter importance parameter-wise.<n>We unify these strategies into a more general merging framework, and introduce Dynamic Fisher-weighted Merging (DF-Merge)<n>We show that DF-Merge outperforms strong baselines across models of different sizes and a variety of tasks.
  arXiv  Detail & Related papers  (2025-04-26T18:31:14Z)
• RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustness [28.437105789298244]
  RobustMerge is a training-free parameter-efficient merging method with complementary parameter adaptation to maintain direction robustness.<n>We establish a benchmark consisting of diverse multimodal tasks, on which we conduct experiments to certify the outstanding performance and generalizability of our method.
  arXiv  Detail & Related papers  (2025-02-24T13:52:05Z)
• Wolf2Pack: The AutoFusion Framework for Dynamic Parameter Fusion [4.164728134421114]
  We introduce AutoFusion, a framework that fuses distinct model parameters for multi-task learning without pre-trained checkpoints. We validate AutoFusion's effectiveness through experiments on commonly used benchmark datasets. Our framework offers a scalable and flexible solution for model integration, positioning it as a powerful tool for future research and practical applications.
  arXiv  Detail & Related papers  (2024-10-08T07:21:24Z)
• FusionBench: A Comprehensive Benchmark of Deep Model Fusion [78.80920533793595]
  Deep model fusion is a technique that unifies the predictions or parameters of several deep neural networks into a single model. FusionBench is the first comprehensive benchmark dedicated to deep model fusion.
  arXiv  Detail & Related papers  (2024-06-05T13:54:28Z)
• EMR-Merging: Tuning-Free High-Performance Model Merging [55.03509900949149]
  We show that Elect, Mask & Rescale-Merging (EMR-Merging) shows outstanding performance compared to existing merging methods. EMR-Merging is tuning-free, thus requiring no data availability or any additional training while showing impressive performance.
  arXiv  Detail & Related papers  (2024-05-23T05:25:45Z)
• Merging Multi-Task Models via Weight-Ensembling Mixture of Experts [64.94129594112557]
  Merging Transformer-based models trained on different tasks into a single unified model can execute all the tasks concurrently. Previous methods, exemplified by task arithmetic, have been proven to be both effective and scalable. We propose to merge most of the parameters while upscaling the Transformer layers to a weight-ensembling mixture of experts (MoE) module.
  arXiv  Detail & Related papers  (2024-02-01T08:58:57Z)
• A Lightweight Feature Fusion Architecture For Resource-Constrained Crowd Counting [3.5066463427087777]
  We introduce two lightweight models to enhance the versatility of crowd-counting models. These models maintain the same downstream architecture while incorporating two distinct backbones: MobileNet and MobileViT. We leverage Adjacent Feature Fusion to extract diverse scale features from a Pre-Trained Model (PTM) and subsequently combine these features seamlessly.
  arXiv  Detail & Related papers  (2024-01-11T15:13:31Z)
• Parameter Efficient Multi-task Model Fusion with Partial Linearization [97.23530944186078]
  We propose a novel method to improve multi-task fusion for parameter-efficient fine-tuning techniques. Our approach partially linearizes only the adapter modules and applies task arithmetic over the linearized adapters. We demonstrate that our partial linearization technique enables a more effective fusion of multiple tasks into a single model.
  arXiv  Detail & Related papers  (2023-10-07T08:55:54Z)
• Model ensemble instead of prompt fusion: a sample-specific knowledge transfer method for few-shot prompt tuning [85.55727213502402]
  We focus on improving the few-shot performance of prompt tuning by transferring knowledge from soft prompts of source tasks. We propose Sample-specific Ensemble of Source Models (SESoM) SESoM learns to adjust the contribution of each source model for each target sample separately when ensembling source model outputs.
  arXiv  Detail & Related papers  (2022-10-23T01:33:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.