Feature Projection Learning for Better Vision-Language Reasoning

• URL: http://arxiv.org/abs/2601.20224v1
• Date: Wed, 28 Jan 2026 03:54:36 GMT
• Title: Feature Projection Learning for Better Vision-Language Reasoning
• Authors: Yi Zhang, Weicheng Lin, Liang-Jie Zhang,
• Abstract summary: We propose a method called textittextbfFeature textbfProjection textbfL(FPL) to address these problems.<n>FPL delivers superior accuracy, surpassing the current state-of-the-art methods by a substantial margin.
• Score: 9.360663595659988
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language Pre-Trained models, notably CLIP, that utilize contrastive learning have proven highly adept at extracting generalizable visual features. To inherit the well-learned knowledge of VLP models for downstream tasks, several approaches aim to adapt them efficiently with limited supervision. However, these methods either suffer from limited performance, excessive learnable parameters, or extended training times, all of which hinder their effectiveness in adapting the CLIP model to downstream tasks. In this work, we propose a simple yet efficient and effective method called \textit{\textbf{F}eature \textbf{P}rojection \textbf{L}earning(FPL)} to address these problems. Specifically, we develop a projection model that projects class prototype features into the query image feature space and reconstructs the query image feature map. The negative average squared reconstruction error is used as the class score. In this way, we transform the classification problem into a feature projection problem. The final output of this method is a combination of the prediction from the projection model and the original pre-trained CLIP. Comprehensive empirical evaluations confirm that FPL delivers superior accuracy, surpassing the current state-of-the-art methods by a substantial margin.

Related papers

• Attribution-guided Pruning for Compression, Circuit Discovery, and Targeted Correction in LLMs [15.23174472320989]
  Large Language Models (LLMs) are central to many contemporary AI applications.<n>Recent works in eXplainable AI (XAI) suggest that interpretability can also enable model compression.
  arXiv  Detail & Related papers  (2025-06-16T17:38:36Z)
• Test-time Distribution Learning Adapter for Cross-modal Visual Reasoning [16.998833621046117]
  We propose Test-Time Distribution LearNing Adapter (TT-DNA) which directly works during the testing period. Specifically, we estimate Gaussian distributions to model visual features of the few-shot support images to capture the knowledge from the support set. Our extensive experimental results on visual reasoning for human object interaction demonstrate that our proposed TT-DNA outperforms existing state-of-the-art methods by large margins.
  arXiv  Detail & Related papers  (2024-03-10T01:34:45Z)
• Spurious Feature Eraser: Stabilizing Test-Time Adaptation for Vision-Language Foundation Model [86.9619638550683]
  Vision-language foundation models have exhibited remarkable success across a multitude of downstream tasks due to their scalability on extensive image-text paired data.<n>However, these models display significant limitations when applied to downstream tasks, such as fine-grained image classification, as a result of decision shortcuts''
  arXiv  Detail & Related papers  (2024-03-01T09:01:53Z)
• FD-Align: Feature Discrimination Alignment for Fine-tuning Pre-Trained Models in Few-Shot Learning [21.693779973263172]
  In this paper, we introduce a fine-tuning approach termed Feature Discrimination Alignment (FD-Align) Our method aims to bolster the model's generalizability by preserving the consistency of spurious features. Once fine-tuned, the model can seamlessly integrate with existing methods, leading to performance improvements.
  arXiv  Detail & Related papers  (2023-10-23T17:12:01Z)
• Unsupervised Prototype Adapter for Vision-Language Models [29.516767588241724]
  We design an unsupervised fine-tuning approach for vision-language models called Unsupervised Prototype Adapter (UP-Adapter) Specifically, for the unannotated target datasets, we leverage the text-image aligning capability of CLIP to automatically select the most confident samples for each class. After fine-tuning, the prototype model prediction is combined with the original CLIP's prediction by a residual connection to perform downstream recognition tasks.
  arXiv  Detail & Related papers  (2023-08-22T15:28:49Z)
• RanPAC: Random Projections and Pre-trained Models for Continual Learning [59.07316955610658]
  Continual learning (CL) aims to learn different tasks (such as classification) in a non-stationary data stream without forgetting old ones. We propose a concise and effective approach for CL with pre-trained models.
  arXiv  Detail & Related papers  (2023-07-05T12:49:02Z)
• Leaving the Nest: Going Beyond Local Loss Functions for Predict-Then-Optimize [57.22851616806617]
  We show that our method achieves state-of-the-art results in four domains from the literature. Our approach outperforms the best existing method by nearly 200% when the localness assumption is broken.
  arXiv  Detail & Related papers  (2023-05-26T11:17:45Z)
• Defense-Prefix for Preventing Typographic Attacks on CLIP [14.832208701208414]
  Some adversarial attacks fool a model into false or absurd classifications. We introduce our simple yet effective method: Defense-Prefix (DP), which inserts the DP token before a class name to make words "robust" against typographic attacks. Our method significantly improves the accuracy of classification tasks for typographic attack datasets, while maintaining the zero-shot capabilities of the model.
  arXiv  Detail & Related papers  (2023-04-10T11:05:20Z)
• CLIP-guided Prototype Modulating for Few-shot Action Recognition [49.11385095278407]
  This work aims to transfer the powerful multimodal knowledge of CLIP to alleviate the inaccurate prototype estimation issue. We present a CLIP-guided prototype modulating framework called CLIP-FSAR, which consists of a video-text contrastive objective and a prototype modulation.
  arXiv  Detail & Related papers  (2023-03-06T09:17:47Z)
• CLIPood: Generalizing CLIP to Out-of-Distributions [73.86353105017076]
  Contrastive language-image pre-training (CLIP) models have shown impressive zero-shot ability, but the further adaptation of CLIP on downstream tasks undesirably degrades OOD performances. We propose CLIPood, a fine-tuning method that can adapt CLIP models to OOD situations where both domain shifts and open classes may occur on unseen test data. Experiments on diverse datasets with different OOD scenarios show that CLIPood consistently outperforms existing generalization techniques.
  arXiv  Detail & Related papers  (2023-02-02T04:27:54Z)
• Model-Agnostic Multitask Fine-tuning for Few-shot Vision-Language Transfer Learning [59.38343286807997]
  We propose Model-Agnostic Multitask Fine-tuning (MAMF) for vision-language models on unseen tasks. Compared with model-agnostic meta-learning (MAML), MAMF discards the bi-level optimization and uses only first-order gradients. We show that MAMF consistently outperforms the classical fine-tuning method for few-shot transfer learning on five benchmark datasets.
  arXiv  Detail & Related papers  (2022-03-09T17:26:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.