CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models

• URL: http://arxiv.org/abs/2603.21077v2
• Date: Fri, 27 Mar 2026 06:54:52 GMT
• Title: CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models
• Abstract summary: We show that visual fine-tuning methods fail to consistently outperform the frozen baseline across multimodal tasks.<n>This instability arises from visual preference conflicts, where the context-agnostic nature of vision encoders induces divergent parameter updates.<n>We propose the Context- Vector Extraction (CoVFT) framework, which explicitly incorporates multimodal context into visual adaptation.
• Score: 33.73062179456684
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal large language models (MLLMs) achieve remarkable progress in cross-modal perception and reasoning, yet a fundamental question remains unresolved: should the vision encoder be fine-tuned or frozen? Despite the success of models such as LLaVA and Qwen-VL, inconsistent design choices and heterogeneous training setups hinder a unified understanding of visual fine-tuning (VFT) in MLLMs. Through a configuration-aligned benchmark, we find that existing VFT methods fail to consistently outperform the frozen baseline across multimodal tasks. Our analysis suggests that this instability arises from visual preference conflicts, where the context-agnostic nature of vision encoders induces divergent parameter updates under diverse multimodal context. To address this issue, we propose the Context-aware Visual Fine-tuning (CoVFT) framework, which explicitly incorporates multimodal context into visual adaptation. By integrating a Context Vector Extraction (CVE) and a Contextual Mixture-of-Experts (CoMoE) module, CoVFT decomposes conflicting optimization signals and enables stable, context-sensitive visual updates. Extensive experiments on 12 multimodal benchmarks demonstrate that CoVFT achieves state-of-the-art performance with superior stability. Notably, fine-tuning a 7B MLLM with CoVFT surpasses the average performance of its 13B counterpart, revealing substantial untapped potential in visual encoder optimization within MLLMs.

Related papers

• Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models [58.9387276447485]
  Vision Inference Former (VIF) is a lightweight architectural module that establishes a bridge between pure visual representations and the model's output space.<n>We conduct experiments on 14 benchmark tasks covering general reasoning, OCR, table understanding, vision-centric evaluation, and hallucination.
  arXiv  Detail & Related papers  (2026-05-18T10:04:22Z)
• VLM2Rec: Resolving Modality Collapse in Vision-Language Model Embedders for Multimodal Sequential Recommendation [15.053120780189461]
  We investigate the use of Vision-Language Models (VLMs) as multimodal encoders for Sequential Recommendation (SR)<n>We find that standard contrastive supervised fine-tuning (SFT) can amplify its inherent modality collapse.<n>We propose VLM2Rec, a VLM embedder-based framework for sequential recommendation designed to ensure balanced modality utilization.
  arXiv  Detail & Related papers  (2026-03-18T07:46:30Z)
• Refer-Agent: A Collaborative Multi-Agent System with Reasoning and Reflection for Referring Video Object Segmentation [50.22481337087162]
  Referring Video Object (RVOS) aims to segment objects in videos based on textual queries.<n>Refer-Agent is a collaborative multi-agent system with alternating reasoning-reflection mechanisms.
  arXiv  Detail & Related papers  (2026-02-03T14:48:12Z)
• Vision-Centric Activation and Coordination for Multimodal Large Language Models [42.26911585599856]
  Multimodal large language models (MLLMs) integrate image features from visual encoders with LLMs, demonstrating advanced comprehension capabilities.<n>However, mainstream MLLMs are solely supervised by the next-token prediction of textual tokens, neglecting critical vision-centric information.<n>We introduce VaCo, which optimize MLLM representations through Vision-Centric activation and Coordination.
  arXiv  Detail & Related papers  (2025-10-16T06:38:39Z)
• Visual CoT Makes VLMs Smarter but More Fragile [79.32638667101817]
  Chain-of-Thought (CoT) techniques have significantly enhanced reasoning in Vision-Language Models (VLMs)<n>Visual CoT integrates explicit visual edits, such as cropping or annotating regions of interest, into the reasoning process.<n>We present the first systematic evaluation of Visual CoT robustness under visual perturbations.
  arXiv  Detail & Related papers  (2025-09-28T10:19:59Z)
• VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models [82.05514464090172]
  Multimodal large language models (MLLMs) have significantly advanced the integration of visual and textual understanding.<n>However, their ability to generate code from multimodal inputs remains limited.<n>We introduce VisCodex, a unified framework that seamlessly merges vision and coding language models.
  arXiv  Detail & Related papers  (2025-08-13T17:00:44Z)
• MIMIC: Multimodal Inversion for Model Interpretation and Conceptualization [52.66401137323065]
  We propose a Multimodal Inversion for Model Interpretation and Conceptualization (MIMIC) framework to visualize the internal representations of Vision Language Models (VLMs)<n>MIMIC uses a joint VLM-based inversion and a feature alignment objective to account for VLM's autoregressive processing.<n>We quantitatively and qualitatively evaluate MIMIC by inverting visual concepts over a range of varying-length free-form VLM output texts.
  arXiv  Detail & Related papers  (2025-08-11T10:36:58Z)
• EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignment [39.870809905905325]
  We propose Empowering Multi-modal Mamba with Structural and Hierarchical Alignment (EMMA) to extract fine-grained visual information. Our model shows lower latency than other Mamba-based MLLMs and is nearly four times faster than transformer-based MLLMs of similar scale during inference.
  arXiv  Detail & Related papers  (2024-10-08T11:41:55Z)
• MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model [49.931663904599205]
  MaVEn is an innovative framework designed to enhance the capabilities of Multimodal Large Language Models (MLLMs) in multi-image reasoning. We show that MaVEn significantly enhances MLLMs' understanding in complex multi-image scenarios, while also improving performance in single-image contexts.
  arXiv  Detail & Related papers  (2024-08-22T11:57:16Z)
• Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback [38.708690624594794]
  Video and text multimodal alignment remains challenging, primarily due to the deficient volume and quality of multimodal instruction-tune data. We present a novel alignment strategy that employs multimodal AI system to oversee itself called Reinforcement Learning from AI Feedback (RLAIF) In specific, we propose context-aware reward modeling by providing detailed video descriptions as context during the generation of preference feedback.
  arXiv  Detail & Related papers  (2024-02-06T06:27:40Z)
• MMICT: Boosting Multi-Modal Fine-Tuning with In-Context Examples [63.78384552789171]
  This paper introduces Multi-Modal In-Context Tuning (MMICT), a novel multi-modal fine-tuning paradigm. We propose the Multi-Modal Hub (M-Hub), a unified module that captures various multi-modal features according to different inputs and objectives. Based on M-Hub, MMICT enables MM-LLMs to learn from in-context visual-guided textual features and subsequently generate outputs conditioned on the textual-guided visual features.
  arXiv  Detail & Related papers  (2023-12-11T13:11:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.