MQAdapter: Multi-Modal Quantum Adapter for Coarse-to-Fine VLM Fine-tuning

• URL: http://arxiv.org/abs/2607.12418v1
• Date: Tue, 14 Jul 2026 06:46:24 GMT
• Title: MQAdapter: Multi-Modal Quantum Adapter for Coarse-to-Fine VLM Fine-tuning
• Abstract summary: We propose a novel coarse-to-fine VLM fine-tuning approach for few-shot learning that leverages quantum computation.<n>MQAdapter first retrieves the Top-K category candidates most similar to the input image and uses them as semantic anchors.<n>It then employs a cross-modal quantum learning mechanism to refine visual features under the guidance of these anchors.
• Score: 14.855850083028988
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large-scale Vision-Language Models have demonstrated impressive transfer learning capabilities across a wide range of tasks. For few-shot classification, we observe that VLMs exhibit a notable ability to filter candidate categories and thus achieve high Top-K accuracy. However, they often struggle with fine-grained discrimination among visually similar categories, resulting in unsatisfactory Top-1 performance, as shown in Figure 1. Existing studies on VLM adapters generally focus on global alignment between visual and textual representations in the feature space, but fail to exploit semantically similar categories to refine fine-grained visual representations. Based on these observations, we propose a novel coarse-to-fine VLM fine-tuning approach for few-shot learning that leverages quantum computation, termed the Multi-Modal Quantum Adapter (MQAdapter). Specifically, MQAdapter first retrieves the Top-K category candidates most similar to the input image and uses them as semantic anchors. It then employs a cross-modal quantum learning mechanism to refine visual features under the guidance of these anchors. The core of this mechanism is the encoding of visual and textual features into quantum states. By leveraging quantum entanglement and superposition in a high-dimensional Hilbert space, MQAdapter effectively models higher-order cross-modal interactions, producing more discriminative representations than traditional Euclidean adapters. MQAdapter is parameter-efficient and can be integrated with various existing fine-tuning algorithms to achieve further performance gains. Evaluations on 15 datasets demonstrate the effectiveness of MQAdapter while requiring fewer trainable parameters.

Related papers

• A Contrastive Learning Framework Empowered by Attention-based Feature Adaptation for Street-View Image Classification [0.7746379804154433]
  Street-view image attribute classification is a vital downstream task of image classification, enabling applications such as autonomous driving, urban analytics, and high-definition map construction.<n>We propose CLIP-MHAdapter, a variant of the current lightweight CLIP adaptation paradigm that appends a bottleneck equipped with multi-head self-attention to model inter-patch dependencies.<n>With approximately 1.4 million trainable parameters, CLIP-MHAdapter achieves superior or competitive accuracy across eight attribute classification tasks on the Global StreetScapes dataset, attaining new state-of-the-art results while maintaining low computational cost.
  arXiv  Detail & Related papers  (2026-02-18T16:41:32Z)
• QuIC: A Quantum-Inspired Interaction Classifier for Revitalizing Shallow CNNs in Fine-Grained Recognition [1.2747974581480312]
  QuIC is a lightweight, plug-and-play module that supports stable, single-stage end-to-end training without exploding feature dimensions.<n>It boosts the Top-1 accuracy of VGG16 by nearly 20% and outperforms state-of-the-art attention mechanisms (SE-Block) on ResNet18.
  arXiv  Detail & Related papers  (2026-01-05T15:09:18Z)
• Q-Adapter: Visual Query Adapter for Extracting Textually-related Features in Video Captioning [5.762008844570409]
  We propose a lightweight visual adapter module designed to enhance Multimodal Large Language Models (MLLMs)<n>Q-Adapter introduces learnable query tokens and a gating layer into Vision, enabling effective extraction of sparse, caption-relevant features without relying on external supervision.<n>We evaluate Q-Adapter on two well-known video captioning datasets, MSR-VTT and MSVD, where it achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-10-11T04:58:21Z)
• Demystifying the Visual Quality Paradox in Multimodal Large Language Models [49.154146792279946]
  Recent Multimodal Large Language Models (MLLMs) excel on benchmark vision-language tasks, yet little is known about how input visual quality shapes their responses.<n>We conduct the first systematic study spanning leading MLLMs and a suite of vision-language benchmarks.<n>We uncover a visual-quality paradox: model, task, and even individual-instance performance can improve when images deviate from human-perceived fidelity.
  arXiv  Detail & Related papers  (2025-06-18T17:14:07Z)
• An Efficient Quantum Classifier Based on Hamiltonian Representations [50.467930253994155]
  Quantum machine learning (QML) is a discipline that seeks to transfer the advantages of quantum computing to data-driven tasks.<n>We propose an efficient approach that circumvents the costs associated with data encoding by mapping inputs to a finite set of Pauli strings.<n>We evaluate our approach on text and image classification tasks, against well-established classical and quantum models.
  arXiv  Detail & Related papers  (2025-04-13T11:49:53Z)
• Q-Adapt: Adapting LMM for Visual Quality Assessment with Progressive Instruction Tuning [49.07442840323135]
  We propose a new paradigm for perception-oriented instruction tuning, i.e., Q-Adapt.<n>Our proposed Q-Adapt can achieve a lightweight visual quality evaluator, demonstrating comparable performance.
  arXiv  Detail & Related papers  (2025-04-02T12:02:57Z)
• p-Laplacian Adaptation for Generative Pre-trained Vision-Language Models [10.713680139939354]
  Vision-Language models (VLMs) pre-trained on large corpora have demonstrated notable success across a range of downstream tasks. PETL has garnered attention as a viable alternative to full fine-tuning. We propose a new adapter architecture, $p$-adapter, which employs $p$-Laplacian message passing in Graph Neural Networks (GNNs)
  arXiv  Detail & Related papers  (2023-12-17T05:30:35Z)
• Meta-Adapter: An Online Few-shot Learner for Vision-Language Model [64.21017759533474]
  Contrastive vision-language pre-training, known as CLIP, demonstrates remarkable potential in perceiving open-world visual concepts. Few-shot learning methods based on CLIP typically require offline fine-tuning of the parameters on few-shot samples. We propose the Meta-Adapter, a lightweight residual-style adapter, to refine the CLIP features guided by the few-shot samples in an online manner.
  arXiv  Detail & Related papers  (2023-11-07T07:27:16Z)
• Approximately Equivariant Quantum Neural Network for $p4m$ Group Symmetries in Images [30.01160824817612]
  This work proposes equivariant Quantum Convolutional Neural Networks (EquivQCNNs) for image classification under planar $p4m$ symmetry. We present the results tested in different use cases, such as phase detection of the 2D Ising model and classification of the extended MNIST dataset.
  arXiv  Detail & Related papers  (2023-10-03T18:01:02Z)
• UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling [49.134517040512414]
  This paper proposes UniAdapter, which unifies unimodal and multimodal adapters for parameter-efficient cross-modal adaptation on vision-language models. Experiments show that UniAdapter not only outperforms the state-of-the-arts, but even beats the full fine-tuning strategy.
  arXiv  Detail & Related papers  (2023-02-13T18:59:10Z)
• CLIP-Adapter: Better Vision-Language Models with Feature Adapters [84.88106370842883]
  We show that there is an alternative path to achieve better vision-language models other than prompt tuning.<n>CLIP-Adapter adopts an additional bottleneck layer to learn new features and performs residual-style feature blending.<n> Experiments and extensive ablation studies on various visual classification tasks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2021-10-09T11:39:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.