When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs

• URL: http://arxiv.org/abs/2509.16633v1
• Date: Sat, 20 Sep 2025 11:12:23 GMT
• Title: When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs
• Authors: Abhirama Subramanyam Penamakuri, Navlika Singh, Piyush Arora, Anand Mishra,
• Abstract summary: Large Vision-Language Models (L-VLMs) have demonstrated remarkable performance in various vision and language tasks.<n>Small Vision-Language Models (S-VLMs) offer efficiency but suffer from a significant performance gap compared to their larger counterparts.<n>We introduce the Model Parity Aligner (MPA), a novel framework designed to systematically improve S-VLMs.
• Score: 4.296395082987112
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large Vision-Language Models (L-VLMs) have demonstrated remarkable performance in various vision and language tasks, including visual question answering (VQA). However, their high computational cost makes them impractical for resource-constrained settings and inference-heavy applications. In contrast, Small Vision-Language Models (S-VLMs) offer efficiency but suffer from a significant performance gap compared to their larger counterparts. In this work, we introduce the Model Parity Aligner (MPA), a novel framework designed to systematically improve S-VLMs by leveraging unlabeled images and effective knowledge transfer from L-VLMs. Instead of traditional knowledge distillation methods that rely on labeled training data, MPA employs a strategic parity-based approach that precisely identifies the knowledge disparities between S-VLMs and L-VLMs, and optimizes training by targeting only these disparities. We conduct extensive experiments on four diverse VQA benchmarks, namely TextVQA, ST-VQA, ChartQA, and OKVQA, each of which requires specialized reasoning capabilities such as text recognition, chart interpretation, and commonsense and factual understanding. Our results demonstrate that MPA consistently enhances the performance of S-VLMs on all benchmarks, reducing the performance gap while maintaining computational efficiency. We make our code publicly available.

Related papers

• VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models [43.09726338623949]
  Vision-Language-Action (VLA) models integrate pretrained large Vision-Language Models (VLM) into their policy backbone.<n>This paper revisits a fundamental yet seldom systematically studied question: how VLM choice and competence translate to downstream VLA policies performance.<n>We introduce VLM4VLA, a minimal adaptation pipeline that converts general-purpose VLMs into VLA policies using only a small set of new learnable parameters.
  arXiv  Detail & Related papers  (2026-01-06T09:58:24Z)
• HoneyBee: Data Recipes for Vision-Language Reasoners [90.83745691506329]
  We introduce several data curation approaches and study their impacts on vision-language models (VLMs)<n>We analyze the effects of context (image and question pair) sources, implement targeted data interventions, and explore scaling up images, questions, and chain-of-thought (CoT) solutions.<n>Motivated by these insights, we introduce HoneyBee, a large-scale, high-quality CoT reasoning dataset with 2.5M examples consisting 350K image-question pairs.
  arXiv  Detail & Related papers  (2025-10-14T07:23:44Z)
• Training Vision-Language Process Reward Models for Test-Time Scaling in Multimodal Reasoning: Key Insights and Lessons Learned [29.44294456857936]
  Process Reward Models (PRMs) improve the reliability of reasoning in large language models.<n>Existing Vision-Language PRMs rely on Monte Carlo Tree Search (MCTS) for data construction.<n>We introduce a hybrid data framework that combines MCTS with judgments from a strong VLM, producing more accurate step-level labels.
  arXiv  Detail & Related papers  (2025-09-27T10:56:58Z)
• Enhancing Scientific Visual Question Answering via Vision-Caption aware Supervised Fine-Tuning [26.89241254462218]
  We introduce Vision-Caption aware Supervised FineTuning (VCASFT)<n>VCASFT is a learning paradigm designed to enhance the performance of smaller Vision Language Models (VLMs)<n>We benchmark it on ScienceQA, which consists of questions across diverse languages, subjects, and fields.<n>To further demonstrate the effectiveness of this technique on lowresource languages, we developed HiSciVQA, a dataset comprising 2,245 high-quality, hand-annotated Hindi multimodal Q&A pairs.
  arXiv  Detail & Related papers  (2025-09-20T11:07:36Z)
• VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning [95.89543460132413]
  Vision-language models (VLMs) have improved performance by increasing the number of visual tokens.<n>However, most real-world scenarios do not require such an extensive number of visual tokens.<n>We present a new paradigm for visual token compression, namely, VisionThink.
  arXiv  Detail & Related papers  (2025-07-17T17:59:55Z)
• Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models [19.361686225381447]
  Vision-language models (VLMs) are widely assumed to exhibit in-context learning (ICL)<n>We propose a new MM-ICL with Reasoning pipeline that augments each demonstration with a generated rationale alongside the answer.
  arXiv  Detail & Related papers  (2025-06-09T16:55:32Z)
• Caption This, Reason That: VLMs Caught in the Middle [3.4820139118440676]
  Vision-Language Models (VLMs) have shown remarkable progress in visual understanding in recent years.<n>They still lag behind human capabilities in specific visual tasks such as counting or relational reasoning.<n>We analyze VLM performance along core cognitive axes: Perception, Attention, and Memory.
  arXiv  Detail & Related papers  (2025-05-24T14:25:48Z)
• Elevating Visual Perception in Multimodal LLMs with Auxiliary Embedding Distillation [109.5893580175657]
  In recent times, the standard practice for developing MLLMs is to feed features from vision encoder(s) into the LLM and train with natural language supervision.<n>This approach often causes models to lean towards language comprehension and undermine the rich visual perception signals present in the data.<n>We propose VisPer-LM, the first approach that infuses visual perception knowledge from expert vision encoders into the LLM's hidden representations.
  arXiv  Detail & Related papers  (2024-12-12T18:55:18Z)
• Scaling Inference-Time Search with Vision Value Model for Improved Visual Comprehension [95.63899307791665]
  Vision Value Model (VisVM) can guide VLM inference-time search to generate responses with better visual comprehension.<n>In this paper, we present VisVM that can guide VLM inference-time search to generate responses with better visual comprehension.
  arXiv  Detail & Related papers  (2024-12-04T20:35:07Z)
• Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Features [79.45405711339322]
  Generative Large Multimodal Models (LMMs) excel at a wide variety of vision-language (VL) tasks.<n>Despite strong performance, LMMs' generative outputs are not specialized for vision-language classification tasks.<n>We propose an approach that leverages multimodal feature extraction from the LMM's latent space.
  arXiv  Detail & Related papers  (2024-11-28T18:55:41Z)
• AutoBench-V: Can Large Vision-Language Models Benchmark Themselves? [65.92331309449015]
  We introduce AutoBench-V, an automated framework for serving evaluation on demand, i.e., benchmarking LVLMs based on specific aspects of model capability.<n>Through an extensive evaluation of nine popular LVLMs across five demanded user inputs, the framework shows effectiveness and reliability.
  arXiv  Detail & Related papers  (2024-10-28T17:55:08Z)
• Large Vision-Language Models as Emotion Recognizers in Context Awareness [14.85890824622433]
  Context-aware emotion recognition (CAER) is a complex and significant task that requires perceiving emotions from various contextual cues. Previous approaches primarily focus on designing sophisticated architectures to extract emotional cues from images. This paper systematically explore the potential of leveraging Large Vision-Language Models (LVLMs) to empower the CAER task.
  arXiv  Detail & Related papers  (2024-07-16T01:28:06Z)
• Multi-Modal Prompt Learning on Blind Image Quality Assessment [65.0676908930946]
  Image Quality Assessment (IQA) models benefit significantly from semantic information, which allows them to treat different types of objects distinctly. Traditional methods, hindered by a lack of sufficiently annotated data, have employed the CLIP image-text pretraining model as their backbone to gain semantic awareness. Recent approaches have attempted to address this mismatch using prompt technology, but these solutions have shortcomings. This paper introduces an innovative multi-modal prompt-based methodology for IQA.
  arXiv  Detail & Related papers  (2024-04-23T11:45:32Z)
• Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models [57.95366341738857]
  In-depth analyses show that instruction-tuned LVLMs exhibit modality gap, showing discrepancy when given textual and visual inputs that correspond to the same concept.<n>We propose a multiple attribute-centric evaluation benchmark, Finer, to evaluate LVLMs' fine-grained visual comprehension ability and provide significantly improved explainability.
  arXiv  Detail & Related papers  (2024-02-26T05:43:51Z)
• VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Models [21.549122658275383]
  Recent advances in vision-language pre-training have demonstrated impressive performance in a range of vision-language tasks. We introduce the Vision-Language Understanding Evaluation benchmark, a multi-task multi-dimension benchmark for evaluating the generalization capabilities and the efficiency-performance trade-off.
  arXiv  Detail & Related papers  (2022-05-30T16:52:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.