VERITAS: Leveraging Vision Priors and Expert Fusion to Improve Multimodal Data

• URL: http://arxiv.org/abs/2510.15317v1
• Date: Fri, 17 Oct 2025 05:13:50 GMT
• Title: VERITAS: Leveraging Vision Priors and Expert Fusion to Improve Multimodal Data
• Authors: Tingqiao Xu, Ziru Zeng, Jiayu Chen,
• Abstract summary: VERITAS is a pipeline that integrates vision priors and multiple state-of-the-art LMMs to enhance SFT data quality.<n>Three LMMs evaluate the original answers, providing critique rationales and scores that are statistically fused into a high-confidence consensus score.<n>Our critic model exhibits enhanced capability comparable to state-of-the-art LMMs while being significantly more efficient.
• Score: 3.638465758795032
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The quality of supervised fine-tuning (SFT) data is crucial for the performance of large multimodal models (LMMs), yet current data enhancement methods often suffer from factual errors and hallucinations due to inadequate visual perception. To address this challenge, we propose VERITAS, a pipeline that systematically integrates vision priors and multiple state-of-the-art LMMs with statistical methods to enhance SFT data quality. VERITAS leverages visual recognition models (RAM++) and OCR systems (PP-OCRv4) to extract structured vision priors, which are combined with images, questions, and answers. Three LMMs (GPT-4o, Gemini-2.5-Pro, Doubao-1.5-pro) evaluate the original answers, providing critique rationales and scores that are statistically fused into a high-confidence consensus score serving as ground truth. Using this consensus, we train a lightweight critic model via Group Relative Policy Optimization (GRPO), enhancing reasoning capabilities efficiently. Each LMM then refines the original answers based on the critiques, generating new candidate answers; we select the highest-scoring one as the final refined answer. Experiments across six multimodal benchmarks demonstrate that models fine-tuned with data processed by VERITAS consistently outperform those using raw data, particularly in text-rich and fine-grained reasoning tasks. Our critic model exhibits enhanced capability comparable to state-of-the-art LMMs while being significantly more efficient. We release our pipeline, datasets, and model checkpoints to advance research in multimodal data optimization.

Related papers

• Long Grounded Thoughts: Distilling Compositional Visual Reasoning Chains at Scale [70.23466957404891]
  We introduce a new reasoning data generation framework spanning diverse skills and levels of complexity with over 1M high-quality synthetic vision-centric questions.<n>We show that finetuning Qwen2.5-VL-7B on our data outperforms all open-data baselines across all evaluated vision-centric benchmarks.
  arXiv  Detail & Related papers  (2025-11-07T20:50:54Z)
• On Evaluating the Adversarial Robustness of Foundation Models for Multimodal Entity Linking [11.268639885321884]
  We conduct the first comprehensive evaluation of the robustness of mainstream MEL models under different adversarial attack scenarios.<n>Experiments on five datasets demonstrate that LLM-RetLink improves the accuracy of MEL by 0.4%-35.7%.<n>This research highlights a previously unexplored facet of MEL robustness, constructs and releases the first MEL adversarial example dataset.
  arXiv  Detail & Related papers  (2025-08-21T11:57:37Z)
• High-Resolution Visual Reasoning via Multi-Turn Grounding-Based Reinforcement Learning [43.8114307203968]
  State-of-the-art large multi-modal models (LMMs) face challenges when processing high-resolution images.<n>In this paper, we propose Multi-turn Grounding-based Policy Optimization (MGPO)<n>MGPO enables LMMs to iteratively focus on key visual regions by automatically cropping sub-images.
  arXiv  Detail & Related papers  (2025-07-08T12:05:05Z)
• MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency [63.23935582919081]
  Chain-of-Thought (CoT) has significantly enhanced the reasoning capabilities of Large Language Models (LLMs)<n>We introduce MME-CoT, a specialized benchmark evaluating the CoT reasoning performance of LMMs.<n>We conduct an in-depth analysis of state-of-the-art LMMs, uncovering several key insights.
  arXiv  Detail & Related papers  (2025-02-13T18:59:46Z)
• Multimodal Preference Data Synthetic Alignment with Reward Model [23.978820500281213]
  We propose a new framework in generating synthetic data using a reward model as a proxy of human preference for effective multimodal alignment with DPO training.<n>Experiment results indicate that integrating selected synthetic data, such as from generative and rewards models can effectively reduce reliance on human-annotated data.
  arXiv  Detail & Related papers  (2024-12-23T09:29:40Z)
• EACO: Enhancing Alignment in Multimodal LLMs via Critical Observation [58.546205554954454]
  We propose Enhancing Alignment in MLLMs via Critical Observation (EACO)<n>EACO aligns MLLMs by self-generated preference data using only 5k images economically.<n>EACO reduces the overall hallucinations by 65.6% on HallusionBench and improves the reasoning ability by 21.8% on MME-Cognition.
  arXiv  Detail & Related papers  (2024-12-06T09:59:47Z)
• Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuning [71.2981957820888]
  We propose a novel Star-Agents framework, which automates the enhancement of data quality across datasets. The framework initially generates diverse instruction data with multiple LLM agents through a bespoke sampling method. The generated data undergo a rigorous evaluation using a dual-model method that assesses both difficulty and quality.
  arXiv  Detail & Related papers  (2024-11-21T02:30:53Z)
• Training Language Models to Critique With Multi-agent Feedback [102.42751835338233]
  MultiCritique pipeline improves critique ability of LLMs by utilizing multi-agent feedback. pipeline aggregates high-quality critiques from multiple agents instead of a single model. Our fine-tuned 7B model significantly surpasses other advanced 7B-13B open-source models.
  arXiv  Detail & Related papers  (2024-10-20T04:57:45Z)
• The Inter-Intra Modal Measure: A Predictive Lens on Fine-Tuning Outcomes in Vision-Language Models [6.7181844004432385]
  We introduce the Inter-Intra Modal Measure (IIMM) - a predictive metric that quantifies the relationship between intra-modal image embedding similarity and inter-modal misalignment.<n>Compared to existing transferability measures, the IIMM demonstrates significantly stronger predictive power for accuracy changes post fine-tuning in dual-encoder models.<n>We provide a theoretical bound, proving that changes in IIMM are limited by the Wasserstein distance between pre- and post-fine-tuning embedding.
  arXiv  Detail & Related papers  (2024-07-22T15:35:09Z)
• ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models [45.040292339670096]
  Large vision-language models (LVLMs) have shown premise in a broad range of vision-language tasks with their strong reasoning and generalization capabilities. This study aims to bridge the performance gap between traditional-scale LVLMs and resource-friendly lite versions by adopting high-quality training data.
  arXiv  Detail & Related papers  (2024-02-18T19:26:49Z)
• Silkie: Preference Distillation for Large Visual Language Models [56.10697821410489]
  This paper explores preference distillation for large vision language models (LVLMs) We first build a vision-language feedback dataset utilizing AI annotation. We adopt GPT-4V to assess the generated outputs regarding helpfulness, visual faithfulness, and ethical considerations. The resulting model Silkie, achieves 6.9% and 9.5% relative improvement on the MME benchmark regarding the perception and cognition capabilities.
  arXiv  Detail & Related papers  (2023-12-17T09:44:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.