Picking the Cream of the Crop: Visual-Centric Data Selection with Collaborative Agents

• URL: http://arxiv.org/abs/2502.19917v1
• Date: Thu, 27 Feb 2025 09:37:30 GMT
• Title: Picking the Cream of the Crop: Visual-Centric Data Selection with Collaborative Agents
• Authors: Zhenyu Liu, Yunxin Li, Baotian Hu, Wenhan Luo, Yaowei Wang, Min Zhang,
• Abstract summary: We propose a textbfVisual-Centric textbfSelection approach via textbfAgents Collaboration (ViSA)<n>Our approach consists of 1) an image information quantification method via visual agents collaboration to select images with rich visual information, and 2) a visual-centric instruction quality assessment method to select high-quality instruction data related to high-quality images.
• Score: 62.616106562146776
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To improve Multimodal Large Language Models' (MLLMs) ability to process images and complex instructions, researchers predominantly curate large-scale visual instruction tuning datasets, which are either sourced from existing vision tasks or synthetically generated using LLMs and image descriptions. However, they often suffer from critical flaws, including misaligned instruction-image pairs and low-quality images. Such issues hinder training efficiency and limit performance improvements, as models waste resources on noisy or irrelevant data with minimal benefit to overall capability. To address this issue, we propose a \textbf{Vi}sual-Centric \textbf{S}election approach via \textbf{A}gents Collaboration (ViSA), which centers on image quality assessment and image-instruction relevance evaluation. Specifically, our approach consists of 1) an image information quantification method via visual agents collaboration to select images with rich visual information, and 2) a visual-centric instruction quality assessment method to select high-quality instruction data related to high-quality images. Finally, we reorganize 80K instruction data from large open-source datasets. Extensive experiments demonstrate that ViSA outperforms or is comparable to current state-of-the-art models on seven benchmarks, using only 2.5\% of the original data, highlighting the efficiency of our data selection approach. Moreover, we conduct ablation studies to validate the effectiveness of each component of our method. The code is available at https://github.com/HITsz-TMG/ViSA.

Related papers

• Weaving Context Across Images: Improving Vision-Language Models through Focus-Centric Visual Chains [31.828341309787042]
  Vision-language models (VLMs) achieve remarkable success in single-image tasks. Real-world scenarios often involve intricate multi-image inputs, leading to a notable performance decline. We propose Focus-Centric Visual Chain, a novel paradigm that enhances VLMs'perception, comprehension, and reasoning abilities in multi-image scenarios.
  arXiv  Detail & Related papers  (2025-04-28T19:02:18Z)
• SMIR: Efficient Synthetic Data Pipeline To Improve Multi-Image Reasoning [26.986638043619397]
  We introduce SMiR, a synthetic data-generation pipeline for multi-image reasoning.<n>We produce 160K synthetic training samples, offering a cost-effective alternative to closed-source solutions.<n>We also present SMiR-Bench, a multi-image reasoning benchmark comprising 200 diverse examples.
  arXiv  Detail & Related papers  (2025-01-07T10:21:21Z)
• Large-Scale Data-Free Knowledge Distillation for ImageNet via Multi-Resolution Data Generation [53.95204595640208]
  Data-Free Knowledge Distillation (DFKD) is an advanced technique that enables knowledge transfer from a teacher model to a student model without relying on original training data. Previous approaches have generated synthetic images at high resolutions without leveraging information from real images. MUSE generates images at lower resolutions while using Class Activation Maps (CAMs) to ensure that the generated images retain critical, class-specific features.
  arXiv  Detail & Related papers  (2024-11-26T02:23:31Z)
• Enhancing Large Vision Language Models with Self-Training on Image Comprehension [131.14381425260706]
  We introduce Self-Training on Image (STIC), which emphasizes a self-training approach specifically for image comprehension. First, the model self-constructs a preference for image descriptions using unlabeled images. To further self-improve reasoning on the extracted visual information, we let the model reuse a small portion of existing instruction-tuning data.
  arXiv  Detail & Related papers  (2024-05-30T05:53:49Z)
• Multi-Modal Prompt Learning on Blind Image Quality Assessment [65.0676908930946]
  Image Quality Assessment (IQA) models benefit significantly from semantic information, which allows them to treat different types of objects distinctly. Traditional methods, hindered by a lack of sufficiently annotated data, have employed the CLIP image-text pretraining model as their backbone to gain semantic awareness. Recent approaches have attempted to address this mismatch using prompt technology, but these solutions have shortcomings. This paper introduces an innovative multi-modal prompt-based methodology for IQA.
  arXiv  Detail & Related papers  (2024-04-23T11:45:32Z)
• Less is More: High-value Data Selection for Visual Instruction Tuning [127.38740043393527]
  We propose a high-value data selection approach TIVE, to eliminate redundancy within the visual instruction data and reduce the training cost. Our approach using only about 15% data can achieve comparable average performance to the full-data fine-tuned model across eight benchmarks.
  arXiv  Detail & Related papers  (2024-03-14T16:47:25Z)
• Composed Image Retrieval using Contrastive Learning and Task-oriented CLIP-based Features [32.138956674478116]
  Given a query composed of a reference image and a relative caption, the Composed Image Retrieval goal is to retrieve images visually similar to the reference one. We use features from the OpenAI CLIP model to tackle the considered task. We train a Combiner network that learns to combine the image-text features integrating the bimodal information.
  arXiv  Detail & Related papers  (2023-08-22T15:03:16Z)
• Semi-Supervised Image Captioning by Adversarially Propagating Labeled Data [95.0476489266988]
  We present a novel data-efficient semi-supervised framework to improve the generalization of image captioning models. Our proposed method trains a captioner to learn from a paired data and to progressively associate unpaired data. Our extensive and comprehensive empirical results both on (1) image-based and (2) dense region-based captioning datasets followed by comprehensive analysis on the scarcely-paired dataset.
  arXiv  Detail & Related papers  (2023-01-26T15:25:43Z)
• MSTRIQ: No Reference Image Quality Assessment Based on Swin Transformer with Multi-Stage Fusion [8.338999282303755]
  We propose a novel algorithm based on the Swin Transformer. It aggregates information from both local and global features to better predict the quality. It ranks 2nd in the no-reference track of NTIRE 2022 Perceptual Image Quality Assessment Challenge.
  arXiv  Detail & Related papers  (2022-05-20T11:34:35Z)
• Mix-up Self-Supervised Learning for Contrast-agnostic Applications [33.807005669824136]
  We present the first mix-up self-supervised learning framework for contrast-agnostic applications. We address the low variance across images based on cross-domain mix-up and build the pretext task based on image reconstruction and transparency prediction.
  arXiv  Detail & Related papers  (2022-04-02T16:58:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.