Weaving Context Across Images: Improving Vision-Language Models through Focus-Centric Visual Chains

• URL: http://arxiv.org/abs/2504.20199v1
• Date: Mon, 28 Apr 2025 19:02:18 GMT
• Title: Weaving Context Across Images: Improving Vision-Language Models through Focus-Centric Visual Chains
• Authors: Juntian Zhang, Chuanqi cheng, Yuhan Liu, Wei Liu, Jian Luan, Rui Yan,
• Abstract summary: Vision-language models (VLMs) achieve remarkable success in single-image tasks.<n>Real-world scenarios often involve intricate multi-image inputs, leading to a notable performance decline.<n>We propose Focus-Centric Visual Chain, a novel paradigm that enhances VLMs'perception, comprehension, and reasoning abilities in multi-image scenarios.
• Score: 31.828341309787042
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) achieve remarkable success in single-image tasks. However, real-world scenarios often involve intricate multi-image inputs, leading to a notable performance decline as models struggle to disentangle critical information scattered across complex visual features. In this work, we propose Focus-Centric Visual Chain, a novel paradigm that enhances VLMs'perception, comprehension, and reasoning abilities in multi-image scenarios. To facilitate this paradigm, we propose Focus-Centric Data Synthesis, a scalable bottom-up approach for synthesizing high-quality data with elaborate reasoning paths. Through this approach, We construct VISC-150K, a large-scale dataset with reasoning data in the form of Focus-Centric Visual Chain, specifically designed for multi-image tasks. Experimental results on seven multi-image benchmarks demonstrate that our method achieves average performance gains of 3.16% and 2.24% across two distinct model architectures, without compromising the general vision-language capabilities. our study represents a significant step toward more robust and capable vision-language systems that can handle complex visual scenarios.

Related papers

• Learning 1D Causal Visual Representation with De-focus Attention Networks [108.72931590504406]
  This paper explores the feasibility of representing images using 1D causal modeling. We propose De-focus Attention Networks, which employ learnable bandpass filters to create varied attention patterns.
  arXiv  Detail & Related papers  (2024-06-06T17:59:56Z)
• Enhancing Large Vision Language Models with Self-Training on Image Comprehension [131.14381425260706]
  We introduce Self-Training on Image (STIC), which emphasizes a self-training approach specifically for image comprehension. First, the model self-constructs a preference for image descriptions using unlabeled images. To further self-improve reasoning on the extracted visual information, we let the model reuse a small portion of existing instruction-tuning data.
  arXiv  Detail & Related papers  (2024-05-30T05:53:49Z)
• Chain-of-Spot: Interactive Reasoning Improves Large Vision-Language Models [81.71651422951074]
  Chain-of-Spot (CoS) method is a novel approach that enhances feature extraction by focusing on key regions of interest. This technique allows LVLMs to access more detailed visual information without altering the original image resolution. Our empirical findings demonstrate a significant improvement in LVLMs' ability to understand and reason about visual content.
  arXiv  Detail & Related papers  (2024-03-19T17:59:52Z)
• Multi-modal Auto-regressive Modeling via Visual Words [96.25078866446053]
  We propose the concept of visual tokens, which maps the visual features to probability distributions over Large Multi-modal Models' vocabulary. We further explore the distribution of visual features in the semantic space within LMM and the possibility of using text embeddings to represent visual information.
  arXiv  Detail & Related papers  (2024-03-12T14:58:52Z)
• PaLM2-VAdapter: Progressively Aligned Language Model Makes a Strong Vision-language Adapter [21.45490901191175]
  PaLM2-VAdapter employs a progressively aligned language model as the vision-language adapter. Our method achieves these advancements with 3070% fewer parameters than the state-of-the-art large vision-language models.
  arXiv  Detail & Related papers  (2024-02-16T18:54:47Z)
• Question Aware Vision Transformer for Multimodal Reasoning [14.188369270753347]
  We introduce QA-ViT, a Question Aware Vision Transformer approach for multimodal reasoning. It embeds question awareness directly within the vision encoder. This integration results in dynamic visual features focusing on relevant image aspects to the posed question.
  arXiv  Detail & Related papers  (2024-02-08T08:03:39Z)
• UniDiff: Advancing Vision-Language Models with Generative and Discriminative Learning [86.91893533388628]
  This paper presents UniDiff, a unified multi-modal model that integrates image-text contrastive learning (ITC), text-conditioned image synthesis learning (IS), and reciprocal semantic consistency modeling (RSC) UniDiff demonstrates versatility in both multi-modal understanding and generative tasks.
  arXiv  Detail & Related papers  (2023-06-01T15:39:38Z)
• mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections [104.14624185375897]
  mPLUG is a new vision-language foundation model for both cross-modal understanding and generation. It achieves state-of-the-art results on a wide range of vision-language downstream tasks, such as image captioning, image-text retrieval, visual grounding and visual question answering.
  arXiv  Detail & Related papers  (2022-05-24T11:52:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.