Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment

• URL: http://arxiv.org/abs/2405.17871v2
• Date: Tue, 05 Nov 2024 02:26:51 GMT
• Title: Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment
• Authors: Xin Xiao, Bohong Wu, Jiacong Wang, Chunyuan Li, Xun Zhou, Haoyuan Guo,
• Abstract summary: Existing image-text modality alignment in Vision Language Models (VLMs) treats each text token equally in an autoregressive manner. We advocate for assigning distinct contributions for each text token based on its visual correlation. We introduce Contrastive ALignment (CAL), a simple yet effective re-weighting strategy that prioritizes training visually correlated tokens.
• Score: 40.63340635482609
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing image-text modality alignment in Vision Language Models (VLMs) treats each text token equally in an autoregressive manner. Despite being simple and effective, this method results in sub-optimal cross-modal alignment by over-emphasizing the text tokens that are less correlated with or even contradictory with the input images. In this paper, we advocate for assigning distinct contributions for each text token based on its visual correlation. Specifically, we present by contrasting image inputs, the difference in prediction logits on each text token provides strong guidance of visual correlation. We therefore introduce Contrastive ALignment (CAL), a simple yet effective re-weighting strategy that prioritizes training visually correlated tokens. Our experimental results demonstrate that CAL consistently improves different types of VLMs across different resolutions and model sizes on various benchmark datasets. Importantly, our method incurs minimal additional computational overhead, rendering it highly efficient compared to alternative data scaling strategies. Codes are available at https://github.com/foundation-multimodal-models/CAL.

Related papers

• Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations [33.11867433769496]
  This paper presents a framework that attempts to unify visual understanding and generation within a shared semantic representation.<n>At its core is the Text-Aligned Tokenizer (TA-Tok), which converts images into discrete tokens using a text-aligned codebook projected from a large language model's (LLM) vocabulary.<n> Experiments across benchmarks show that Tar matches or surpasses existing multimodal LLM methods, achieving faster convergence and greater training efficiency.
  arXiv  Detail & Related papers  (2025-06-23T17:59:14Z)
• Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs [43.94713826224876]
  Vision-Language models (VLMs) show impressive abilities to answer questions on visual inputs, yet demonstrate higher accuracies when performing an analogous task on text.<n>We investigate this accuracy gap by identifying and comparing the textitcircuits in different modalities.<n>To overcome this, we patch the representations of visual data tokens from later layers back into earlier layers.
  arXiv  Detail & Related papers  (2025-06-10T17:59:21Z)
• FocusDiff: Advancing Fine-Grained Text-Image Alignment for Autoregressive Visual Generation through RL [78.59912944698992]
  We propose FocusDiff to enhance fine-grained text-image semantic alignment.<n>We construct a new dataset of paired texts and images with similar overall expressions but distinct local semantics.<n>Our approach achieves state-of-the-art performance on existing text-to-image benchmarks and significantly outperforms prior methods on PairComp.
  arXiv  Detail & Related papers  (2025-06-05T18:36:33Z)
• Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Training [78.60953331455565]
  PRIOR is a vision-language pre-training approach that prioritizes image-related tokens through differential weighting in the NTP loss.<n>We observe 19% and 8% average relative improvement, respectively, on several vision-language benchmarks compared to NTP.
  arXiv  Detail & Related papers  (2025-05-13T21:27:52Z)
• Aligning Text to Image in Diffusion Models is Easier Than You Think [47.623236425067326]
  We introduce a lightweight contrastive fine tuning strategy called SoftREPA that uses soft text tokens. Our method explicitly increases the mutual information between text and image representations, leading to enhanced semantic consistency.
  arXiv  Detail & Related papers  (2025-03-11T10:14:22Z)
• Rethinking Homogeneity of Vision and Text Tokens in Large Vision-and-Language Models [29.611769371733672]
  We propose De Attention (D-Attn), a novel method that processes visual and textual embeddings differently. D-Attn diagonalizes visual-to-visual self-attention, reducing computation from $mathcalO(|V|2)$ to $mathcalO(|V|)$ for $|V|$ visual embeddings without compromising performance.
  arXiv  Detail & Related papers  (2025-02-04T00:46:11Z)
• Balancing Performance and Efficiency: A Multimodal Large Language Model Pruning Method based Image Text Interaction [6.467840081978855]
  multimodal large language models (MM-LLMs) have achieved great success in many multimodal tasks, but their high computational costs limit their further promotion and application. We studied the visual tokens of MM-LLMs and designed a dynamic pruning algorithm to address this issue. Our proposed method can achieve performance that competes with the original performance when using an average of 22% of the original token quantity.
  arXiv  Detail & Related papers  (2024-09-02T10:49:10Z)
• Token-level Correlation-guided Compression for Efficient Multimodal Document Understanding [54.532578213126065]
  Most document understanding methods preserve all tokens within sub-images and treat them equally. This neglects their different informativeness and leads to a significant increase in the number of image tokens. We propose Token-level Correlation-guided Compression, a parameter-free and plug-and-play methodology to optimize token processing.
  arXiv  Detail & Related papers  (2024-07-19T16:11:15Z)
• Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models [21.17975741743583]
  It has recently been discovered that using a pre-trained vision-language model (VLM), e.g., CLIP, to align a whole query image with several finer text descriptions can significantly enhance zero-shot performance. In this paper, we empirically find that the finer descriptions tend to align more effectively with local areas of the query image rather than the whole image.
  arXiv  Detail & Related papers  (2024-06-05T04:08:41Z)
• Improving fine-grained understanding in image-text pre-training [37.163228122323865]
  We introduce SPARse Fine-grained Contrastive Alignment (SPARC), a simple method for pretraining more fine-grained multimodal representations from image-text pairs. We show improved performance over competing approaches over both image-level tasks relying on coarse-grained information.
  arXiv  Detail & Related papers  (2024-01-18T10:28:45Z)
• Revisiting Multimodal Representation in Contrastive Learning: From Patch and Token Embeddings to Finite Discrete Tokens [76.40196364163663]
  We propose a learning-based vision-language pre-training approach, such as CLIP. We show that our method can learn more comprehensive representations and capture meaningful cross-modal correspondence.
  arXiv  Detail & Related papers  (2023-03-27T00:58:39Z)
• Leveraging per Image-Token Consistency for Vision-Language Pre-training [52.825150269820696]
  Cross-modal masked language modeling (CMLM) is insufficient for vision-language pre-training. We propose EPIC (lEveraging Per Image-Token Consistency for vision-language pre-training) The proposed EPIC method is easily combined with pre-training methods.
  arXiv  Detail & Related papers  (2022-11-20T12:10:53Z)
• Token-Label Alignment for Vision Transformers [93.58540411138164]
  Data mixing strategies (e.g., CutMix) have shown the ability to greatly improve the performance of convolutional neural networks (CNNs) We identify a token fluctuation phenomenon that has suppressed the potential of data mixing strategies. We propose a token-label alignment (TL-Align) method to trace the correspondence between transformed tokens and the original tokens to maintain a label for each token.
  arXiv  Detail & Related papers  (2022-10-12T17:54:32Z)
• TokenFlow: Rethinking Fine-grained Cross-modal Alignment in Vision-Language Retrieval [30.429340065755436]
  We devise a new model-agnostic formulation for fine-grained cross-modal alignment. Inspired by optimal transport theory, we introduce emphTokenFlow, an instantiation of the proposed scheme.
  arXiv  Detail & Related papers  (2022-09-28T04:11:05Z)
• COTS: Collaborative Two-Stream Vision-Language Pre-Training Model for Cross-Modal Retrieval [59.15034487974549]
  We propose a novel COllaborative Two-Stream vision-language pretraining model termed COTS for image-text retrieval. Our COTS achieves the highest performance among all two-stream methods and comparable performance with 10,800X faster in inference. Importantly, our COTS is also applicable to text-to-video retrieval, yielding new state-ofthe-art on the widely-used MSR-VTT dataset.
  arXiv  Detail & Related papers  (2022-04-15T12:34:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.