Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionality

• URL: http://arxiv.org/abs/2606.13288v1
• Date: Thu, 11 Jun 2026 12:45:25 GMT
• Title: Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionality
• Abstract summary: Contrastively trained vision-language models like CLIP, have made remarkable progress in learning joint image-text representations.<n>They often exhibit a "bag-of-words" behavior--struggling to capture the object relations, attribute-object bindings, and word order dependencies.<n>We propose MACCO, a framework that masks compositional concepts in one modality and reconstructs them conditioned on the full contextual information from the other.
• Score: 18.431462064948178
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Contrastively trained vision-language models like CLIP, have made remarkable progress in learning joint image-text representations, but still face challenges in compositional understanding. They often exhibit a "bag-of-words" behavior--struggling to capture the object relations, attribute-object bindings, and word order dependencies. This limitation arises not only from the reliance on global, single-vector representations for optimization, but also from the insufficient exploitation and modeling of the rich compositional information inherently present in paired image text data. In this work, we propose MACCO (MAsked Compositional Concept MOdeling), a framework that masks compositional concepts in one modality and reconstructs them conditioned on the full contextual information from the other, enabling the model to capture and align cross-modal compositional structures more effectively. To facilitate this process, we introduce two auxiliary objectives that jointly align and regularize masked features both inter-modally and intra-modally. Extensive experiments on five compositional benchmarks, along with in-depth analyses, demonstrate that our approach not only significantly enhances compositionality in VLMs but also improves their ability to capture syntactic structure and linguistic information. Additionally, the improved compositionality also benefits text-to-image generation and multimodal large language model. Code is available at https://github.com/hiker-lw/MACCO.

Related papers

• Advancing Aesthetic Image Generation via Composition Transfer [57.624601729777986]
  Composer is a framework designed to model composition in a semantic-agnostic manner.<n>It supports composition transfer by extracting key composition-aware representations from a reference image.<n>It also supports theme-driven composition retrieval by leveraging the in-context learning capabilities of Large Vision-Language Models.
  arXiv  Detail & Related papers  (2026-05-06T07:56:16Z)
• HiMo-CLIP: Modeling Semantic Hierarchy and Monotonicity in Vision-Language Alignment [13.584710249222105]
  HiMo-CLIP is a representation-level framework that enhances CLIP-style models without modifying the encoder architecture.<n>HiMo-CLIP introduces two key components: a hierarchical decomposition (HiDe) module that extracts latent semantic components from long-form text via in-batch PCA, and a monotonicity-aware contrastive loss (MoLo)<n>Experiments on multiple image-text retrieval benchmarks show that HiMo-CLIP consistently outperforms strong baselines.
  arXiv  Detail & Related papers  (2025-11-10T03:04:36Z)
• Exploring a Unified Vision-Centric Contrastive Alternatives on Multi-Modal Web Documents [99.62178668680578]
  We propose Vision-Centric Contrastive Learning (VC2L), a unified framework that models text, images, and their combinations using a single vision transformer.<n> VC2L operates entirely in pixel space by rendering all inputs, whether textual, visual, or combined, as images.<n>To capture complex cross-modal relationships in web documents, VC2L employs a snippet-level contrastive learning objective that aligns consecutive multimodal segments.
  arXiv  Detail & Related papers  (2025-10-21T14:59:29Z)
• Object-centric Binding in Contrastive Language-Image Pretraining [9.376583779399834]
  We propose a novel approach that diverges from commonly used strategies, which rely on the design of hard-negative augmentations.<n>Our work focuses on integrating inductive biases into pre-trained CLIP-like models to improve their compositional understanding without using any additional hard-negatives.<n>Our resulting model paves the way towards more accurate and sample-efficient image-text matching of complex scenes.
  arXiv  Detail & Related papers  (2025-02-19T21:30:51Z)
• Causal Graphical Models for Vision-Language Compositional Understanding [36.24185263818946]
  We show that our method significantly outperforms all the state-of-the-art compositional approaches by a large margin.<n>It also improves over methods trained using much larger datasets.
  arXiv  Detail & Related papers  (2024-12-12T15:22:03Z)
• ComAlign: Compositional Alignment in Vision-Language Models [2.3250871476216814]
  We introduce Compositional Alignment (ComAlign) to discover more exact correspondence of text and image components. Our methodology emphasizes that the compositional structure extracted from the text modality must also be retained in the image modality. We train a lightweight network lying on top of existing visual and language encoders using a small dataset.
  arXiv  Detail & Related papers  (2024-09-12T16:46:41Z)
• InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition [111.65584066987036]
  InternLM-XComposer is a vision-language large model that enables advanced image-text comprehension and composition. It can effortlessly generate coherent and contextual articles that seamlessly integrate images. It can intelligently identify the areas in the text where images would enhance the content and automatically insert the most appropriate visual candidates.
  arXiv  Detail & Related papers  (2023-09-26T17:58:20Z)
• Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality [50.48859793121308]
  Contrastively trained vision-language models have achieved remarkable progress in vision and language representation learning. Recent research has highlighted severe limitations in their ability to perform compositional reasoning over objects, attributes, and relations.
  arXiv  Detail & Related papers  (2023-05-23T08:28:38Z)
• Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representations [70.41385310930846]
  We present an end-to-end framework Structure-CLIP to enhance multi-modal structured representations. We use scene graphs to guide the construction of semantic negative examples, which results in an increased emphasis on learning structured representations. A Knowledge-Enhance (KEE) is proposed to leverage SGK as input to further enhance structured representations.
  arXiv  Detail & Related papers  (2023-05-06T03:57:05Z)
• Decomposed Soft Prompt Guided Fusion Enhancing for Compositional Zero-Shot Learning [15.406125901927004]
  We propose a novel framework termed Decomposed Fusion with Soft Prompt (DFSP)1, by involving vision-language models (VLMs) for unseen composition recognition. Specifically, DFSP constructs a vector combination of learnable soft prompts with state and object to establish the joint representation of them. In addition, a cross-modal fusion module is designed between the language and image branches, which decomposes state and object among language features instead of image features.
  arXiv  Detail & Related papers  (2022-11-19T12:29:12Z)
• Masked Vision and Language Modeling for Multi-modal Representation Learning [62.15254888833132]
  We study how to use masked signal modeling in vision and language (V+L) representation learning. We propose to build joint masked vision and language modeling, where the masked signal of one modality is reconstructed with the help from another modality. Our experiments on various V+L tasks show that the proposed method achieves state-of-the-art performances by using a large amount of data.
  arXiv  Detail & Related papers  (2022-08-03T15:11:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.