CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning

• URL: http://arxiv.org/abs/2609.08242v1
• Date: Tue, 08 Sep 2026 04:37:06 GMT
• Title: CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning
• Abstract summary: Vision-language models (VLMs) demonstrate strong performance across compositional reasoning benchmarks.<n>Existing compositionality-aware VLMs exhibit element-specific biases, often underperforming vanilla CLIP on certain compositional elements.<n>We propose Compositional Scene Graph-guided CLIP (CS-CLIP), which uses scene graphs to identify compositional elements and construct structured negatives via selective masking.
• Score: 16.760639344669386
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Vision-language models (VLMs) demonstrate strong performance across compositional reasoning benchmarks, which require reasoning over semantic perturbations of objects, attributes, relations, and their interactions. However, our controlled analysis reveals that existing compositionality-aware VLMs exhibit element-specific biases, often underperforming vanilla CLIP on certain compositional elements. To address this, we propose Compositional Scene Graph-guided CLIP (CS-CLIP), which uses scene graphs to identify compositional elements and construct structured negatives via selective masking. We further retain negatives that are most contradictory to the original caption, forcing the model to rely on compositional structure rather than surface cues. CS-CLIP achieves state-of-the-art compositional reasoning with robust performance across compositional elements. It also preserves general vision-language capabilities such as cross-modal retrieval and downstream visual reasoning, while requiring fewer training samples than prior methods.

Related papers

• ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIP [82.88378269290965]
  We present ReasonCLIP-58M, a continual pretraining framework that integrates large-scale reasoning supervision into CLIP-style models.<n>We train a family of ReasonCLIP that improves visually grounded commonsense and compositional reasoning while also enhancing zero-shot retrieval performance.
  arXiv  Detail & Related papers  (2026-06-25T09:27:54Z)
• Advancing Aesthetic Image Generation via Composition Transfer [57.624601729777986]
  Composer is a framework designed to model composition in a semantic-agnostic manner.<n>It supports composition transfer by extracting key composition-aware representations from a reference image.<n>It also supports theme-driven composition retrieval by leveraging the in-context learning capabilities of Large Vision-Language Models.
  arXiv  Detail & Related papers  (2026-05-06T07:56:16Z)
• Structure-aware Prompt Adaptation from Seen to Unseen for Open-Vocabulary Compositional Zero-Shot Learning [86.58227205147546]
  The goal of Compositional Zero-Shot Learning (OV-CZSL) is to recognize iteration-object compositions in the open-vocabulary setting.<n>We propose Structure-aware Prompt Adaptation (SPA) method, which enables models to generalize from seen to unseen attributes and objects.
  arXiv  Detail & Related papers  (2026-03-04T07:54:28Z)
• Not Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models [26.525531111141717]
  Vision-Language Models learn a shared feature space for text and images, enabling the comparison of inputs of different modalities.<n>We investigate compositionality in the image domain, where the analysis of compositional properties is challenged by noise and sparsity of visual data.<n>We propose a framework, called Geodesically Decomposable Embeddings (GDE), that approximates image representations with geometry-aware compositional structures in the latent space.
  arXiv  Detail & Related papers  (2025-03-21T13:46:53Z)
• TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives [65.82577305915643]
  Contrastive Language-Image Pretraining (CLIP) models maximize the mutual information between text and visual modalities to learn representations. We show that generating hard'' negative captions via in-context learning and corresponding negative images with text-to-image generators offers a solution. We demonstrate that our method, named TripletCLIP, enhances the compositional capabilities of CLIP, resulting in an absolute improvement of over 9% on the SugarCrepe benchmark.
  arXiv  Detail & Related papers  (2024-11-04T19:24:59Z)
• ComAlign: Compositional Alignment in Vision-Language Models [2.3250871476216814]
  We introduce Compositional Alignment (ComAlign) to discover more exact correspondence of text and image components. Our methodology emphasizes that the compositional structure extracted from the text modality must also be retained in the image modality. We train a lightweight network lying on top of existing visual and language encoders using a small dataset.
  arXiv  Detail & Related papers  (2024-09-12T16:46:41Z)
• Graph-guided Cross-composition Feature Disentanglement for Compositional Zero-shot Learning [54.08741382593959]
  Disentanglement of visual features of primitives (i.e., attributes and objects) has shown exceptional results in Compositional Zero-shot Learning (CZSL)<n>It is challenging to learn disentangled primitive features that are general across different compositions.<n>We propose the solution of cross-composition feature disentanglement, which takes multiple primitive-sharing compositions as inputs.
  arXiv  Detail & Related papers  (2024-08-19T08:23:09Z)
• Linear Spaces of Meanings: Compositional Structures in Vision-Language Models [110.00434385712786]
  We investigate compositional structures in data embeddings from pre-trained vision-language models (VLMs) We first present a framework for understanding compositional structures from a geometric perspective. We then explain what these structures entail probabilistically in the case of VLM embeddings, providing intuitions for why they arise in practice.
  arXiv  Detail & Related papers  (2023-02-28T08:11:56Z)
• ComCLIP: Training-Free Compositional Image and Text Matching [19.373706257771673]
  Contrastive Language-Image Pretraining has demonstrated great zero-shot performance for matching images and text. We propose a novel textbftextittraining-free compositional CLIP model (ComCLIP) ComCLIP disentangles input images into subjects, objects, and action sub-images and composes CLIP's vision encoder and text encoder to perform evolving matching over compositional text embedding and sub-image embeddings.
  arXiv  Detail & Related papers  (2022-11-25T01:37:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.