Learning Structured Visual Compositional Representations for Weakly Supervised Referring Expression Comprehension

• URL: http://arxiv.org/abs/2607.04638v1
• Date: Mon, 06 Jul 2026 03:43:32 GMT
• Title: Learning Structured Visual Compositional Representations for Weakly Supervised Referring Expression Comprehension
• Abstract summary: Referring expression comprehension aims to localize the object in an image described by natural language.<n>Existing approaches primarily operate on anchor-level visual representations.<n>We propose a Structured Visual Compositional Representation learning framework for Weakly supervised REC.
• Score: 73.15757782297459
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Referring expression comprehension (REC) aims to localize the object in an image described by natural language. In Weakly supervised REC (WREC), existing approaches primarily operate on anchor-level visual representations. Even when enriched with auxiliary cues, relational interactions remain implicitly encoded within individual anchor features. The resulting visual representation remains flat and unary-only, limiting its ability to align with the structured nature of language. In this work, we propose a Structured Visual Compositional Representation (SVCR) learning framework for WREC. Rather than implicitly encoding relations within unary anchors, the proposed SVCR explicitly models both unary object embeddings and pairwise relational embeddings, forming a structured visual representation space. We further introduce a compositional alignment mechanism that matches unary and pairwise visual representations with their corresponding textual embeddings in a unified manner, enabling compositional visual-textual matching under weak supervision. Extensive experiments on RefCOCO, RefCOCO+, and RefCOCOg show that the proposed SVCR achieves state-of-the-art performance. These results demonstrate the effectiveness of explicit structured visual representations and visual-textual alignment for WREC.

Related papers

• Learning to Compose: Revisiting Proxy Task Design for Zero-Shot Composed Image Retrieval [31.201448685871558]
  Composed Image Retrieval (CIR) retrieves a target image from a reference image and a textual modification.<n>We propose FoCo, which models composition as two coordinated stages: focusing on modification-relevant visual content, and then completing the target semantics.<n>Experiments on four ZS-CIR benchmarks show FoCo's state-of-the-art performance and improved generalization.
  arXiv  Detail & Related papers  (2026-07-01T03:20:06Z)
• SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models [63.28435103335999]
  We introduce SOCO, a new benchmark for Semantic Object Correspondence.<n>We show that vision foundation backbones encode strong semantic structure but transfer correspondences poorly across related categories.<n>We also show that LVLMs are stronger at text-prompted part localization than at visual-reference cross-image matching.
  arXiv  Detail & Related papers  (2026-05-29T17:58:48Z)
• Interpretable Zero-shot Referring Expression Comprehension with Query-driven Scene Graphs [18.414159451507153]
  Zero-shot referring expression comprehension (REC) aims to locate target objects in images given natural language queries.<n>Existing Vision-Language Models (VLMs) commonly address zero-shot REC by measuring feature similarities between textual queries and image regions.<n>We propose textbfSGREC, an interpretable zero-shot REC method leveraging query-driven scene graphs as structured intermediaries.
  arXiv  Detail & Related papers  (2026-03-26T04:05:30Z)
• Communication-Inspired Tokenization for Structured Image Representations [74.17163003465537]
  COMmunication inspired Tokenization (COMiT) is a framework for learning structured discrete visual token sequences.<n>Our experiments demonstrate that while semantic alignment provides grounding, attentive sequential tokenization is critical for inducing interpretable, object-centric token structure.
  arXiv  Detail & Related papers  (2026-02-24T09:53:50Z)
• Constrained Prompt Enhancement for Improving Zero-Shot Generalization of Vision-Language Models [57.357091028792325]
  Vision-language models (VLMs) pre-trained on web-scale data exhibit promising zero-shot generalization but often suffer from semantic misalignment.<n>We propose a novel constrained prompt enhancement (CPE) method to improve visual-textual alignment.<n>Our approach consists of two key components: Topology-Guided Synonymous Semantic Generation (TGSSG) and Category-Agnostic Discriminative Region Selection (CADRS)
  arXiv  Detail & Related papers  (2025-08-24T15:45:22Z)
• Referring Expression Instance Retrieval and A Strong End-to-End Baseline [37.47466772169063]
  Text-Image Retrieval retrieves a target image from a gallery based on an image-level description.<n>Referring Expression localizes a target object within a given image using an instance-level description.<n>We introduce a new task called textbfReferring Expression Instance Retrieval (REIR), which supports both instance-level retrieval and localization.
  arXiv  Detail & Related papers  (2025-06-23T02:28:44Z)
• Not Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models [26.525531111141717]
  Vision-Language Models learn a shared feature space for text and images, enabling the comparison of inputs of different modalities.<n>We investigate compositionality in the image domain, where the analysis of compositional properties is challenged by noise and sparsity of visual data.<n>We propose a framework, called Geodesically Decomposable Embeddings (GDE), that approximates image representations with geometry-aware compositional structures in the latent space.
  arXiv  Detail & Related papers  (2025-03-21T13:46:53Z)
• ComAlign: Compositional Alignment in Vision-Language Models [2.3250871476216814]
  We introduce Compositional Alignment (ComAlign) to discover more exact correspondence of text and image components. Our methodology emphasizes that the compositional structure extracted from the text modality must also be retained in the image modality. We train a lightweight network lying on top of existing visual and language encoders using a small dataset.
  arXiv  Detail & Related papers  (2024-09-12T16:46:41Z)
• Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representations [70.41385310930846]
  We present an end-to-end framework Structure-CLIP to enhance multi-modal structured representations. We use scene graphs to guide the construction of semantic negative examples, which results in an increased emphasis on learning structured representations. A Knowledge-Enhance (KEE) is proposed to leverage SGK as input to further enhance structured representations.
  arXiv  Detail & Related papers  (2023-05-06T03:57:05Z)
• Linear Spaces of Meanings: Compositional Structures in Vision-Language Models [110.00434385712786]
  We investigate compositional structures in data embeddings from pre-trained vision-language models (VLMs) We first present a framework for understanding compositional structures from a geometric perspective. We then explain what these structures entail probabilistically in the case of VLM embeddings, providing intuitions for why they arise in practice.
  arXiv  Detail & Related papers  (2023-02-28T08:11:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.