SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models

• URL: http://arxiv.org/abs/2605.31597v2
• Date: Mon, 01 Jun 2026 17:53:39 GMT
• Title: SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models
• Abstract summary: We introduce SOCO, a new benchmark for Semantic Object Correspondence.<n>We show that vision foundation backbones encode strong semantic structure but transfer correspondences poorly across related categories.<n>We also show that LVLMs are stronger at text-prompted part localization than at visual-reference cross-image matching.
• Score: 63.28435103335999
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Measuring structured object understanding in vision foundation models remains challenging due to inconsistent evaluation protocols and limited part-level supervision. Semantic correspondence (SC) evaluates this capability by testing whether object parts can be matched across instances and categories under large variations in appearance, viewpoint, and geometry. To enable a systematic SC evaluation, we introduce SOCO, a new benchmark for Semantic Object Correspondence that introduces a taxonomy of correspondence types and provides consistent, functionally meaningful keypoint annotations across 100 categories and over 1M correspondence pairs. In addition, SOCO includes keypoint language descriptions, enabling the evaluation of large vision-language models (LVLMs) and their fine-grained part-level understanding. Comprehensive experiments reveal that (i) vision foundation backbones encode strong semantic structure but transfer correspondences poorly across related categories and only partially capture object-part position, (ii) LVLMs are stronger at text-prompted part localization than at visual-reference cross-image matching, exposing a gap between language-grounded localization and fine-grained visual correspondence, and (iii) correspondence performance predicts performance on dense downstream tasks, including segmentation, tracking, 3D pose estimation, and 3D detection, more strongly than ImageNet classification. Together, these findings position SOCO as a benchmark for structured, part-level representation quality in vision and multimodal foundation models.

Related papers

• Learning Cross-View Semantic Priors for Single-Reference Unseen Object Pose Estimation [18.011730388391232]
  We build a correspondence pipeline around an early cross-view semantic prior.<n>Our method achieves state-of-the-art performance while maintaining comparable inference speed.
  arXiv  Detail & Related papers  (2026-06-20T14:50:51Z)
• Beyond Referring Expressions: Scenario Comprehension Visual Grounding [18.49037321302958]
  Existing benchmarks primarily evaluate alignment between image regions and literal referring expressions, where models can often succeed by matching a prominent named category.<n>We explore a complementary and more challenging setting of scenario-based visual grounding, where the target must be inferred from roles, intentions, and context rather than explicit naming.<n>We introduce Referring Scenario (RSC), a benchmark designed for this setting. RSC contains approximately 31k training examples, 4k in-domain test examples, and a 3k out-of-distribution split with unseen object categories.
  arXiv  Detail & Related papers  (2026-04-02T17:59:08Z)
• PCA-Seg: Revisiting Cost Aggregation for Open-Vocabulary Semantic and Part Segmentation [58.1914505657064]
  This paper proposes a simple yet effective parallel cost aggregation (PCA-Seg) paradigm to alleviate the challenge of knowledge interference between class-level semantics and spatial context.<n>Experiments on eight benchmarks show that each parallel block in PCA-Seg adds merely 0.35M parameters while achieving state-of-the-art OSPS performance.
  arXiv  Detail & Related papers  (2026-03-18T09:26:43Z)
• CoT Referring: Improving Referring Expression Tasks with Grounded Reasoning [67.18702329644526]
  CoT Referring enhances model reasoning across modalities through a structured, chain-of-thought training data structure.<n>We restructure the training data to enforce a new output form, providing new annotations for existing datasets.<n>We also integrate detection and segmentation capabilities into a unified MLLM framework, training it with a novel adaptive weighted loss to optimize performance.
  arXiv  Detail & Related papers  (2025-10-03T08:50:21Z)
• Prompt the Unseen: Evaluating Visual-Language Alignment Beyond Supervision [22.712690974750007]
  Vision-Language Models (VLMs) combine a vision encoder and a large language model (LLM) through alignment training.<n>Despite its importance, the projection layer's ability to generalize to unseen visual concepts has not been systematically evaluated.<n>This study introduces a new evaluation framework for alignment generalization.
  arXiv  Detail & Related papers  (2025-08-31T05:00:51Z)
• Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perception [71.26728044621458]
  DeCLIP is a novel framework that enhances CLIP by decoupling the self-attention module to obtain content'' and context'' features respectively.<n>It consistently achieves state-of-the-art performance across a broad spectrum of tasks, including 2D detection and segmentation, 3D instance segmentation, video instance segmentation, and 6D object pose estimation.
  arXiv  Detail & Related papers  (2025-08-15T06:43:51Z)
• Composed Object Retrieval: Object-level Retrieval via Composed Expressions [71.47650333199628]
  Composed Object Retrieval (COR) is a brand-new task that goes beyond image-level retrieval to achieve object-level precision.<n>We construct COR127K, the first large-scale COR benchmark that contains 127,166 retrieval triplets with various semantic transformations in 408 categories.<n>We also present CORE, a unified end-to-end model that integrates reference region encoding, adaptive visual-textual interaction, and region-level contrastive learning.
  arXiv  Detail & Related papers  (2025-08-06T13:11:40Z)
• Bridge the Gap Between Visual and Linguistic Comprehension for Generalized Zero-shot Semantic Segmentation [39.17707407384492]
  Generalized zero-shot semantic segmentation (GZS3) aims to achieve the human-level capability of segmenting seen and unseen classes.<n>We propose a Decoupled Vision-Language Matching (DeVLMatch) framework, composed of spatial-part (SPMatch) and channel-state (CSMatch) matching modules.
  arXiv  Detail & Related papers  (2025-03-31T07:39:14Z)
• HMGIE: Hierarchical and Multi-Grained Inconsistency Evaluation for Vision-Language Data Cleansing [54.970275599061594]
  We design an adaptive evaluation framework, called Hierarchical and Multi-Grained Inconsistency Evaluation (HMGIE)<n>HMGIE can provide multi-grained evaluations covering both accuracy and completeness for various image-caption pairs.<n>To verify the efficacy and flexibility of the proposed framework, we construct MVTID, an image-caption dataset with diverse types and granularities of inconsistencies.
  arXiv  Detail & Related papers  (2024-12-07T15:47:49Z)
• HIP: Hierarchical Point Modeling and Pre-training for Visual Information Extraction [24.46493675079128]
  OCR-dependent methods rely on offline OCR engines, while OCR-free methods might produce outputs that lack interpretability or contain hallucinated content. We propose HIP, which models entities as HIerarchical Points to better conform to the hierarchical nature of the end-to-end VIE task. Specifically, such hierarchical points can be flexibly encoded and subsequently decoded into desired text transcripts, centers of various regions, and categories of entities.
  arXiv  Detail & Related papers  (2024-11-02T05:00:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.