Shape-of-You: Fused Gromov-Wasserstein Optimal Transport for Semantic Correspondence in-the-Wild

• URL: http://arxiv.org/abs/2603.11618v2
• Date: Mon, 16 Mar 2026 02:06:17 GMT
• Title: Shape-of-You: Fused Gromov-Wasserstein Optimal Transport for Semantic Correspondence in-the-Wild
• Abstract summary: Shape-of-You is an unsupervised learning framework for semantic correspondence.<n>It achieves state-of-the-art performance on SPair-71k and AP-10k datasets.
• Score: 7.4746628878102195
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Semantic correspondence is essential for handling diverse in-the-wild images lacking explicit correspondence annotations. While recent 2D foundation models offer powerful features, adapting them for unsupervised learning via nearest-neighbor pseudo-labels has key limitations: it operates locally, ignoring structural relationships, and consequently its reliance on 2D appearance fails to resolve geometric ambiguities arising from symmetries or repetitive features. In this work, we address this by reformulating pseudo-label generation as a Fused Gromov-Wasserstein (FGW) problem, which jointly optimizes inter-feature similarity and intra-structural consistency. Our framework, Shape-of-You (SoY), leverages a 3D foundation model to define this intra-structure in the geometric space, resolving abovementioned ambiguity. However, since FGW is a computationally prohibitive quadratic problem, we approximate it through anchor-based linearization. The resulting probabilistic transport plan provides a structurally consistent but noisy supervisory signal. Thus, we introduce a soft-target loss dynamically blending guidance from this plan with network predictions to build a learning framework robust to this noise. SoY achieves state-of-the-art performance on SPair-71k and AP-10k datasets, establishing a new benchmark in semantic correspondence without explicit geometric annotations. Code is available at Shape-of-You.

Related papers

• Stable and Scalable Bundle Adjustment of Holistic 3D Structures [99.03818732955783]
  We present a new version of Bundle Adjustment (BA) for 3D computer vision.<n>BA is developed to jointly optimize cameras, poses and 3D points.<n>By group-induced measurements and cross-feature reprojection, we preserve the sparsity of classical point-only 3D regularization.
  arXiv  Detail & Related papers  (2026-09-03T16:08:03Z)
• LeCropFollow: Latent Space Planning for Navigation in Unstructured Crop Fields [7.898784066160524]
  We present LeCropFollow, a visual navigation framework that bypasses explicit geometric modeling in favor of a learned latent representation.<n>We show that LeCropFollow matches state-of-the-art baselines in unstructured rows but significantly outperforms them in plantation gaps.<n>These results suggest that latent planning offers a robust alternative to geometric estimation for operations in heterogeneous agricultural environments.
  arXiv  Detail & Related papers  (2026-06-30T16:47:08Z)
• SGSoft: Learning Fused Semantic-Geometric Features for 3D Shape Correspondence via Template-Guided Soft Signals [23.94595783675059]
  Learning dense correspondences across deformable 3D shapes remains a long-standing challenge due to structural variability, non-isometric deformation, and inconsistent topology.<n>We propose SGSoft, a unified intrinsic pipeline that constructs a geodesic correspondence field on a canonical template and learns multimodal dense descriptors guided by pretrained semantic priors.<n> SGSoft achieves state-of-the-art inter-category generalization while offering the best accuracy-efficiency trade-off among prior methods.
  arXiv  Detail & Related papers  (2026-05-18T08:31:04Z)
• SGMatch: Semantic-Guided Non-Rigid Shape Matching with Flow Regularization [43.12848612518533]
  We introduce SGMatch, a learning-based framework for semantic-guided non-rigid shape matching.<n> Specifically, we design a Semantic-Guided Local Cross-Attention module that integrates semantic features from vision foundation models into geometric descriptors.<n>We introduce a regularization objective based on conditional flow matching, which supervises a time-varying velocity field to encourage spatial smoothness of the recovered correspondences.
  arXiv  Detail & Related papers  (2026-03-13T12:31:43Z)
• Interp3D: Correspondence-aware Interpolation for Generative Textured 3D Morphing [63.141976759536625]
  We propose Interp3D, a training-free framework for textured 3D morphing.<n>It harnesses generative priors and adopts a progressive alignment principle to ensure both geometric fidelity and texture coherence.<n>For comprehensive evaluations, we construct a dedicated dataset, Interp3DData, with graded difficulty levels and assess generation results from fidelity, transition smoothness, and plausibility.
  arXiv  Detail & Related papers  (2026-01-20T16:03:22Z)
• Generative Adversarial Gumbel MCTS for Abstract Visual Composition Generation [29.755551944026738]
  We study abstract visual composition in which identity is determined by the configuration and relations among a small set of geometric primitives.<n>An AlphaGo-style search enforces feasibility, while a fine-tuned vision-language model scores semantic alignment as reward signals.<n>Inspired by the Generative Adversarial Network, we use the generated instances for adversarial reward refinement.
  arXiv  Detail & Related papers  (2025-12-01T03:38:44Z)
• Non-Rigid Structure-from-Motion via Differential Geometry with Recoverable Conformal Scale [17.935227965480475]
  We introduce a novel method, called Con-NRSfM, for NRSfM under conformal deformations.<n>Our approach performs point-wise reconstruction using 2D selected image warps optimized through a graph-based framework.<n>Our framework decouples constraints on depth and conformal scale, which are inseparable in other approaches.
  arXiv  Detail & Related papers  (2025-10-02T04:46:46Z)
• Dense Semantic Matching with VGGT Prior [49.42199006453071]
  We propose an approach that retains VGGT's intrinsic strengths by reusing early feature stages, fine-tuning later ones, and adding a semantic head for bidirectional correspondences.<n>Our approach achieves superior geometry awareness, matching reliability, and manifold preservation, outperforming previous baselines.
  arXiv  Detail & Related papers  (2025-09-25T14:56:11Z)
• ARC-Flow : Articulated, Resolution-Agnostic, Correspondence-Free Matching and Interpolation of 3D Shapes Under Flow Fields [4.706075725469252]
  This work presents a unified framework for the unsupervised prediction of physically plausibles between two 3D articulated shapes.<n>Interpolation is modelled as a diffeomorphic transformation using a smooth, time-varying flow field governed by Neural Ordinary Differential Equations (ODEs)<n>Correspondence is recovered using an efficient Varifold formulation, that is effective on high-fidelity surfaces with differing parameterisations.
  arXiv  Detail & Related papers  (2025-03-04T13:28:05Z)
• Hierarchical Context Alignment with Disentangled Geometric and Temporal Modeling for Semantic Occupancy Prediction [61.484280369655536]
  Camera-based 3D Semantic Occupancy Prediction (SOP) is crucial for understanding complex 3D scenes from limited 2D image observations.<n>Existing SOP methods typically aggregate contextual features to assist the occupancy representation learning.<n>We introduce a new Hierarchical context alignment paradigm for a more accurate SOP (Hi-SOP)
  arXiv  Detail & Related papers  (2024-12-11T09:53:10Z)
• Temporally-Consistent Surface Reconstruction using Metrically-Consistent Atlases [131.50372468579067]
  We propose a method for unsupervised reconstruction of a temporally-consistent sequence of surfaces from a sequence of time-evolving point clouds. We represent the reconstructed surfaces as atlases computed by a neural network, which enables us to establish correspondences between frames. Our approach outperforms state-of-the-art ones on several challenging datasets.
  arXiv  Detail & Related papers  (2021-11-12T17:48:25Z)
• Dense Non-Rigid Structure from Motion: A Manifold Viewpoint [162.88686222340962]
  Non-Rigid Structure-from-Motion (NRSfM) problem aims to recover 3D geometry of a deforming object from its 2D feature correspondences across multiple frames. We show that our approach significantly improves accuracy, scalability, and robustness against noise.
  arXiv  Detail & Related papers  (2020-06-15T09:15:54Z)
• Deep Semantic Matching with Foreground Detection and Cycle-Consistency [103.22976097225457]
  We address weakly supervised semantic matching based on a deep network. We explicitly estimate the foreground regions to suppress the effect of background clutter. We develop cycle-consistent losses to enforce the predicted transformations across multiple images to be geometrically plausible and consistent.
  arXiv  Detail & Related papers  (2020-03-31T22:38:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.