Probabilistic Warp Consistency for Weakly-Supervised Semantic Correspondences

• URL: http://arxiv.org/abs/2203.04279v2
• Date: Tue, 31 Oct 2023 14:06:18 GMT
• Title: Probabilistic Warp Consistency for Weakly-Supervised Semantic Correspondences
• Authors: Prune Truong and Martin Danelljan and Fisher Yu and Luc Van Gool
• Abstract summary: We propose Probabilistic Warp Consistency, a weakly-supervised learning objective for semantic matching. We first construct an image triplet by applying a known warp to one of the images in a pair depicting different instances of the same object class. Our objective also brings substantial improvements in the strongly-supervised regime, when combined with keypoint annotations.
• Score: 118.6018141306409
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose Probabilistic Warp Consistency, a weakly-supervised learning objective for semantic matching. Our approach directly supervises the dense matching scores predicted by the network, encoded as a conditional probability distribution. We first construct an image triplet by applying a known warp to one of the images in a pair depicting different instances of the same object class. Our probabilistic learning objectives are then derived using the constraints arising from the resulting image triplet. We further account for occlusion and background clutter present in real image pairs by extending our probabilistic output space with a learnable unmatched state. To supervise it, we design an objective between image pairs depicting different object classes. We validate our method by applying it to four recent semantic matching architectures. Our weakly-supervised approach sets a new state-of-the-art on four challenging semantic matching benchmarks. Lastly, we demonstrate that our objective also brings substantial improvements in the strongly-supervised regime, when combined with keypoint annotations.

Related papers

• Semantics Meets Temporal Correspondence: Self-supervised Object-centric Learning in Videos [63.94040814459116]
  Self-supervised methods have shown remarkable progress in learning high-level semantics and low-level temporal correspondence. We propose a novel semantic-aware masked slot attention on top of the fused semantic features and correspondence maps. We adopt semantic- and instance-level temporal consistency as self-supervision to encourage temporally coherent object-centric representations.
  arXiv  Detail & Related papers  (2023-08-19T09:12:13Z)
• Semantic-Aware Fine-Grained Correspondence [8.29030327276322]
  We propose to learn semantic-aware fine-grained correspondence using image-level self-supervised methods. We design a pixel-level self-supervised learning objective which specifically targets fine-grained correspondence. Our method surpasses previous state-of-the-art self-supervised methods using convolutional networks on a variety of visual correspondence tasks.
  arXiv  Detail & Related papers  (2022-07-21T12:51:41Z)
• Learning Contrastive Representation for Semantic Correspondence [150.29135856909477]
  We propose a multi-level contrastive learning approach for semantic matching. We show that image-level contrastive learning is a key component to encourage the convolutional features to find correspondence between similar objects.
  arXiv  Detail & Related papers  (2021-09-22T18:34:14Z)
• A Low Rank Promoting Prior for Unsupervised Contrastive Learning [108.91406719395417]
  We construct a novel probabilistic graphical model that effectively incorporates the low rank promoting prior into the framework of contrastive learning. Our hypothesis explicitly requires that all the samples belonging to the same instance class lie on the same subspace with small dimension. Empirical evidences show that the proposed algorithm clearly surpasses the state-of-the-art approaches on multiple benchmarks.
  arXiv  Detail & Related papers  (2021-08-05T15:58:25Z)
• Warp Consistency for Unsupervised Learning of Dense Correspondences [116.56251250853488]
  Key challenge in learning dense correspondences is lack of ground-truth matches for real image pairs. We propose Warp Consistency, an unsupervised learning objective for dense correspondence regression. Our approach sets a new state-of-the-art on several challenging benchmarks, including MegaDepth, RobotCar and TSS.
  arXiv  Detail & Related papers  (2021-04-07T17:58:22Z)
• Unsupervised semantic discovery through visual patterns detection [4.129225533930966]
  Our algorithm is able to hierarchically find visual categories and produce a segmentation mask where previous methods fail. Our algorithm is composed by two phases. A filtering phase, which selects semantical hotsposts by means of an accumulator space, then a clustering phase which propagates the semantic properties of the hotspots on a superpixels basis.
  arXiv  Detail & Related papers  (2021-02-24T11:13:15Z)
• Unsupervised Landmark Learning from Unpaired Data [117.81440795184587]
  Recent attempts for unsupervised landmark learning leverage synthesized image pairs that are similar in appearance but different in poses. We propose a cross-image cycle consistency framework which applies the swapping-reconstruction strategy twice to obtain the final supervision. Our proposed framework is shown to outperform strong baselines by a large margin.
  arXiv  Detail & Related papers  (2020-06-29T13:57:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.