Segment Anything Model is a Good Teacher for Local Feature Learning

• URL: http://arxiv.org/abs/2309.16992v3
• Date: Tue, 18 Jun 2024 03:11:59 GMT
• Title: Segment Anything Model is a Good Teacher for Local Feature Learning
• Authors: Jingqian Wu, Rongtao Xu, Zach Wood-Doughty, Changwei Wang, Shibiao Xu, Edmund Y. Lam,
• Abstract summary: Local feature detection and description play an important role in many computer vision tasks. Data-driven local feature learning methods need to rely on pixel-level correspondence for training. We propose SAMFeat to introduce SAM as a teacher to guide local feature learning.
• Score: 19.66262816561457
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Local feature detection and description play an important role in many computer vision tasks, which are designed to detect and describe keypoints in "any scene" and "any downstream task". Data-driven local feature learning methods need to rely on pixel-level correspondence for training, which is challenging to acquire at scale, thus hindering further improvements in performance. In this paper, we propose SAMFeat to introduce SAM (segment anything model), a fundamental model trained on 11 million images, as a teacher to guide local feature learning and thus inspire higher performance on limited datasets. To do so, first, we construct an auxiliary task of Attention-weighted Semantic Relation Distillation (ASRD), which distillates feature relations with category-agnostic semantic information learned by the SAM encoder into a local feature learning network, to improve local feature description using semantic discrimination. Second, we develop a technique called Weakly Supervised Contrastive Learning Based on Semantic Grouping (WSC), which utilizes semantic groupings derived from SAM as weakly supervised signals, to optimize the metric space of local descriptors. Third, we design an Edge Attention Guidance (EAG) to further improve the accuracy of local feature detection and description by prompting the network to pay more attention to the edge region guided by SAM. SAMFeat's performance on various tasks such as image matching on HPatches, and long-term visual localization on Aachen Day-Night showcases its superiority over previous local features. The release code is available at https://github.com/vignywang/SAMFeat.

Related papers

• Self-supervised Learning via Cluster Distance Prediction for Operating Room Context Awareness [44.15562068190958]
  In the Operating Room, semantic segmentation is at the core of creating robots aware of clinical surroundings. State-of-the-art semantic segmentation and activity recognition approaches are fully supervised, which is not scalable. We propose a new 3D self-supervised task for OR scene understanding utilizing OR scene images captured with ToF cameras.
  arXiv  Detail & Related papers  (2024-07-07T17:17:52Z)
• Auxiliary Tasks Enhanced Dual-affinity Learning for Weakly Supervised Semantic Segmentation [79.05949524349005]
  We propose AuxSegNet+, a weakly supervised auxiliary learning framework to explore the rich information from saliency maps. We also propose a cross-task affinity learning mechanism to learn pixel-level affinities from the saliency and segmentation feature maps.
  arXiv  Detail & Related papers  (2024-03-02T10:03:21Z)
• AEGIS-Net: Attention-guided Multi-Level Feature Aggregation for Indoor Place Recognition [12.728087388529028]
  AEGIS-Net is a novel indoor place recognition model that takes in RGB point clouds and generates global place descriptors. Our AEGIS-Net is made of a semantic encoder, a semantic decoder and an attention-guided feature embedding. We evaluate our AEGIS-Net on the ScanNetPR dataset and compare its performance with a pre-deep-learning feature-based method and five state-of-the-art deep-learning-based methods.
  arXiv  Detail & Related papers  (2023-12-15T05:09:08Z)
• Location-Aware Self-Supervised Transformers [74.76585889813207]
  We propose to pretrain networks for semantic segmentation by predicting the relative location of image parts. We control the difficulty of the task by masking a subset of the reference patch features visible to those of the query. Our experiments show that this location-aware pretraining leads to representations that transfer competitively to several challenging semantic segmentation benchmarks.
  arXiv  Detail & Related papers  (2022-12-05T16:24:29Z)
• LEAD: Self-Supervised Landmark Estimation by Aligning Distributions of Feature Similarity [49.84167231111667]
  Existing works in self-supervised landmark detection are based on learning dense (pixel-level) feature representations from an image. We introduce an approach to enhance the learning of dense equivariant representations in a self-supervised fashion. We show that having such a prior in the feature extractor helps in landmark detection, even under drastically limited number of annotations.
  arXiv  Detail & Related papers  (2022-04-06T17:48:18Z)
• Remote Sensing Images Semantic Segmentation with General Remote Sensing Vision Model via a Self-Supervised Contrastive Learning Method [13.479068312825781]
  We propose Global style and Local matching Contrastive Learning Network (GLCNet) for remote sensing semantic segmentation. Specifically, the global style contrastive module is used to learn an image-level representation better. The local features matching contrastive module is designed to learn representations of local regions which is beneficial for semantic segmentation.
  arXiv  Detail & Related papers  (2021-06-20T03:03:40Z)
• Learning to Relate Depth and Semantics for Unsupervised Domain Adaptation [87.1188556802942]
  We present an approach for encoding visual task relationships to improve model performance in an Unsupervised Domain Adaptation (UDA) setting. We propose a novel Cross-Task Relation Layer (CTRL), which encodes task dependencies between the semantic and depth predictions. Furthermore, we propose an Iterative Self-Learning (ISL) training scheme, which exploits semantic pseudo-labels to provide extra supervision on the target domain.
  arXiv  Detail & Related papers  (2021-05-17T13:42:09Z)
• Goal-Oriented Gaze Estimation for Zero-Shot Learning [62.52340838817908]
  We introduce a novel goal-oriented gaze estimation module (GEM) to improve the discriminative attribute localization. We aim to predict the actual human gaze location to get the visual attention regions for recognizing a novel object guided by attribute description. This work implies the promising benefits of collecting human gaze dataset and automatic gaze estimation algorithms on high-level computer vision tasks.
  arXiv  Detail & Related papers  (2021-03-05T02:14:57Z)
• Learning Invariant Representations for Reinforcement Learning without Reconstruction [98.33235415273562]
  We study how representation learning can accelerate reinforcement learning from rich observations, such as images, without relying either on domain knowledge or pixel-reconstruction. Bisimulation metrics quantify behavioral similarity between states in continuous MDPs. We demonstrate the effectiveness of our method at disregarding task-irrelevant information using modified visual MuJoCo tasks.
  arXiv  Detail & Related papers  (2020-06-18T17:59:35Z)
• Learning Local Features with Context Aggregation for Visual Localization [24.167882373322957]
  Keypoint detection and description is fundamental yet important in many vision applications. Most existing methods use detect-then-describe or detect-and-describe strategy to learn local features without considering their context information. In this paper, we focus on the fusion of low-level textual information and high-level semantic context information to improve the discrimitiveness of local features.
  arXiv  Detail & Related papers  (2020-05-26T17:19:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.