Yes, we CANN: Constrained Approximate Nearest Neighbors for local feature-based visual localization

• URL: http://arxiv.org/abs/2306.09012v3
• Date: Fri, 29 Dec 2023 10:35:52 GMT
• Title: Yes, we CANN: Constrained Approximate Nearest Neighbors for local feature-based visual localization
• Authors: Dror Aiger, Andr\'e Araujo, Simon Lynen
• Abstract summary: Constrained Approximate Nearest Neighbors (CANN) is a joint solution of k-nearest-neighbors across both the geometry and appearance space using only local features. Our method significantly outperforms both state-of-the-art global feature-based retrieval and approaches using local feature aggregation schemes.
• Score: 2.915868985330569
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large-scale visual localization systems continue to rely on 3D point clouds built from image collections using structure-from-motion. While the 3D points in these models are represented using local image features, directly matching a query image's local features against the point cloud is challenging due to the scale of the nearest-neighbor search problem. Many recent approaches to visual localization have thus proposed a hybrid method, where first a global (per image) embedding is used to retrieve a small subset of database images, and local features of the query are matched only against those. It seems to have become common belief that global embeddings are critical for said image-retrieval in visual localization, despite the significant downside of having to compute two feature types for each query image. In this paper, we take a step back from this assumption and propose Constrained Approximate Nearest Neighbors (CANN), a joint solution of k-nearest-neighbors across both the geometry and appearance space using only local features. We first derive the theoretical foundation for k-nearest-neighbor retrieval across multiple metrics and then showcase how CANN improves visual localization. Our experiments on public localization benchmarks demonstrate that our method significantly outperforms both state-of-the-art global feature-based retrieval and approaches using local feature aggregation schemes. Moreover, it is an order of magnitude faster in both index and query time than feature aggregation schemes for these datasets. Code: \url{https://github.com/google-research/google-research/tree/master/cann}

Related papers

• FUSELOC: Fusing Global and Local Descriptors to Disambiguate 2D-3D Matching in Visual Localization [57.59857784298536]
  Direct 2D-3D matching algorithms require significantly less memory but suffer from lower accuracy due to the larger and more ambiguous search space. We address this ambiguity by fusing local and global descriptors using a weighted average operator within a 2D-3D search framework. We consistently improve the accuracy over local-only systems and achieve performance close to hierarchical methods while halving memory requirements.
  arXiv  Detail & Related papers  (2024-08-21T23:42:16Z)
• Are Local Features All You Need for Cross-Domain Visual Place Recognition? [13.519413608607781]
  Visual Place Recognition aims to predict the coordinates of an image based solely on visual clues. Despite recent advances, recognizing the same place when the query comes from a significantly different distribution is still a major hurdle for state of the art retrieval methods. In this work we explore whether re-ranking methods based on spatial verification can tackle these challenges.
  arXiv  Detail & Related papers  (2023-04-12T14:46:57Z)
• MeshLoc: Mesh-Based Visual Localization [54.731309449883284]
  We explore a more flexible alternative based on dense 3D meshes that does not require features matching between database images to build the scene representation. Surprisingly competitive results can be obtained when extracting features on renderings of these meshes, without any neural rendering stage. Our results show that dense 3D model-based representations are a promising alternative to existing representations and point to interesting and challenging directions for future research.
  arXiv  Detail & Related papers  (2022-07-21T21:21:10Z)
• Investigating the Role of Image Retrieval for Visual Localization -- An exhaustive benchmark [46.166955777187816]
  This paper focuses on understanding the role of image retrieval for multiple visual localization paradigms. We introduce a novel benchmark setup and compare state-of-the-art retrieval representations on multiple datasets. Using these tools and in-depth analysis, we show that retrieval performance on classical landmark retrieval or place recognition tasks correlates only for some but not all paradigms to localization performance.
  arXiv  Detail & Related papers  (2022-05-31T12:59:01Z)
• Is Geometry Enough for Matching in Visual Localization? [12.984256838490795]
  GoMatch is an alternative to visual-based matching that relies on geometric information for matching image keypoints to maps, represented as sets of bearing vectors. GoMatch improves over prior geometric-based matching work with a reduction of ($10.67m, 95.7circ$) and ($1.43m$, $34.7circ$) in average median pose errors on Cambridge Landmarks and 7-Scenes.
  arXiv  Detail & Related papers  (2022-03-24T10:55:17Z)
• Hierarchical Attention Fusion for Geo-Localization [7.544917072241684]
  We introduce a hierarchical attention fusion network using multi-scale features for geo-localization. We extract the hierarchical feature maps from a convolutional neural network (CNN) and organically fuse the extracted features for image representations. Our training is self-supervised using adaptive weights to control the attention of feature emphasis from each hierarchical level.
  arXiv  Detail & Related papers  (2021-02-18T07:07:03Z)
• Leveraging Local and Global Descriptors in Parallel to Search Correspondences for Visual Localization [6.326242067588544]
  We propose a novel parallel search framework to get nearest neighbor candidates of a query local feature. We also utilize local descriptors to construct random tree structures for obtaining nearest neighbor candidates of the query local feature.
  arXiv  Detail & Related papers  (2020-09-23T01:49:03Z)
• Inter-Image Communication for Weakly Supervised Localization [77.2171924626778]
  Weakly supervised localization aims at finding target object regions using only image-level supervision. We propose to leverage pixel-level similarities across different objects for learning more accurate object locations. Our method achieves the Top-1 localization error rate of 45.17% on the ILSVRC validation set.
  arXiv  Detail & Related papers  (2020-08-12T04:14:11Z)
• Zero-Shot Multi-View Indoor Localization via Graph Location Networks [66.05980368549928]
  indoor localization is a fundamental problem in location-based applications. We propose a novel neural network based architecture Graph Location Networks (GLN) to perform infrastructure-free, multi-view image based indoor localization. GLN makes location predictions based on robust location representations extracted from images through message-passing networks. We introduce a novel zero-shot indoor localization setting and tackle it by extending the proposed GLN to a dedicated zero-shot version.
  arXiv  Detail & Related papers  (2020-08-06T07:36:55Z)
• DH3D: Deep Hierarchical 3D Descriptors for Robust Large-Scale 6DoF Relocalization [56.15308829924527]
  We propose a Siamese network that jointly learns 3D local feature detection and description directly from raw 3D points. For detecting 3D keypoints we predict the discriminativeness of the local descriptors in an unsupervised manner. Experiments on various benchmarks demonstrate that our method achieves competitive results for both global point cloud retrieval and local point cloud registration.
  arXiv  Detail & Related papers  (2020-07-17T20:21:22Z)
• Multi-View Optimization of Local Feature Geometry [70.18863787469805]
  We address the problem of refining the geometry of local image features from multiple views without known scene or camera geometry. Our proposed method naturally complements the traditional feature extraction and matching paradigm. We show that our method consistently improves the triangulation and camera localization performance for both hand-crafted and learned local features.
  arXiv  Detail & Related papers  (2020-03-18T17:22:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.