Results of the NeurIPS'21 Challenge on Billion-Scale Approximate Nearest Neighbor Search

• URL: http://arxiv.org/abs/2205.03763v1
• Date: Sun, 8 May 2022 02:41:54 GMT
• Title: Results of the NeurIPS'21 Challenge on Billion-Scale Approximate Nearest Neighbor Search
• Abstract summary: This competition compares ANNS algorithms at billion-scale by hardware cost, accuracy and performance. We compiled six diverse billion-scale datasets, four newly released for this competition.
• Score: 57.18075258042082
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Despite the broad range of algorithms for Approximate Nearest Neighbor Search, most empirical evaluations of algorithms have focused on smaller datasets, typically of 1 million points~\citep{Benchmark}. However, deploying recent advances in embedding based techniques for search, recommendation and ranking at scale require ANNS indices at billion, trillion or larger scale. Barring a few recent papers, there is limited consensus on which algorithms are effective at this scale vis-\`a-vis their hardware cost. This competition compares ANNS algorithms at billion-scale by hardware cost, accuracy and performance. We set up an open source evaluation framework and leaderboards for both standardized and specialized hardware. The competition involves three tracks. The standard hardware track T1 evaluates algorithms on an Azure VM with limited DRAM, often the bottleneck in serving billion-scale indices, where the embedding data can be hundreds of GigaBytes in size. It uses FAISS~\citep{Faiss17} as the baseline. The standard hardware track T2 additional allows inexpensive SSDs in addition to the limited DRAM and uses DiskANN~\citep{DiskANN19} as the baseline. The specialized hardware track T3 allows any hardware configuration, and again uses FAISS as the baseline. We compiled six diverse billion-scale datasets, four newly released for this competition, that span a variety of modalities, data types, dimensions, deep learning models, distance functions and sources. The outcome of the competition was ranked leaderboards of algorithms in each track based on recall at a query throughput threshold. Additionally, for track T3, separate leaderboards were created based on recall as well as cost-normalized and power-normalized query throughput.

Related papers

• Consensus-gated Multi-Agent Neural Architecture Search for Seismic Fault Segmentation [0.17188280334580194]
  Neural networks for seismic fault segmentation are often borrowed from computer vision and medical imaging domains.<n>Classical neural architecture search (NAS) is restricted to hand-crafted search spaces and large compute budgets.<n>We present an agentic NAS system in which a panel of three large language models (Claude, GPT-5.1, and Gemini2.5Pro) debates each candidate architecture to unanimous consensus.
  arXiv  Detail & Related papers  (2026-08-14T02:41:37Z)
• Ember: An Extensible Benchmark Suite for Quantum Annealing Embedding Algorithms [0.4132158161225705]
  Minor embedding is required to map logical problem graphs onto sparse hardware topologies.<n>Prior studies use incompatible graph libraries, inconsistent metrics, and non-reproducible experimental setups, making cross-algorithm comparisons unreliable.<n>We present Ember, an open-source benchmarking framework addressing this gap.
  arXiv  Detail & Related papers  (2026-04-28T09:40:10Z)
• Structured search algorithm: A quantum leap [0.0]
  We introduce a structured quantum search algorithm that leverages entanglement maps and a fixed-point method to minimize oracle query complexity in unsorted datasets.<n> Experimental results on IBM Kyiv hardware demonstrate successful searches in datasets with up to 5 TB of unsorted data.
  arXiv  Detail & Related papers  (2025-04-04T13:16:53Z)
• Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation? [90.30635552818875]
  We present Touchstone, a large-scale collaborative segmentation benchmark of 9 types of abdominal organs. This benchmark is based on 5,195 training CT scans from 76 hospitals around the world and 5,903 testing CT scans from 11 additional hospitals. We invited 14 inventors of 19 AI algorithms to train their algorithms, while our team, as a third party, independently evaluated these algorithms on three test sets.
  arXiv  Detail & Related papers  (2024-11-06T05:09:34Z)
• ParlayANN: Scalable and Deterministic Parallel Graph-Based Approximate Nearest Neighbor Search Algorithms [5.478671305092084]
  We introduce ParlayANN, a library of deterministic and parallel graph-based approximate nearest neighbor search algorithms. We develop novel parallel implementations for four state-of-the-art graph-based ANNS algorithms that scale to billion-scale datasets.
  arXiv  Detail & Related papers  (2023-05-07T19:28:23Z)
• MLGWSC-1: The first Machine Learning Gravitational-Wave Search Mock Data Challenge [110.7678032481059]
  We present the results of the first Machine Learning Gravitational-Wave Search Mock Data Challenge (MLGWSC-1). For this challenge, participating groups had to identify gravitational-wave signals from binary black hole mergers of increasing complexity and duration embedded in progressively more realistic noise. Our results show that current machine learning search algorithms may already be sensitive enough in limited parameter regions to be useful for some production settings.
  arXiv  Detail & Related papers  (2022-09-22T16:44:59Z)
• A Metaheuristic Algorithm for Large Maximum Weight Independent Set Problems [58.348679046591265]
  Given a node-weighted graph, find a set of independent (mutually nonadjacent) nodes whose node-weight sum is maximum. Some of the graphs airsing in this application are large, having hundreds of thousands of nodes and hundreds of millions of edges. We develop a new local search algorithm, which is a metaheuristic in the greedy randomized adaptive search framework.
  arXiv  Detail & Related papers  (2022-03-28T21:34:16Z)
• Are we ready for beyond-application high-volume data? The Reeds robot perception benchmark dataset [3.781421673607643]
  This paper presents a dataset, called Reeds, for research on robot perception algorithms. The dataset aims to provide demanding benchmark opportunities for algorithms, rather than providing an environment for testing application-specific solutions.
  arXiv  Detail & Related papers  (2021-09-16T23:21:42Z)
• OPANAS: One-Shot Path Aggregation Network Architecture Search for Object Detection [82.04372532783931]
  Recently, neural architecture search (NAS) has been exploited to design feature pyramid networks (FPNs) We propose a novel One-Shot Path Aggregation Network Architecture Search (OPANAS) algorithm, which significantly improves both searching efficiency and detection accuracy.
  arXiv  Detail & Related papers  (2021-03-08T01:48:53Z)
• NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size [31.903475598150152]
  We propose NATS-Bench, a unified benchmark on searching for both architecture topology and size. NATS-Bench includes the search space of 15,625 neural cell candidates for architecture topology and 32,768 for architecture size on three datasets.
  arXiv  Detail & Related papers  (2020-08-28T21:34:56Z)
• Distributed Tera-Scale Similarity Search with MPI: Provably Efficient Similarity Search over billions without a Single Distance Computation [40.75034970144169]
  SLASH is a distributed system for approximate similarity search over terabyte scale datasets. We show how SLASH can index this 2.3 terabyte data over 20 nodes in under an hour, with query times in a fraction of milliseconds.
  arXiv  Detail & Related papers  (2020-08-05T18:15:36Z)
• Local Search is a Remarkably Strong Baseline for Neural Architecture Search [0.0]
  We consider, for the first time, a simple Local Search (LS) algorithm for Neural Architecture Search (NAS) We release two benchmark datasets, named MacroNAS-C10 and MacroNAS-C100, containing 200K saved network evaluations for two established image classification tasks.
  arXiv  Detail & Related papers  (2020-04-20T00:08:34Z)
• NWPU-Crowd: A Large-Scale Benchmark for Crowd Counting and Localization [101.13851473792334]
  We construct a large-scale congested crowd counting and localization dataset, NWPU-Crowd, consisting of 5,109 images, in a total of 2,133,375 annotated heads with points and boxes. Compared with other real-world datasets, it contains various illumination scenes and has the largest density range (020,033) We describe the data characteristics, evaluate the performance of some mainstream state-of-the-art (SOTA) methods, and analyze the new problems that arise on the new data.
  arXiv  Detail & Related papers  (2020-01-10T09:26:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.