Reveal Hidden Pitfalls and Navigate Next Generation of Vector Similarity Search from Task-Centric Views

• URL: http://arxiv.org/abs/2512.12980v1
• Date: Mon, 15 Dec 2025 04:49:33 GMT
• Title: Reveal Hidden Pitfalls and Navigate Next Generation of Vector Similarity Search from Task-Centric Views
• Authors: Tingyang Chen, Cong Fu, Jiahua Wu, Haotian Wu, Hua Fan, Xiangyu Ke, Yunjun Gao, Yabo Ni, Anxiang Zeng,
• Abstract summary: Vector Similarity Search (VSS) in high-dimensional spaces is rapidly emerging as core functionality in next-generation database systems.<n>Current benchmarks evaluate VSS primarily on the recall-latency trade-off against a ground truth defined solely by distance metrics.<n>We present Iceberg, a holistic benchmark suite for end-to-end evaluation of VSS methods in realistic application contexts.
• Score: 24.456069497637035
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Vector Similarity Search (VSS) in high-dimensional spaces is rapidly emerging as core functionality in next-generation database systems for numerous data-intensive services -- from embedding lookups in large language models (LLMs), to semantic information retrieval and recommendation engines. Current benchmarks, however, evaluate VSS primarily on the recall-latency trade-off against a ground truth defined solely by distance metrics, neglecting how retrieval quality ultimately impacts downstream tasks. This disconnect can mislead both academic research and industrial practice. We present Iceberg, a holistic benchmark suite for end-to-end evaluation of VSS methods in realistic application contexts. From a task-centric view, Iceberg uncovers the Information Loss Funnel, which identifies three principal sources of end-to-end performance degradation: (1) Embedding Loss during feature extraction; (2) Metric Misuse, where distances poorly reflect task relevance; (3) Data Distribution Sensitivity, highlighting index robustness across skews and modalities. For a more comprehensive assessment, Iceberg spans eight diverse datasets across key domains such as image classification, face recognition, text retrieval, and recommendation systems. Each dataset, ranging from 1M to 100M vectors, includes rich, task-specific labels and evaluation metrics, enabling assessment of retrieval algorithms within the full application pipeline rather than in isolation. Iceberg benchmarks 13 state-of-the-art VSS methods and re-ranks them based on application-level metrics, revealing substantial deviations from traditional rankings derived purely from recall-latency evaluations. Building on these insights, we define a set of task-centric meta-features and derive an interpretable decision tree to guide practitioners in selecting and tuning VSS methods for their specific workloads.

Related papers

• GISA: A Benchmark for General Information-Seeking Assistant [102.30831921333755]
  GISA is a benchmark for General Information-Seeking Assistants comprising 373 human-crafted queries.<n>It integrates both deep reasoning and broad information aggregation within unified tasks, and includes a live subset with periodically updated answers to resist memorization.<n>Experiments on mainstream LLMs and commercial search products reveal that even the best-performing model achieves only 19.30% exact match score.
  arXiv  Detail & Related papers  (2026-02-09T11:44:15Z)
• Analyzing VLM-Based Approaches for Anomaly Classification and Segmentation [0.0]
  Vision-Language Models (VLMs) have revolutionized anomaly detection by enabling zero-shot and few-shot defect identification without extensive labeled datasets.<n>This project presents a comprehensive analysis of VLM-based approaches for anomaly classification (AC) and anomaly segmentation (AS)
  arXiv  Detail & Related papers  (2026-01-19T22:55:30Z)
• Few-Shot Remote Sensing Image Scene Classification with CLIP and Prompt Learning [0.9558392439655014]
  We explore prompt learning as a lightweight and efficient adaptation strategy for few-shot remote sensing image scene classification.<n>We benchmark these prompt-learning methods against two standard baselines: zero-shot CLIP with hand-crafted prompts and a linear probe trained on frozen CLIP features.<n>Our findings underscore prompt learning as a scalable and efficient solution for bridging the domain gap in satellite and aerial imagery.
  arXiv  Detail & Related papers  (2025-10-28T11:39:22Z)
• Exploring Efficient Open-Vocabulary Segmentation in the Remote Sensing [55.291219073365546]
  Open-Vocabulary Remote Sensing Image (OVRSIS) is an emerging task that adapts Open-Vocabulary (OVS) to the remote sensing (RS) domain.<n>textbfRSKT-Seg is a novel open-vocabulary segmentation framework tailored for remote sensing.<n> RSKT-Seg consistently outperforms strong OVS baselines by +3.8 mIoU and +5.9 mACC, while achieving 2x faster inference through efficient aggregation.
  arXiv  Detail & Related papers  (2025-09-15T15:24:49Z)
• Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting [70.83781268763215]
  Vision-language models (VLMs) have achieved impressive performance across diverse multimodal tasks by leveraging large-scale pre-training.<n>VLMs face unique challenges such as cross-modal feature drift, parameter interference due to shared architectures, and zero-shot capability erosion.<n>This survey aims to serve as a comprehensive and diagnostic reference for researchers developing lifelong vision-language systems.
  arXiv  Detail & Related papers  (2025-08-06T09:03:10Z)
• Benchmarking Deep Search over Heterogeneous Enterprise Data [73.55304268238474]
  We present a new benchmark for evaluating a form of retrieval-augmented generation (RAG)<n>RAG requires source-aware, multi-hop reasoning over diverse, sparsed, but related sources.<n>We build it using a synthetic data pipeline that simulates business across product planning, development, and support stages.
  arXiv  Detail & Related papers  (2025-06-29T08:34:59Z)
• Human Re-ID Meets LVLMs: What can we expect? [14.370360290704197]
  We compare the performance of the leading large vision-language models in the human re-identification task.<n>Our results confirm the strengths of LVLMs, but also their severe limitations that often lead to catastrophic answers.
  arXiv  Detail & Related papers  (2025-01-30T19:00:40Z)
• Hyperspectral Imaging-Based Perception in Autonomous Driving Scenarios: Benchmarking Baseline Semantic Segmentation Models [3.6096086024478775]
  Hyperspectral Imaging (HSI) is known for its advantages over traditional RGB imaging in remote sensing, agriculture, and medicine.<n>Several HSI datasets such as HyKo, HSI-Drive, HSI-Road, and Hyperspectral City have been made available.<n>However, a comprehensive evaluation of semantic segmentation models (SSM) using these datasets is lacking.<n>This study establishes a baseline SSM benchmark on available annotated datasets for future evaluation of HSI-based ADAS perception.
  arXiv  Detail & Related papers  (2024-10-29T14:54:13Z)
• IRSC: A Zero-shot Evaluation Benchmark for Information Retrieval through Semantic Comprehension in Retrieval-Augmented Generation Scenarios [14.336896748878921]
  This paper introduces the IRSC benchmark for evaluating the performance of embedding models in multilingual RAG tasks. The benchmark encompasses five retrieval tasks: query retrieval, title retrieval, part-of-paragraph retrieval, keyword retrieval, and summary retrieval. Our contributions include: 1) the IRSC benchmark, 2) the SSCI and RCCI metrics, and 3) insights into the cross-lingual limitations of embedding models.
  arXiv  Detail & Related papers  (2024-09-24T05:39:53Z)
• DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection [55.70982767084996]
  A critical yet frequently overlooked challenge in the field of deepfake detection is the lack of a standardized, unified, comprehensive benchmark. We present the first comprehensive benchmark for deepfake detection, called DeepfakeBench, which offers three key contributions. DeepfakeBench contains 15 state-of-the-art detection methods, 9CL datasets, a series of deepfake detection evaluation protocols and analysis tools, as well as comprehensive evaluations.
  arXiv  Detail & Related papers  (2023-07-04T01:34:41Z)
• Exposing Query Identification for Search Transparency [69.06545074617685]
  We explore the feasibility of approximate exposing query identification (EQI) as a retrieval task by reversing the role of queries and documents in two classes of search systems. We derive an evaluation metric to measure the quality of a ranking of exposing queries, as well as conducting an empirical analysis focusing on various practical aspects of approximate EQI.
  arXiv  Detail & Related papers  (2021-10-14T20:19:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.