Filtered Vector Search in a Disaggregated Lakehouse: Composing Table-Format Pruning with Per-File ANN

• URL: http://arxiv.org/abs/2608.05441v1
• Date: Wed, 05 Aug 2026 22:20:01 GMT
• Title: Filtered Vector Search in a Disaggregated Lakehouse: Composing Table-Format Pruning with Per-File ANN
• Abstract summary: We embed an IVF index in each Parquet file's footer.<n>The planner prunes data files by the predicate first, then runs IVF only over the survivors.<n>On an 11.5M x 768 table, warm IVF search is 32x faster than brute force at recall@10 >= 0.90.
• Score: 1.0109137908797041
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Approximate nearest-neighbor (ANN) search increasingly runs alongside structured data - "find the 10 nearest documents where tenant='acme' AND lang='en'" - yet similarity and filtering are usually bolted together: a specialized vector index for one, a separate filter step for the other. We ask what happens when both live inside an open lakehouse table (Apache Iceberg over Parquet on object storage), where the engine already owns a mature file-pruning stack (partition pruning, zone-maps, a bitmap index). We embed an IVF index in place in each Parquet file's footer and make filtered vector queries fast not with a new filtering algorithm but by composing the table's existing file pruning with per-file ANN: the planner prunes data files by the predicate first, then runs IVF only over the survivors. The index is built distributed and non-destructively - a metadata-only Iceberg replace that every other engine still reads - and a rendezvous-hashed per-file cache keeps object-store read latency from swamping the algorithmic win. The payoff comes entirely from file pruning. On an 11.5M x 768 table, warm IVF search is ~32x faster than brute force at recall@10 >= 0.90, a selective predicate having pruned 355 of 444 data files before ANN runs; on 5M real IBM Granite embeddings, a filter arriving across a join prunes four of five region partitions and runs nearly two orders of magnitude (~94x: 14.7 s -> 157 ms) faster than the query-time join at identical top-k, once the reduction is materialized into a region-partitioned layout. We characterize when the composition pays off - it requires file-level locality on the filter column, and the residual predicate is only safe to push into the search over a provably pure (partitioned) column, not a merely sorted one - and report the failure modes we hit bolting ANN onto a lakehouse engine.

Related papers

• PIVOT: Efficient Query-Group Indexing for Token-Level Sparse Attention [19.844672835223676]
  Token-level sparse attention, as implemented by DeepSeek S Attention (DSA) in production systems, makes the downstream attention efficient but shifts the bottleneck to the indexer that feeds it.<n>We exploit these properties in PIVOT, Proxy Indexing Via One full-parse Traversal, a training-free, drop-in replacement for the DSA indexer.<n>On DeepSeek-V3.2 and GLM-5.1 across LongBench and RULER, PIVOT matches the accuracy of the dense DSA indexer while accelerating it by up to 4x and reducing end-to-end latency by
  arXiv  Detail & Related papers  (2026-07-27T15:58:07Z)
• Improving Long-Context Retrieval with Multi-Prefix Embedding [64.08968715893504]
  Multi-Prefix Embedding partitions a document into chunks separated by EOS tokens, encodes the full sequence in a single causal forward pass, and extracts one embedding at each prefix boundary.<n>MPE retains cross-chunk context, enables chunk-level MaxSim matching, and trains with only document-level relevance labels.
  arXiv  Detail & Related papers  (2026-06-22T17:31:02Z)
• MLSkip: Data Skipping for ML Filters via Lightweight Metadata [8.09725161771284]
  Database vendors recently released AI functions that can be used in filter predicates.<n>Traditional data skipping techniques for integer and string data fail to be applicable to the new filter type.<n>We make the case that Parquet's default min-max metadata is enough to enable pruning.
  arXiv  Detail & Related papers  (2026-06-02T17:36:06Z)
• HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention [62.79085204939384]
  HISA (Hierarchical Indexed Sparse Attention) rewrites the search path from a flat token scan into a two-stage hierarchical procedure.<n>On kernel-level benchmarks, HISA achieves up to speedup at 64K context.
  arXiv  Detail & Related papers  (2026-03-30T13:59:51Z)
• IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse [68.18308357205586]
  Longcontext agentic have emerged as a defining use case for large language models.<n>Sparse attention addresses this challenge effectively, and DeepSeek Sparse Attention (DSA) is a representative production-grade solution.<n>We present IndexCache, which exploits cross-layer redundancy by partitioning layers into a small set of Full layers that run their own indexers and a majority of Shared layers that simply reuse the nearest Full layer's top-k indices.
  arXiv  Detail & Related papers  (2026-03-12T17:27:21Z)
• B+ANN: A Fast Billion-Scale Disk-based Nearest-Neighbor Index [3.4720326275852]
  We present a novel disk-based ANN index, B+ANN, to address issues of the HNSW algorithm.<n>It partitions input data into blocks containing semantically similar items, then builds an B+ tree variant to store blocks both in-memory and on disks.<n>It improves both quality (Recall value), and execution performance (Queries per second/QPS) over HNSW.
  arXiv  Detail & Related papers  (2025-11-19T15:50:28Z)
• FuseSampleAgg: Fused Neighbor Sampling and Aggregation for Mini-batch GNNs [51.56484100374058]
  FuseSampleAgg fuses neighbor and sampling mean aggregation into a single pass for GraphSAGE.<n>Operator is deterministic, integrates with standard PyTorchs, and ships with scripts that reproduce all tables and figures from CSV logs.
  arXiv  Detail & Related papers  (2025-11-17T17:57:18Z)
• DISTRIBUTEDANN: Efficient Scaling of a Single DISKANN Graph Across Thousands of Computers [9.854925314375024]
  We present DISTRIBUTEDANN, a distributed vector search service that makes it possible to search over a single 50 billion vector graph index spread across over a thousand machines.<n>This is 6x more efficient than existing partitioning and routing strategies that route the vector query to a subset of partitions in a scale out vector search system.
  arXiv  Detail & Related papers  (2025-09-07T13:13:02Z)
• FP-NAS: Fast Probabilistic Neural Architecture Search [49.21560787752714]
  Probabilistic NAS, such as PARSEC, learns a distribution over high-performing architectures, and uses only as much memory as needed to train a single model. We propose a sampling method adaptive to the distribution entropy, drawing more samples to encourage explorations at the beginning, and reducing samples as learning proceeds. We show that Fast Probabilistic NAS (FP-NAS) can sample 64% fewer architectures and search 2.1x faster than PARSEC.
  arXiv  Detail & Related papers  (2020-11-22T06:10:05Z)
• The Case for Learned Spatial Indexes [62.88514422115702]
  We use techniques proposed from a state-of-the art learned multi-dimensional index structure (namely, Flood) to answer spatial range queries. We show that (i) machine learned search within a partition is faster by 11.79% to 39.51% than binary search when using filtering on one dimension. We also refine using machine learned indexes is 1.23x to 1.83x times faster than closest competitor which filters on two dimensions.
  arXiv  Detail & Related papers  (2020-08-24T12:09:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.