100x Cost & Latency Reduction: Performance Analysis of AI Query Approximation using Lightweight Proxy Models

• URL: http://arxiv.org/abs/2603.15970v3
• Date: Tue, 24 Mar 2026 20:09:39 GMT
• Title: 100x Cost & Latency Reduction: Performance Analysis of AI Query Approximation using Lightweight Proxy Models
• Abstract summary: We present an AI query approximation approach that enables low cost analytics and database applications to benefit from AI queries.<n>The approach delivers >100x cost and latency reduction for the semantic filter operator and also important gains for semantic ranking.<n>We show that despite the massive gains in latency and cost, these proxy models preserve accuracy and occasionally improve accuracy across various benchmark datasets.
• Score: 6.985494432089493
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Several data warehouse and database providers have recently introduced extensions to SQL called AI Queries, enabling users to specify functions and conditions in SQL that are evaluated by LLMs, thereby broadening significantly the kinds of queries one can express over the combination of structured and unstructured data. LLMs offer remarkable semantic reasoning capabilities, making them an essential tool for complex and nuanced queries that blend structured and unstructured data. While extremely powerful, these AI queries can become prohibitively costly when invoked thousands of times. This paper provides an extensive evaluation of a recent AI query approximation approach that enables low cost analytics and database applications to benefit from AI queries. The approach delivers >100x cost and latency reduction for the semantic filter operator and also important gains for semantic ranking. The cost and performance gains come from utilizing cheap and accurate proxy models over embedding vectors. We show that despite the massive gains in latency and cost, these proxy models preserve accuracy and occasionally improve accuracy across various benchmark datasets, including the extended Amazon reviews benchmark that has 10M rows. We present an OLAP-friendly architecture within Google BigQuery for this approach for purely online (ad hoc) queries, and a low-latency HTAP database-friendly architecture in AlloyDB that could further improve the latency by moving the proxy model training offline. We present techniques that accelerate the proxy model training.

Related papers

• FinAI Data Assistant: LLM-based Financial Database Query Processing with the OpenAI Function Calling API [1.1985612872852671]
  FinAI Data Assistant is a practical approach for natural-ahead querying over financial databases.<n>System routes user requests to a small library of vetted, parameterized queries.<n>Result: Ticker-mapping accuracy is near-perfect for NASDAQ-100 and high for S&P500 firms.
  arXiv  Detail & Related papers  (2025-10-15T23:19:27Z)
• Reasoning-enhanced Query Understanding through Decomposition and Interpretation [87.56450566014625]
  ReDI is a Reasoning-enhanced approach for query understanding through Decomposition and Interpretation.<n>We compiled a large-scale dataset of real-world complex queries from a major search engine.<n> Experiments on BRIGHT and BEIR demonstrate that ReDI consistently surpasses strong baselines in both sparse and dense retrieval paradigms.
  arXiv  Detail & Related papers  (2025-09-08T10:58:42Z)
• The Case for Instance-Optimized LLMs in OLAP Databases [0.7090165638014332]
  Large Language Models (LLMs) can enhance analytics systems with powerful data summarization, cleaning, and semantic transformation capabilities.<n>We present IOLMDB, a novel system that makes LLM-enhanced database queries practical through query-specific model optimization.
  arXiv  Detail & Related papers  (2025-07-07T13:10:01Z)
• ZeroLM: Data-Free Transformer Architecture Search for Language Models [54.83882149157548]
  Current automated proxy discovery approaches suffer from extended search times, susceptibility to data overfitting, and structural complexity.<n>This paper introduces a novel zero-cost proxy methodology that quantifies model capacity through efficient weight statistics.<n>Our evaluation demonstrates the superiority of this approach, achieving a Spearman's rho of 0.76 and Kendall's tau of 0.53 on the FlexiBERT benchmark.
  arXiv  Detail & Related papers  (2025-03-24T13:11:22Z)
• Enhancing Legal Case Retrieval via Scaling High-quality Synthetic Query-Candidate Pairs [67.54302101989542]
  Legal case retrieval aims to provide similar cases as references for a given fact description. Existing works mainly focus on case-to-case retrieval using lengthy queries. Data scale is insufficient to satisfy the training requirements of existing data-hungry neural models.
  arXiv  Detail & Related papers  (2024-10-09T06:26:39Z)
• Cost-Effective Proxy Reward Model Construction with On-Policy and Active Learning [70.22819290458581]
  Reinforcement learning with human feedback (RLHF) is a widely adopted approach in current large language model pipelines. Our approach introduces two key innovations: (1) on-policy query to avoid OOD and imbalance issues in seed data, and (2) active learning to select the most informative data for preference queries.
  arXiv  Detail & Related papers  (2024-07-02T10:09:19Z)
• UQE: A Query Engine for Unstructured Databases [71.49289088592842]
  We investigate the potential of Large Language Models to enable unstructured data analytics. We propose a new Universal Query Engine (UQE) that directly interrogates and draws insights from unstructured data collections.
  arXiv  Detail & Related papers  (2024-06-23T06:58:55Z)
• PixelsDB: Serverless and NL-Aided Data Analytics with Flexible Service Levels and Prices [17.048398987952332]
  PixelsDB is an open-source data analytic system that allows users to explore data efficiently.<n>The queries are executed by a serverless query engine that offers varying prices for different performance service levels (SLAs)<n>We demonstrate that the combination of a serverless paradigm, a natural-language-aided interface, and flexible SLAs and prices will substantially improve the usability of cloud data analytic systems.
  arXiv  Detail & Related papers  (2024-05-30T07:48:43Z)
• ERATTA: Extreme RAG for Table To Answers with Large Language Models [1.3318204310917532]
  Large language models (LLMs) with retrieval augmented-generation (RAG) have been the optimal choice for scalable generative AI solutions. We propose a unique LLM-based system where multiple LLMs can be invoked to enable data authentication, user-query routing, data-retrieval and custom prompting for question-answering capabilities from Enterprise-data tables. Our proposed system and scoring metrics achieve >90% confidence scores across hundreds of user queries in the sustainability, financial health and social media domains.
  arXiv  Detail & Related papers  (2024-05-07T02:49:59Z)
• Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity [59.57065228857247]
  Retrieval-augmented Large Language Models (LLMs) have emerged as a promising approach to enhancing response accuracy in several tasks, such as Question-Answering (QA) We propose a novel adaptive QA framework, that can dynamically select the most suitable strategy for (retrieval-augmented) LLMs based on the query complexity. We validate our model on a set of open-domain QA datasets, covering multiple query complexities, and show that ours enhances the overall efficiency and accuracy of QA systems.
  arXiv  Detail & Related papers  (2024-03-21T13:52:30Z)
• Improving Text Matching in E-Commerce Search with A Rationalizable, Intervenable and Fast Entity-Based Relevance Model [78.80174696043021]
  We propose a novel model called the Entity-Based Relevance Model (EBRM) The decomposition allows us to use a Cross-encoder QE relevance module for high accuracy. We also show that pretraining the QE module with auto-generated QE data from user logs can further improve the overall performance.
  arXiv  Detail & Related papers  (2023-07-01T15:44:53Z)
• Approximating Aggregated SQL Queries With LSTM Networks [31.528524004435933]
  We present a method for query approximation, also known as approximate query processing (AQP) We use LSTM network to learn the relationship between queries and their results, and to provide a rapid inference layer for predicting query results. Our method was able to predict up to 120,000 queries in a second, and with a single query latency of no more than 2ms.
  arXiv  Detail & Related papers  (2020-10-25T16:17:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.