Pre-Execution Query Slot-Time Prediction in Cloud Data Warehouses: A Feature-Scoped Machine Learning Approach

• URL: http://arxiv.org/abs/2604.20145v1
• Date: Wed, 22 Apr 2026 03:15:16 GMT
• Title: Pre-Execution Query Slot-Time Prediction in Cloud Data Warehouses: A Feature-Scoped Machine Learning Approach
• Abstract summary: Cloud data warehouses bill compute based on slot-time consumed.<n>In shared multitenant environments, query cost is variable and hard to estimate before execution.<n>We present a machine learning approach that predicts BigQuery slot-time before execution using only pre-execution observable signals.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Cloud data warehouses bill compute based on slot-time consumed. In shared multi-tenant environments, query cost is highly variable and hard to estimate before execution, causing budget overruns and degraded scheduling. Static query-planner heuristics fail to capture complex SQL structure, data skew, and workload contention. We present a feature-scoped machine learning approach that predicts BigQuery slot-time before execution using only pre-execution observable signals: a structured query complexity score derived from SQL operator costs, data volume features from planner estimates and workload metadata, and textual features from query text. We deliberately exclude runtime factors (slot-pool utilization, cache state, realized skew) unknowable at submission. The model uses a HistGradientBoostingRegressor trained on log-transformed slot-time, with a TF-IDF + TruncatedSVD-512 text pipeline fused with numeric and categorical features. Trained on 749 queries across seven deployment environments and evaluated out-of-distribution on 746 queries from two held-out environments, the model achieves MAE 1.17 slot-minutes, RMSE 4.71, and 74% explained variance on the full workload. On cost-significant queries (slot-time >= 0.01 min, N=282) the model achieves MAE 3.10 versus 4.95 for a predict-mean baseline and 4.54 for predict-median, a 30-37% reduction. On long-tail queries (>= 20 min, N=22) the model does not outperform trivial baselines, consistent with the hypothesis that long-tail queries are dominated by unobserved runtime factors outside the current feature scope. A complexity-routed dual-model architecture is described as a practical refinement, and directions for closing the long-tail gap are identified as future work.

Related papers

• Conversational Query Engine for Mixed-Modality Heterogeneous Enterprise Data Sources [0.816316992802876]
  Existing AI-enabled interfaces force users to select the right tool.<n>We present COGNI, a production conversational BI system that treats natural-language analytics as a heterogeneous query processing problem.
  arXiv  Detail & Related papers  (2026-06-15T17:02:22Z)
• LAPRAS : Learning-Augmented PRivate Answering for linear query Streams [9.706153384025809]
  We present LAPRAS, which assumes access to an oracle that outputs a prediction set of queries likely to appear in the stream.<n>LAPRAS answers predicted queries using the offline-optimal Matrix Mechanism and answers the remaining queries online from a residual budget.<n> Empirically, over two real datasets, we validate the intended consistency--robustness trade-off.
  arXiv  Detail & Related papers  (2026-05-03T16:46:53Z)
• TopBench: A Benchmark for Implicit Prediction and Reasoning over Tabular Question Answering [80.93487993878836]
  A common class of real-world queries is implicitly predictive, requiring the inference of unobserved answers from historical patterns rather than mere retrieval.<n>These queries introduce two challenges: recognizing latent intent and reliable predictive reasoning over massive tables.<n>We introduce TopBench, a benchmark consisting of 779 samples across four sub-tasks, ranging from single-point prediction to decision making, treatment effect analysis, and complex reasoning.
  arXiv  Detail & Related papers  (2026-04-30T16:22:51Z)
• BatchDAG: LLM-Planned Execution Graphs for Scalable Ad-Hoc Analysis Over Enterprise Data [0.0]
  BatchDAG is a system in which an LLM generates a typed acyclic graph (DAG) of operations.<n> BatchDAG processes queries over 50,000+ meetings in under 60 seconds.<n>In production at Brevian, BatchDAG processes queries over 50,000+ meetings in under 60 seconds.
  arXiv  Detail & Related papers  (2026-04-17T19:22:34Z)
• HuggingR$^{4}$: A Progressive Reasoning Framework for Discovering Optimal Model Companions [50.61510609116118]
  HuggingR$4$ is a novel framework that combines Reasoning, Retrieval, Refinement, and Reflection to efficiently select models.<n>It attains a workability rate of 92.03% and a reasonability rate of 82.46%, surpassing existing method by 26.51% and 33.25% respectively.
  arXiv  Detail & Related papers  (2025-11-24T03:13:45Z)
• ARIMA_PLUS: Large-scale, Accurate, Automatic and Interpretable In-Database Time Series Forecasting and Anomaly Detection in Google BigQuery [5.170341372465115]
  Time series forecasting and anomaly detection are common tasks for practitioners in industries such as retail, manufacturing, advertising and energy.<n>We present ARIMA_PLUS, a novel framework to overcome these two challenges by a unique combination of (a) accurate and interpretable time series models and (b) scalable and fully managed system infrastructure.
  arXiv  Detail & Related papers  (2025-10-28T14:18:50Z)
• Sleep-time Compute: Beyond Inference Scaling at Test-time [52.87976180878445]
  Scaling test-time compute has emerged as a key ingredient for enabling large language models to solve difficult problems.<n>We introduce sleep-time compute, which allows models to "think" offline about contexts before queries are presented.
  arXiv  Detail & Related papers  (2025-04-17T17:59:25Z)
• Time-MQA: Time Series Multi-Task Question Answering with Context Enhancement [55.2439260314328]
  Time Series Multi-Task Question Answering (Time-MQA) is a unified framework that enables natural language queries across multiple time series tasks.<n>Central to Time-MQA is the TSQA dataset, a large-scale dataset containing $sim $200k question-answer pairs.
  arXiv  Detail & Related papers  (2025-02-26T13:47:13Z)
• On Aggregation Queries over Predicted Nearest Neighbors [33.06696811081107]
  We introduce AQNNs, a novel type of aggregation queries over the predicted neighborhood of a designated object.<n>AQNNs are prevalent in modern applications where, for instance, a medical professional may want to compute "the average systolic blood pressure of patients whose predicted condition is similar to a given insomnia patient"<n>Since prediction typically involves an expensive deep learning model or a human expert, we formulate query processing as the problem of returning an approximate aggregate.
  arXiv  Detail & Related papers  (2025-02-26T04:17:32Z)
• Improving DBMS Scheduling Decisions with Fine-grained Performance Prediction on Concurrent Queries -- Extended [15.354441937462271]
  This work introduces IconqSched, a new, principled non-intrusive scheduler that optimize execution order and timing of queries.<n>IconqSched features a novel fine-grained predictor, Iconq, which treats the system runtime as a black box.<n>We compare IconqSched to other schedulers in terms of end-to-end runtime using real workload traces.
  arXiv  Detail & Related papers  (2025-01-27T17:55:39Z)
• Improving Text Matching in E-Commerce Search with A Rationalizable, Intervenable and Fast Entity-Based Relevance Model [78.80174696043021]
  We propose a novel model called the Entity-Based Relevance Model (EBRM) The decomposition allows us to use a Cross-encoder QE relevance module for high accuracy. We also show that pretraining the QE module with auto-generated QE data from user logs can further improve the overall performance.
  arXiv  Detail & Related papers  (2023-07-01T15:44:53Z)
• How to Query An Oracle? Efficient Strategies to Label Data [59.89900843097016]
  We consider the basic problem of querying an expert oracle for labeling a dataset in machine learning. We present a randomized batch algorithm that operates on a round-by-round basis to label the samples and achieves a query rate of $O(fracNk2)$. In addition, we present an adaptive greedy query scheme, which achieves an average rate of $approx 0.2N$ queries per sample with triplet queries.
  arXiv  Detail & Related papers  (2021-10-05T20:15:35Z)
• Approximating Aggregated SQL Queries With LSTM Networks [31.528524004435933]
  We present a method for query approximation, also known as approximate query processing (AQP) We use LSTM network to learn the relationship between queries and their results, and to provide a rapid inference layer for predicting query results. Our method was able to predict up to 120,000 queries in a second, and with a single query latency of no more than 2ms.
  arXiv  Detail & Related papers  (2020-10-25T16:17:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.