PromAssistant: Leveraging Large Language Models for Text-to-PromQL

• URL: http://arxiv.org/abs/2503.03114v2
• Date: Fri, 14 Mar 2025 05:57:16 GMT
• Title: PromAssistant: Leveraging Large Language Models for Text-to-PromQL
• Authors: Chenxi Zhang, Bicheng Zhang, Dingyu Yang, Xin Peng, Miao Chen, Senyu Xie, Gang Chen, Wei Bi, Wei Li,
• Abstract summary: We focus on PromQL, which is the metric query DSL provided by the widely used metric monitoring system Prometheus.<n>We propose PromAssistant, a Large Language Model-based text-to-PromQL framework.<n>To the best of our knowledge, this paper is the first study of text-to-PromQL, and PromAssistant pioneered the DSL generation framework for metric querying and analysis.
• Score: 22.44987357626691
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the increasing complexity of modern online service systems, understanding the state and behavior of the systems is essential for ensuring their reliability and stability. Therefore, metric monitoring systems are widely used and become an important infrastructure in online service systems. Engineers usually interact with metrics data by manually writing domain-specific language (DSL) queries to achieve various analysis objectives. However, writing these queries can be challenging and time-consuming, as it requires engineers to have high programming skills and understand the context of the system. In this paper, we focus on PromQL, which is the metric query DSL provided by the widely used metric monitoring system Prometheus. We aim to simplify metrics querying by enabling engineers to interact with metrics data in Prometheus through natural language, and we call this task text-to-PromQL. Building upon the insight, this paper proposes PromAssistant, a Large Language Model-based text-to-PromQL framework. PromAssistant first uses a knowledge graph to describe the complex context of an online service system. Then, through the synergistic reasoning of LLMs and the knowledge graph, PromAssistant transforms engineers' natural language questions into PromQL queries. To evaluate PromAssistant, we manually construct the first text-to-PromQL benchmark dataset which contains 280 metric query questions. The experiment results show that PromAssistant is effective in text-to-PromQL and outperforms baseline approaches. To the best of our knowledge, this paper is the first study of text-to-PromQL, and PromAssistant pioneered the DSL generation framework for metric querying and analysis.

Related papers

• Text-to-SPARQL Goes Beyond English: Multilingual Question Answering Over Knowledge Graphs through Human-Inspired Reasoning [51.203811759364925]
  mKGQAgent breaks down the task of converting natural language questions into SPARQL queries into modular, interpretable subtasks.<n> Evaluated on the DBpedia- and Corporate-based KGQA benchmarks within the Text2SPARQL challenge 2025, our approach took first place among the other participants.
  arXiv  Detail & Related papers  (2025-07-22T19:23:03Z)
• SPARQL Query Generation with LLMs: Measuring the Impact of Training Data Memorization and Knowledge Injection [81.78173888579941]
  Large Language Models (LLMs) are considered a well-suited method to increase the quality of the question-answering functionality.<n>LLMs are trained on web data, where researchers have no control over whether the benchmark or the knowledge graph was already included in the training data.<n>This paper introduces a novel method that evaluates the quality of LLMs by generating a SPARQL query from a natural-language question.
  arXiv  Detail & Related papers  (2025-07-18T12:28:08Z)
• A Survey of Context Engineering for Large Language Models [31.68644305980195]
  This survey introduces Context Engineering, a formal discipline that transcends simple prompt design.<n>We first examine the foundational components: context retrieval and generation, context processing and context management.<n>We then explore how these components are architecturally integrated to create sophisticated system implementations.
  arXiv  Detail & Related papers  (2025-07-17T17:50:36Z)
• Chatbot-Based Ontology Interaction Using Large Language Models and Domain-Specific Standards [41.19948826527649]
  Large Language Models (LLMs) are employed to enhance SPARQL query generation. System converts user inquiries into accurate SPARQL queries. Additional information from established domain-specific standards is integrated into the interface.
  arXiv  Detail & Related papers  (2024-07-22T11:58:36Z)
• DOCBENCH: A Benchmark for Evaluating LLM-based Document Reading Systems [99.17123445211115]
  We introduce DocBench, a benchmark to evaluate large language model (LLM)-based document reading systems. Our benchmark involves the recruitment of human annotators and the generation of synthetic questions. It includes 229 real documents and 1,102 questions, spanning across five different domains and four major types of questions.
  arXiv  Detail & Related papers  (2024-07-15T13:17:42Z)
• UQE: A Query Engine for Unstructured Databases [71.49289088592842]
  We investigate the potential of Large Language Models to enable unstructured data analytics. We propose a new Universal Query Engine (UQE) that directly interrogates and draws insights from unstructured data collections.
  arXiv  Detail & Related papers  (2024-06-23T06:58:55Z)
• Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph [83.90988015005934]
  Uncertainty quantification is a key element of machine learning applications. We introduce a novel benchmark that implements a collection of state-of-the-art UQ baselines. We conduct a large-scale empirical investigation of UQ and normalization techniques across eleven tasks, identifying the most effective approaches.
  arXiv  Detail & Related papers  (2024-06-21T20:06:31Z)
• STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases [93.96463520716759]
  We develop STARK, a large-scale Semi-structure retrieval benchmark on Textual and Knowledge Bases. Our benchmark covers three domains: product search, academic paper search, and queries in precision medicine. We design a novel pipeline to synthesize realistic user queries that integrate diverse relational information and complex textual properties.
  arXiv  Detail & Related papers  (2024-04-19T22:54:54Z)
• NL2KQL: From Natural Language to Kusto Query [1.7931930942711818]
  NL2KQL is an innovative framework that uses large language models (LLMs) to convert natural language queries (NLQs) to Kusto Query Language (KQL) queries.<n>To validate NL2KQL's performance, we utilize an array of online (based on query execution) and offline (based on query parsing) metrics.
  arXiv  Detail & Related papers  (2024-04-03T01:09:41Z)
• From Text to CQL: Bridging Natural Language and Corpus Search Engine [27.56738323943742]
  Corpus Query Language (CQL) is a critical tool for linguistic research and detailed analysis within text corpora. This paper presents the first text-to-CQL task that aims to automate the translation of natural language into CQL.
  arXiv  Detail & Related papers  (2024-02-21T12:11:28Z)
• PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models [72.57329554067195]
  ProxyQA is an innovative framework dedicated to assessing longtext generation. It comprises in-depth human-curated meta-questions spanning various domains, each accompanied by specific proxy-questions with pre-annotated answers. It assesses the generated content's quality through the evaluator's accuracy in addressing the proxy-questions.
  arXiv  Detail & Related papers  (2024-01-26T18:12:25Z)
• Text-to-OverpassQL: A Natural Language Interface for Complex Geodata Querying of OpenStreetMap [17.01783992725517]
  We present Text-to-OverpassQL, a task designed to facilitate a natural language interface for querying geodata from OpenStreetMap (OSM) Generating Overpass queries from natural language input serves multiple use-cases.
  arXiv  Detail & Related papers  (2023-08-30T14:33:25Z)
• A Machine Learning Approach to Classifying Construction Cost Documents into the International Construction Measurement Standard [0.0]
  We introduce the first automated models for classifying natural language descriptions provided in cost documents called "Bills of Quantities" We learn from a dataset of more than 50 thousand descriptions of items retrieved from 24 large infrastructure construction projects across the United Kingdom.
  arXiv  Detail & Related papers  (2022-10-24T11:35:53Z)
• UKP-SQUARE: An Online Platform for Question Answering Research [50.35348764297317]
  We present UKP-SQUARE, an online QA platform for researchers which allows users to query and analyze a large collection of modern Skills. UKP-SQUARE allows users to query and analyze a large collection of modern Skills via a user-friendly web interface and integrated tests.
  arXiv  Detail & Related papers  (2022-03-25T15:00:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.