Agent Bain vs. Agent McKinsey: A New Text-to-SQL Benchmark for the Business Domain

• URL: http://arxiv.org/abs/2510.07309v2
• Date: Thu, 09 Oct 2025 02:27:56 GMT
• Title: Agent Bain vs. Agent McKinsey: A New Text-to-SQL Benchmark for the Business Domain
• Abstract summary: We introduce CORGI, a new benchmark specifically designed for real-world business contexts.<n>It provides questions across four categories of business queries: descriptive, explanatory, predictive, and recommendational.<n>We find that CORGI performance drops on high-level questions, struggling to make accurate predictions and offer actionable plans.
• Score: 10.89800905114692
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In the business domain, where data-driven decision making is crucial, text-to-SQL is fundamental for easy natural language access to structured data. While recent LLMs have achieved strong performance in code generation, existing text-to-SQL benchmarks remain focused on factual retrieval of past records. We introduce CORGI, a new benchmark specifically designed for real-world business contexts. CORGI is composed of synthetic databases inspired by enterprises such as Doordash, Airbnb, and Lululemon. It provides questions across four increasingly complex categories of business queries: descriptive, explanatory, predictive, and recommendational. This challenge calls for causal reasoning, temporal forecasting, and strategic recommendation, reflecting multi-level and multi-step agentic intelligence. We find that LLM performance drops on high-level questions, struggling to make accurate predictions and offer actionable plans. Based on execution success rate, the CORGI benchmark is about 21% more difficult than the BIRD benchmark. This highlights the gap between popular LLMs and the need for real-world business intelligence. We release a public dataset and evaluation framework, and a website for public submissions.

Related papers

• Rethinking Agentic Workflows: Evaluating Inference-Based Test-Time Scaling Strategies in Text2SQL Tasks [21.891522433628893]
  Large language models (LLMs) are increasingly powering Text-to- (Text2) systems, enabling non-expert users to query industrial databases using natural language.<n>While test-time scaling strategies have shown promise in LLM-based solutions, their effectiveness in real-world applications, especially with the latest reasoning models, remains uncertain.<n>This work sheds light on the practical trade-offs between accuracy, efficiency, and complexity when deploying Text2 systems.
  arXiv  Detail & Related papers  (2025-10-13T01:29:54Z)
• BenchPress: A Human-in-the-Loop Annotation System for Rapid Text-to-SQL Benchmark Curation [2.765551649887593]
  BenchPress is a human-in-the-loop system designed to accelerate the creation of domain-specific text-to-sql benchmarks.<n>It uses retrieval-augmented generation (LLMs) and LLMs to propose multiple natural language descriptions.<n>Human experts then select, rank, or edit these drafts to ensure accuracy and domain alignment.
  arXiv  Detail & Related papers  (2025-10-11T23:50:12Z)
• RAISE: Reasoning Agent for Interactive SQL Exploration [47.77323087050061]
  We propose a novel framework that unifies schema linking, query generation, and iterative refinement within a single, end-to-end component.<n>Our method emulates how humans answer questions when working with unfamiliar databases.
  arXiv  Detail & Related papers  (2025-06-02T03:07:08Z)
• Bridging the Gap: Enabling Natural Language Queries for NoSQL Databases through Text-to-NoSQL Translation [25.638927795540454]
  We introduce the Text-to-No task, which aims to convert natural language queries into accessible queries.<n>To promote research in this area, we released a large-scale and open-source dataset for this task, named TEND (short interfaces for Text-to-No dataset)<n>We also designed a SLM (Small Language Model)-assisted and RAG (Retrieval-augmented Generation)-assisted multi-step framework called SMART, which is specifically designed for Text-to-No conversion.
  arXiv  Detail & Related papers  (2025-02-16T17:01:48Z)
• Towards Evaluating Large Language Models for Graph Query Generation [49.49881799107061]
  Large Language Models (LLMs) are revolutionizing the landscape of Generative Artificial Intelligence (GenAI) This paper presents a comparative study addressing the challenge of generating queries a powerful language for interacting with graph databases using open-access LLMs. Our empirical analysis of query generation accuracy reveals that Claude Sonnet 3.5 outperforms its counterparts in this specific domain.
  arXiv  Detail & Related papers  (2024-11-13T09:11:56Z)
• BEAVER: An Enterprise Benchmark for Text-to-SQL [5.984348784381797]
  Existing text-to-the-shelf benchmarks have largely been constructed from web tables with human-generated question- pairs.<n>We introduce a new dataset BEA, the first enterprise text-to-the-shelf benchmark sourced from real private enterprise data warehouses.
  arXiv  Detail & Related papers  (2024-09-03T16:37:45Z)
• Relational Database Augmented Large Language Model [59.38841050766026]
  Large language models (LLMs) excel in many natural language processing (NLP) tasks. They can only incorporate new knowledge through training or supervised fine-tuning processes. This precise, up-to-date, and private information is typically stored in relational databases.
  arXiv  Detail & Related papers  (2024-07-21T06:19:10Z)
• RB-SQL: A Retrieval-based LLM Framework for Text-to-SQL [48.516004807486745]
  Large language models (LLMs) with in-context learning have significantly improved the performance of text-to- task. We propose RB-, a novel retrieval-based framework for in-context prompt engineering. Experiment results demonstrate that our model achieves better performance than several competitive baselines on public datasets BIRD and Spider.
  arXiv  Detail & Related papers  (2024-07-11T08:19:58Z)
• Automating Pharmacovigilance Evidence Generation: Using Large Language Models to Produce Context-Aware SQL [0.0]
  We utilize OpenAI's GPT-4 model within a retrieval-augmented generation (RAG) framework. Business context document is enriched with a business context document, to transform NLQs into Structured Query Language queries. Performance achieved a maximum of 85% when high complexity queries are excluded.
  arXiv  Detail & Related papers  (2024-06-15T17:07:31Z)
• Enhancing Text-to-SQL Translation for Financial System Design [5.248014305403357]
  We consider Large Language Models (LLMs), which have achieved state of the art for various NLP tasks. We propose two novel metrics that were designed to adequately measure the similarity between relational queries.
  arXiv  Detail & Related papers  (2023-12-22T14:34:19Z)
• Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation [76.76046657162306]
  Large language models (LLMs) have emerged as a new paradigm for Text-to- task. Large language models (LLMs) have emerged as a new paradigm for Text-to- task.
  arXiv  Detail & Related papers  (2023-08-29T14:59:54Z)
• SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (extended) [53.95151604061761]
  This paper introduces the framework for enhancing Text-to- filtering using large language models (LLMs) With few-shot prompting, we explore the effectiveness of consistency decoding with execution-based error analyses. With instruction fine-tuning, we delve deep in understanding the critical paradigms that influence the performance of tuned LLMs.
  arXiv  Detail & Related papers  (2023-05-26T21:39:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.