Sigma: A dataset for text-to-code semantic parsing with statistical analysis

• URL: http://arxiv.org/abs/2504.04301v1
• Date: Sat, 05 Apr 2025 23:30:20 GMT
• Title: Sigma: A dataset for text-to-code semantic parsing with statistical analysis
• Abstract summary: SIGMA comprises 6000 questions with corresponding Python code labels, spanning across 160 databases.<n>Half of the questions involve query types, which return information in its original format.<n>The Python code labels in our dataset cover 4 types of query types and 40 types of statistical analysis patterns.
• Score: 9.781031411221395
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In the domain of semantic parsing, significant progress has been achieved in Text-to-SQL and question-answering tasks, both of which focus on extracting information from data sources in their native formats. However, the inherent constraints of their formal meaning representations, such as SQL programming language or basic logical forms, hinder their ability to analyze data from various perspectives, such as conducting statistical analyses. To address this limitation and inspire research in this field, we design SIGMA, a new dataset for Text-to-Code semantic parsing with statistical analysis. SIGMA comprises 6000 questions with corresponding Python code labels, spanning across 160 databases. Half of the questions involve query types, which return information in its original format, while the remaining 50% are statistical analysis questions, which perform statistical operations on the data. The Python code labels in our dataset cover 4 types of query types and 40 types of statistical analysis patterns. We evaluated the SIGMA dataset using three different baseline models: LGESQL, SmBoP, and SLSQL. The experimental results show that the LGESQL model with ELECTRA outperforms all other models, achieving 83.37% structure accuracy. In terms of execution accuracy, the SmBoP model, when combined with GraPPa and T5, reaches 76.38%.

Related papers

• text2ql: Multi-Target Natural Language Querying via a Language-Agnostic Intermediate Representation [0.9927219503124457]
  This paper presents text2ql, an open-source Python architecture for natural language interfaces.<n>It addresses three structural limitations of natural language interfaces to databases.<n>It is based on a language-agnostic Intermediate Representation (QueryIR) architecture.
  arXiv  Detail & Related papers  (2026-09-02T05:09:51Z)
• Spider 2.0-AIFunc: Extending Real-World Text-to-SQL to AI-Native SQL Workflows [54.49251107006261]
  We introduce Spider 2.0-AIFunc, a benchmark of 465 verified instances across 125 real-world databases covering six types of AI functions on the Snowflake platform.<n>We find that the strongest proprietary models reach 67-70% execution accuracy while the best open-source model achieves 58.1%.
  arXiv  Detail & Related papers  (2026-07-07T12:54:08Z)
• ProSPy: A Profiling-Driven SQL-Python Agentic Framework for Enterprise Text-to-SQL [51.98164069124653]
  ProSPy is a Profiling-driven framework for enterprise-scale Text-to-gnostic analysis.<n>ProSPy structures the reasoning process into four stages.<n>It first extracts fine-grained data evidence through automatic profiling.<n>It prunes large schemas into task-relevant contexts.
  arXiv  Detail & Related papers  (2026-06-04T08:13:05Z)
• From Questions to Queries: An AI-powered Multi-Agent Framework for Spatial Text-to-SQL [0.4499833362998488]
  Single-agent approaches often struggle with the semantic and syntactic complexities of spatial queries.<n>We propose a multi-agent framework designed to accurately translate natural language questions into spatial queries.<n>We evaluate our system using both the non-spatial KaggleDBQA benchmark and a new, comprehensive SpatialQA benchmark.
  arXiv  Detail & Related papers  (2025-10-23T22:58:17Z)
• Agentic LLMs for Question Answering over Tabular Data [6.310433217813068]
  Question Answering over Tabular Data (Table QA) presents unique challenges due to the diverse structure, size, and data types of real-world tables.<n>This paper details our methodology, experimental results, and alternative approaches, providing insights into the strengths and limitations of Table QA.
  arXiv  Detail & Related papers  (2025-09-11T08:12:38Z)
• RAISE: Reasoning Agent for Interactive SQL Exploration [47.77323087050061]
  We propose a novel framework that unifies schema linking, query generation, and iterative refinement within a single, end-to-end component.<n>Our method emulates how humans answer questions when working with unfamiliar databases.
  arXiv  Detail & Related papers  (2025-06-02T03:07:08Z)
• FABLE: A Novel Data-Flow Analysis Benchmark on Procedural Text for Large Language Model Evaluation [5.866040886735852]
  We introduce FABLE, a benchmark designed to assess large language models' understanding of data flow using structured, procedural text.<n>We evaluate three types of LLMs: a reasoning-focused model (DeepSeek-R1 8B), a general-purpose model (LLaMA 3.1 8B), and a code-specific model (Granite Code 8B)<n>Results show that the reasoning model achieves higher accuracy, but at the cost of over 20 times slower inference compared to the other models.
  arXiv  Detail & Related papers  (2025-05-30T06:32:34Z)
• LogicCat: A Chain-of-Thought Text-to-SQL Benchmark for Multi-Domain Reasoning Challenges [13.400649304012179]
  The dataset consists of 4,038 English questions, each paired with a uniquesql query and accompanied by 12,114 reasoning annotations, spanning 45 databases across diverse domains.<n>LogicCat substantially increases the difficulty for state-of-the-art models, with the highest execution accuracy reaching only 14.96%.<n> Benchmarking leading public methods on Spider and BIRD further underscores the challenges presented by LogicCat, highlighting the significant opportunities for advancing research in robust, reasoning-driven text-to-funk systems.
  arXiv  Detail & Related papers  (2025-05-24T15:23:43Z)
• CoddLLM: Empowering Large Language Models for Data Analytics [38.23203246023766]
  Large Language Models (LLMs) have the potential to revolutionize data analytics.<n>We unveil a new data recipe for post-Turbo synthesiss.<n>We posttrain a new foundation model, named CoddLLM, based on MistralNeMo-12B.
  arXiv  Detail & Related papers  (2025-02-01T06:03:55Z)
• Text2Insight: Transform natural language text into insights seamlessly using multi-model architecture [0.0]
  Text2Insight is an innovative solution that delivers customized data analysis and visualizations based on user-defined natural language requirements. To enhance analysis capabilities, the system integrates a question-answering model and a predictive model using the BERT framework. Performance evaluation of Text2Insight demonstrates its effectiveness, achieving high accuracy (99%), precision (100%), recall (99%), and F1-score (99%), with a BLEU score of 0.5.
  arXiv  Detail & Related papers  (2024-12-27T16:17:22Z)
• RSL-SQL: Robust Schema Linking in Text-to-SQL Generation [51.00761167842468]
  We propose a novel framework called RSL- that combines bidirectional schema linking, contextual information augmentation, binary selection strategy, and multi-turn self-correction. benchmarks demonstrate that our approach achieves SOTA execution accuracy among open-source solutions, with 67.2% on BIRD and 87.9% on GPT-4ocorrection. Our approach outperforms a series of GPT-4 based Text-to-Seek systems when adopting DeepSeek (much cheaper) with same intact prompts.
  arXiv  Detail & Related papers  (2024-10-31T16:22:26Z)
• KET-QA: A Dataset for Knowledge Enhanced Table Question Answering [63.56707527868466]
  We propose to use a knowledge base (KB) as the external knowledge source for TableQA. Every question requires the integration of information from both the table and the sub-graph to be answered. We design a retriever-reasoner structured pipeline model to extract pertinent information from the vast knowledge sub-graph.
  arXiv  Detail & Related papers  (2024-05-13T18:26:32Z)
• Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data [89.2410799619405]
  We introduce the Quantitative Reasoning with Data benchmark to evaluate Large Language Models' capability in statistical and causal reasoning with real-world data. The benchmark comprises a dataset of 411 questions accompanied by data sheets from textbooks, online learning materials, and academic papers. To compare models' quantitative reasoning abilities on data and text, we enrich the benchmark with an auxiliary set of 290 text-only questions, namely QRText.
  arXiv  Detail & Related papers  (2024-02-27T16:15:03Z)
• Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queries [67.0083902913112]
  We develop the Text2Analysis benchmark, incorporating advanced analysis tasks. We also develop five innovative and effective annotation methods. We evaluate five state-of-the-art models using three different metrics.
  arXiv  Detail & Related papers  (2023-12-21T08:50:41Z)
• #InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Models [83.80695137230134]
  In this work, we propose InsTag, an open-set fine-grained tagger, to tag samples within SFT datasets based on semantics and intentions. We analyze popular open-sourced SFT datasets and find that the model ability grows with more diverse and complex data. The resulting models, TagLM, outperform open-source models based on considerably larger SFT data evaluated by MT-Bench.
  arXiv  Detail & Related papers  (2023-08-14T11:16:28Z)
• AnaMeta: A Table Understanding Dataset of Field Metadata Knowledge Shared by Multi-dimensional Data Analysis Tasks [30.63485722768747]
  We introduce the AnaMeta dataset, a collection of 467k tables with derived supervision labels for four types of commonly used field metadata. We evaluate a wide range of models for inferring metadata as the benchmark. We propose four interfaces for incorporating field metadata into downstream analysis tasks.
  arXiv  Detail & Related papers  (2022-09-02T11:01:45Z)
• AutoQA: From Databases To QA Semantic Parsers With Only Synthetic Training Data [6.522647000820583]
  AutoQA generates semantics that answer questions on databases with no manual effort. It uses automatic paraphrasing combined with template-based attribute parsing to find alternative expressions of an expression. It achieves 69.8% answer accuracy, 16.4% higher than the state-of-the-art zero-shot models and only 5.2% lower than the same model trained with human data.
  arXiv  Detail & Related papers  (2020-10-09T21:06:57Z)
• TableQA: a Large-Scale Chinese Text-to-SQL Dataset for Table-Aware SQL Generation [3.4161707164978137]
  Existing NL2 datasets assume that condition values should appear exactly in natural language questions and the queries are answerable given the table. We present TableQA, a large-scale cross-domain Natural Language dataset in Chinese language consisting 64,891 questions and 20,311 uniquesql queries on 6,000 tables. Two table-aware approaches are proposed to alleviate the problem, the end-to-end approaches obtains 51.3% and 47.4% accuracy on the condition value and logic form tasks, with improvement of 4.7% and 3.4% respectively.
  arXiv  Detail & Related papers  (2020-06-10T03:49:08Z)
• Multi-layer Optimizations for End-to-End Data Analytics [71.05611866288196]
  We introduce Iterative Functional Aggregate Queries (IFAQ), a framework that realizes an alternative approach. IFAQ treats the feature extraction query and the learning task as one program given in the IFAQ's domain-specific language. We show that a Scala implementation of IFAQ can outperform mlpack, Scikit, and specialization by several orders of magnitude for linear regression and regression tree models over several relational datasets.
  arXiv  Detail & Related papers  (2020-01-10T16:14:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.