Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models

• URL: http://arxiv.org/abs/2604.25149v1
• Date: Tue, 28 Apr 2026 02:53:23 GMT
• Title: Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models
• Abstract summary: LLMs deployed for natural-language querying of analytical databases suffer from two intertwined failures.<n>We benchmark three frontier LLMs on 100 natural-language questions over the Cleaned Contoso Retail dataset in ClickHouse.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLMs deployed for natural-language querying of analytical databases suffer from two intertwined failures - incorrect answers and confident hallucinations - both rooted in the same cause: the model is forced to infer business semantics that the schema does not encode. We test whether supplying those semantics as context closes the gap. We benchmark three frontier LLMs (Claude Opus 4.7, Claude Sonnet 4.6, GPT-5.4) on 100 natural-language questions over the Cleaned Contoso Retail Dataset in ClickHouse, using a paired single-shot protocol. Each model is evaluated twice: once given only the warehouse schema, and once given the schema plus a 4 KB hand-authored markdown document describing the dataset's measures, conventions, and disambiguation rules. Adding the document improves accuracy by +17 to +23 percentage points across all three models. With it, the three models are statistically indistinguishable (67.7-68.7%); without it, they are also indistinguishable (45.5-50.5%). Every cross-cluster comparison is significant at p < 0.01. The presence of the semantic-layer document accounts for essentially all of the significant variance; model choice within tier does not. We interpret this as a structural result: explicit business semantics suppress the dominant class of text-to-SQL errors not by making the model more capable, but by changing what the model is being asked to do.

Related papers

• SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics [15.233613140316791]
  SemVerBench is the first benchmark of version-constraint resolution semantics across three ecosystems.<n>We evaluate six frontier models and find no statistically significant self-favoritism.<n>Because the task is verifiable and a free, 100%-correct resolver exists, tool delegation reaches 100%.
  arXiv  Detail & Related papers  (2026-09-10T07:34:44Z)
• Public-Sharing Labels and Verbatim Field Egress in an MCP-to-A2A Agent Configuration: A Controlled Multi-Model Study [0.0]
  Safety properties assessed separately for Model Context Protocol tool use and Agent2Agent (A2A) delegation need not describe behavior when one agent uses both.<n>We measure one such behavior in a single controlled MCP-to-A2A configuration.
  arXiv  Detail & Related papers  (2026-09-01T16:24:11Z)
• Decoupling Perception from Description: Computation-Grounded Representation Alignment between Multivariate Time Series and Language [16.343638992747533]
  Training multimodal models to align time series with language runs into a self-supervision trap.<n>The data can never teach more than the labeler already knows.<n>We resolve this by decoupling perception from description.
  arXiv  Detail & Related papers  (2026-08-05T13:57:08Z)
• Memory Architectures for Multi-Turn Text-to-SQL: A Benchmark and Empirical Study [0.0]
  Multi-turn Text-to-BIRD is central to enterprise analytics yet remains predominantly evaluated in single-turn settings.<n>We evaluate five models -- GPT-5 mini, GPT-5.2, Claude Sonnet 4.5, Sonnet 4.6, and Opus 4.6 -- across five memory conditions.<n>All Claude models are evaluated with extended thinking enabled to maintain parity with GPT reasoning models.
  arXiv  Detail & Related papers  (2026-05-25T23:52:15Z)
• Articulate but Wrong: Self-Review Failures in LLM-Based Code Modernization [1.0164694825170502]
  Large language model (LLM) agents are increasingly used to migrate legacy code to modern stacks.<n>We run 1,980 real modernization calls across 11 production LLMs from 7 distinct families on a balanced 60-snippet legacy-Python-2 corpus.<n>We find that semantic-preservation drift is prevalent and sharply separable from a cleanly-controlled baseline.
  arXiv  Detail & Related papers  (2026-05-20T05:00:31Z)
• BibTeX Citation Hallucinations in Scientific Publishing Agents: Evaluation and Mitigation [34.429649156970015]
  Prior evaluations tested base models without search, which does not reflect current practice.<n>Three search-enabled frontier models generate Bib entries scored on nine fields and a six-way error taxonomy.<n>Overall accuracy is 83.6%, but only 50.9% of entries are fully correct.
  arXiv  Detail & Related papers  (2026-04-03T16:30:58Z)
• ACAR: Adaptive Complexity Routing for Multi-Model Ensembles with Auditable Decision Traces [3.151184728006369]
  We present ACAR, a measurement framework for studying multi-model orchestration under auditable conditions.<n>ACAR uses self-consistency variance (sigma) computed from N=3 probe samples to route tasks across single-model, two-model, and three-model execution modes.<n>We evaluate ACAR on 1,510 tasks spanning four benchmarks, producing more than 7,550 auditable runs.
  arXiv  Detail & Related papers  (2026-02-06T23:27:17Z)
• Regularization Through Reasoning: Systematic Improvements in Language Model Classification via Explanation-Enhanced Fine-Tuning [2.247737938202007]
  We evaluate whether attaching brief explanations to each label during fine-tuning yields better models.<n>We replace human-written explanations with text that is syntactically incoherent yet vocabulary-aligned with the originals.<n>The effect persists across datasets and training seeds, indicating that gains arise less from meaning than from structure.
  arXiv  Detail & Related papers  (2025-11-03T20:25:42Z)
• Evaluating NL2SQL via SQL2NL [45.88028371034407]
  New framework generates semantically equivalent, lexically diverse queries.<n>State-of-the-art models are far more brittle than standard benchmarks suggest.
  arXiv  Detail & Related papers  (2025-09-04T21:03:59Z)
• Hey, That's My Data! Label-Only Dataset Inference in Large Language Models [63.35066172530291]
  CatShift is a label-only dataset-inference framework.<n>It capitalizes on catastrophic forgetting: the tendency of an LLM to overwrite previously learned knowledge when exposed to new data.
  arXiv  Detail & Related papers  (2025-06-06T13:02:59Z)
• Fact-Consistency Evaluation of Text-to-SQL Generation for Business Intelligence Using Exaone 3.5 [0.0]
  Large Language Models (LLMs) have shown promise in enabling natural language interfaces for structured data querying through text-to-generation.<n>We propose a Fact-Consistency Evaluation Framework for assessing the semantic accuracy of LLM-generatedsql outputs using Exaone 3.5.<n>We construct a domain-specific benchmark comprising 219 natural language business questions, derived from actual sales data in LG Electronics' internal BigQuery environment.<n>We evaluate model performance using answer accuracy, execution success rate, semantic error rate, and non-response rate.
  arXiv  Detail & Related papers  (2025-04-30T14:42:18Z)
• Idiosyncrasies in Large Language Models [54.26923012617675]
  We unveil and study idiosyncrasies in Large Language Models (LLMs)<n>We find that fine-tuning text embedding models on LLM-generated texts yields excellent classification accuracy.<n>We leverage LLM as judges to generate detailed, open-ended descriptions of each model's idiosyncrasies.
  arXiv  Detail & Related papers  (2025-02-17T18:59:02Z)
• Beyond Traditional Benchmarks: Analyzing Behaviors of Open LLMs on Data-to-Text Generation [0.0]
  We analyze the behaviors of open large language models (LLMs) on the task of data-to-text (D2T) generation. We find that open LLMs can generate fluent and coherent texts in zero-shot settings from data in common formats collected with Quintd.
  arXiv  Detail & Related papers  (2024-01-18T18:15:46Z)
• Evaluating the Factual Consistency of Large Language Models Through News Summarization [97.04685401448499]
  We propose a new benchmark called FIB(Factual Inconsistency Benchmark) that focuses on the task of summarization. For factually consistent summaries, we use human-written reference summaries that we manually verify as factually consistent. For factually inconsistent summaries, we generate summaries from a suite of summarization models that we have manually annotated as factually inconsistent.
  arXiv  Detail & Related papers  (2022-11-15T18:50:34Z)
• ASDOT: Any-Shot Data-to-Text Generation with Pretrained Language Models [82.63962107729994]
  Any-Shot Data-to-Text (ASDOT) is a new approach flexibly applicable to diverse settings. It consists of two steps, data disambiguation and sentence fusion. Experimental results show that ASDOT consistently achieves significant improvement over baselines.
  arXiv  Detail & Related papers  (2022-10-09T19:17:43Z)
• TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task [80.38130122127882]
  TACRED is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE) In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled.
  arXiv  Detail & Related papers  (2020-04-30T15:07:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.