SANE Schema-aware Natural-language Evaluation of Biological Data

• URL: http://arxiv.org/abs/2606.04500v1
• Date: Wed, 03 Jun 2026 06:31:28 GMT
• Title: SANE Schema-aware Natural-language Evaluation of Biological Data
• Abstract summary: We present SANE-Aware Natural Evaluation, a novel paradigm for domain-specific text-to-language evaluation.<n>We show that, under constrained schemas with structured prompting and guardrails, accurate query generation is achievable without any model training or fine-tuning.
• Score: 0.14337588659482522
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: High-throughput microscopy generates large, structured datasets capturing cellular responses to pharmacological perturbations, but accessing these datasets typically requires SQL expertise. Large language models offer a natural-language alternative, yet their tendency to hallucinate raises concerns about result reliability . We present SANE Schema-Aware Natural-language Evaluation, a novel paradigm for domain-specific text-to-SQL evaluation: schema-grounded, automatically generated benchmarks tied to real and specific experimental structure. SANE makes evaluation more scalable, systematic, and reproducible. Using SANE, we evaluate a few-shot large language model and show that, under constrained schemas with structured prompting and guardrails, accurate query generation is achievable without any model training or fine-tuning. Most failures stem from ambiguous or underspecified inputs and manifest as overly cautious clarification requests or answers to queries that should first be disambiguated, rather than incorrect SQL generation. These results indicate that few-shot large language models can provide reliable database access in well-defined domains when combined with schema-aware prompting.

Related papers

• Agent-Agnostic Evaluation of SQL Accuracy in Production Text-to-SQL Systems [2.051598597332424]
  Text-to-native (T2native) evaluation in production environments poses fundamental challenges that existing benchmarks do not address.<n>We present STEF, a production- evaluation system that operates exclusively on natural language inputs.<n>STEF produces an interpretable 0 to 100 accuracy score via a composite metric that encompasses filter alignment, semantic verdict, and confidence of the evaluator.
  arXiv  Detail & Related papers  (2026-04-30T15:59:28Z)
• IESR:Efficient MCTS-Based Modular Reasoning for Text-to-SQL with Large Language Models [10.758655501692793]
  We propose a framework named IESR(Information Enhanced Structured Reasoning) for lightweight large language models.<n>We show that IESR achieves state-of-the-art performance on the complex reasoning benchmark LogicCat and the Archer dataset.<n>Our analysis reveals that current coder models exhibit notable biases and deficiencies in physical knowledge, mathematical computation, and common-sense reasoning.
  arXiv  Detail & Related papers  (2026-02-05T07:10:45Z)
• OsmT: Bridging OpenStreetMap Queries and Natural Language with Open-source Tag-aware Language Models [30.29148893930439]
  We present OsmT, an open-source tag-aware language model for bridging natural language and structured query languages.<n>We introduce a Tag Retrieval Augmentation (TRA) mechanism that incorporates contextually relevant tag knowledge into the generation process.<n>We also define a reverse task, OverpassQL-to-Text, which translates structured queries into natural language explanations.
  arXiv  Detail & Related papers  (2025-12-04T12:24:36Z)
• From Queries to Insights: Agentic LLM Pipelines for Spatio-Temporal Text-to-SQL [8.496933324334167]
  We present a naive text-to-Act baseline (Rellama-sqlcoder-8b) with orchestration by a Mistral-based Rellama-sqlcoder-8b.<n>We evaluate on 35 natural-language queries over the NYC and Tokyo check-in, covering spatial, temporal multi-dataset reasoning.<n>The agent achieves substantially higher accuracy than the dataset 91.4% vs. 28.6% and enhances usability through maps, and plots structured natural-language summaries.
  arXiv  Detail & Related papers  (2025-10-29T22:18:57Z)
• Exploring Database Normalization Effects on SQL Generation [0.9256004093538199]
  normalization is a critical yet often overlooked factor in natural language tosql (NL2) systems.<n>We evaluate eight leading large language models on synthetic and real-world datasets with varied normalization levels.<n>Our findings suggest that the optimal schema design for NL2 applications depends on the types of queries to be supported.
  arXiv  Detail & Related papers  (2025-10-02T13:11:30Z)
• Who Gets Cited Most? Benchmarking Long-Context Language Models on Scientific Articles [81.89404347890662]
  SciTrek is a novel question-answering benchmark designed to evaluate the long-context reasoning capabilities of large language models (LLMs) using scientific articles.<n>Our analysis reveals systematic shortcomings in models' abilities to perform basic numerical operations and accurately locate specific information in long contexts.
  arXiv  Detail & Related papers  (2025-09-25T11:36:09Z)
• E-SQL: Direct Schema Linking via Question Enrichment in Text-to-SQL [1.187832944550453]
  We introduce E-Seek, a novel pipeline specifically designed to address these challenges through direct schema linking and candidate predicate augmentation.<n>E-Seek enhances the natural language query by incorporating relevant database items (i.e., tables, columns, and values) and conditions directly into the question andsql construction plan, bridging the gap between the query and the database structure.<n> Comprehensive evaluations illustrate that E-Seek achieves competitive performance, particularly excelling in complex queries with a 66.29% execution accuracy on the test set.
  arXiv  Detail & Related papers  (2024-09-25T09:02:48Z)
• UQE: A Query Engine for Unstructured Databases [71.49289088592842]
  We investigate the potential of Large Language Models to enable unstructured data analytics. We propose a new Universal Query Engine (UQE) that directly interrogates and draws insights from unstructured data collections.
  arXiv  Detail & Related papers  (2024-06-23T06:58:55Z)
• Bring Your Own Data! Self-Supervised Evaluation for Large Language Models [52.15056231665816]
  We propose a framework for self-supervised evaluation of Large Language Models (LLMs) We demonstrate self-supervised evaluation strategies for measuring closed-book knowledge, toxicity, and long-range context dependence. We find strong correlations between self-supervised and human-supervised evaluations.
  arXiv  Detail & Related papers  (2023-06-23T17:59:09Z)
• Interpretable Medical Diagnostics with Structured Data Extraction by Large Language Models [59.89454513692417]
  Tabular data is often hidden in text, particularly in medical diagnostic reports. We propose a novel, simple, and effective methodology for extracting structured tabular data from textual medical reports, called TEMED-LLM. We demonstrate that our approach significantly outperforms state-of-the-art text classification models in medical diagnostics.
  arXiv  Detail & Related papers  (2023-06-08T09:12:28Z)
• Proton: Probing Schema Linking Information from Pre-trained Language Models for Text-to-SQL Parsing [66.55478402233399]
  We propose a framework to elicit relational structures via a probing procedure based on Poincar'e distance metric. Compared with commonly-used rule-based methods for schema linking, we found that probing relations can robustly capture semantic correspondences. Our framework sets new state-of-the-art performance on three benchmarks.
  arXiv  Detail & Related papers  (2022-06-28T14:05:25Z)
• Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training [86.91380874390778]
  We present Generation-Augmented Pre-training (GAP), that jointly learns representations of natural language utterances and table schemas by leveraging generation models to generate pre-train data. Based on experimental results, neural semantics that leverage GAP MODEL obtain new state-of-the-art results on both SPIDER and CRITERIA-TO-generative benchmarks.
  arXiv  Detail & Related papers  (2020-12-18T15:53:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.