DIRT: Database-Integrated Random Testing

• URL: http://arxiv.org/abs/2604.16373v1
• Date: Mon, 23 Mar 2026 18:01:00 GMT
• Title: DIRT: Database-Integrated Random Testing
• Abstract summary: We present DIRT, a paradigm designed specifically for testing databases during development.<n>We evaluate DIRT on Turso, an actively developed rather than testing O engine.
• Score: 1.1024591739346294
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Database management systems (DBMSs) are notoriously complex, making them difficult to test effectively, especially during early development when many features are incomplete. Traditional testing tools like SQLancer and SQLSmith are highly effective for mature databases, but they struggle with high false positive rates and low actionability when applied to evolving systems. We present DIRT, a paradigm designed specifically for testing databases during development, which integrates a testing framework directly into the DBMS, enabling the random testing process to evolve in tandem with the system and reducing false positives by construction. We introduce generation actions, an abstraction for allowing database developers rather than testing experts to specify correctness properties. We evaluate DIRT on Turso, an actively developed SQLite-compatible OLTP engine, and show that it finds 23 unique, confirmed bugs--significantly outperforming off-the-shelf SQLancer variants in terms of true positive rate and usefulness of bug reports. Our results demonstrate that embedding testing infrastructure within the DBMS can dramatically improve its effectiveness and usability during development.

Related papers

• SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents [63.65682461568621]
  We introduce SWE-Gate, a repository-level benchmark for software engineering agents that explicitly evaluates review constraint compliance alongside functional correctness.<n>SWE-Gate derives review constraints from real pull request review comments and synthesizes repository-level repair instances around these constraints.<n>Each instance provides separate functional and constraint tests, together with non-compliant and gold patches, enabling explicit separation between issue resolution capability and review constraint compliance.
  arXiv  Detail & Related papers  (2026-09-03T17:53:34Z)
• TTHE: Test-Time Harness Evolution [50.26245555541721]
  Existing approaches optimize such harnesses before deployment, searching training or development data for a fixed agent workflow that is then frozen at test time.<n>We ask whether the harness can instead be optimized during evaluation itself, using only the unlabeled execution traces the agent produces on the test inputs.<n>During evaluation, TTHE maintains a population of candidate harnesses and refines them through an agentic proposer that reasons over their execution traces.<n>A judge then commits an improved harness from execution-derived proxy signals, and the selected program persists to govern subsequent inputs.
  arXiv  Detail & Related papers  (2026-07-09T05:53:39Z)
• From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models [73.72877445629383]
  Interpretability-Guided Data Selection (IGDS) is a framework that first identifies causal task features through frequency recall and interventional filtering.<n>We validate IGDS on mathematical reasoning, summarization, and translation tasks within Gemma-2, LLaMA-3.1, and Qwen3 models.
  arXiv  Detail & Related papers  (2026-04-28T03:16:24Z)
• Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysis [68.28714988482703]
  Process Reward Models (PRMs) have achieved remarkable success in augmenting the reasoning capabilities of Large Language Models (LLMs)<n>We present a study revealing that general-domain PRMs struggle to supervise data analysis agents.<n>We introduce DataPRM, a novel environment-aware generative process reward model.
  arXiv  Detail & Related papers  (2026-04-27T09:00:30Z)
• BugForge: Constructing and Utilizing DBMS Bug Repository to Enhance DBMS Testing [26.533612973604804]
  We propose BugForge, a framework that constructs standardized bug repositories.<n>BugForge generates high-quality test cases to enhance testing.<n>Based on the repository, BugForge uncovered 35 previously unknown bugs with 22 confirmed by developers.
  arXiv  Detail & Related papers  (2026-04-03T13:13:53Z)
• DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems [48.971606069204825]
  DoVer is an intervention-driven debug framework for large language model (LLM)-based multi-agent systems.<n>It augments hypothesis generation with active verification through targeted interventions.<n>DoVer flips 18-28% of failed trials into successes, achieves up to 16% milestone progress, and validates or refutes 30-60% of failure hypotheses.
  arXiv  Detail & Related papers  (2025-12-07T09:23:48Z)
• SEED: Enhancing Text-to-SQL Performance and Practical Usability Through Automatic Evidence Generation [8.638974393417929]
  State-of-the-art text-to-sql studies rely on the BIRD dataset, which assumes that evidence is provided along with questions.<n>We propose SEED, an approach that automatically generates evidence to improve performance and practical usability in real-world scenarios.
  arXiv  Detail & Related papers  (2025-06-09T04:44:31Z)
• RAISE: Reasoning Agent for Interactive SQL Exploration [47.77323087050061]
  We propose a novel framework that unifies schema linking, query generation, and iterative refinement within a single, end-to-end component.<n>Our method emulates how humans answer questions when working with unfamiliar databases.
  arXiv  Detail & Related papers  (2025-06-02T03:07:08Z)
• Constant Optimization Driven Database System Testing [6.246028398098516]
  Logic bugs are bugs that can cause database management systems (DBMSs) to silently produce incorrect results for given queries.<n>We propose Constant-Optimization-Driven Database Testing (CODDTest) as a novel approach for detecting logic bugs in databases.
  arXiv  Detail & Related papers  (2025-01-20T03:32:55Z)
• Enhanced Differential Testing in Emerging Database Systems [11.164715689334912]
  We propose enhanced differential testing to uncover bugs in emerging database systems.<n>Many emerging database systems are conceptually extensions of relational database systems.<n>We find 57 unknown bugs, of which 17 were logic bugs and 40 were internal errors.
  arXiv  Detail & Related papers  (2025-01-02T12:52:58Z)
• Finding Logic Bugs in Spatial Database Engines via Affine Equivalent Inputs [6.291508085458252]
  Spatial Database Management Systems (SDBMSs) aim to store, manipulate, and retrieve spatial data. The presence of logic bugs in SDBMSs can lead to incorrect results. Detecting logic bugs in SDBMSs is challenging due to the lack of ground truth for identifying incorrect results.
  arXiv  Detail & Related papers  (2024-10-16T12:18:16Z)
• Automating Dataset Updates Towards Reliable and Timely Evaluation of Large Language Models [81.27391252152199]
  Large language models (LLMs) have achieved impressive performance across various natural language benchmarks. We propose to automate dataset updating and provide systematic analysis regarding its effectiveness. There are two updating strategies: 1) mimicking strategy to generate similar samples based on original data, and 2) extending strategy that further expands existing samples.
  arXiv  Detail & Related papers  (2024-02-19T07:15:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.