Enhanced Differential Testing in Emerging Database Systems

• URL: http://arxiv.org/abs/2501.01236v1
• Date: Thu, 02 Jan 2025 12:52:58 GMT
• Title: Enhanced Differential Testing in Emerging Database Systems
• Authors: Yuancheng Jiang, Jianing Wang, Chuqi Zhang, Roland Yap, Zhenkai Liang, Manuel Rigger,
• Abstract summary: We propose enhanced differential testing to uncover bugs in emerging database systems. Many emerging database systems are conceptually extensions of relational database systems. We find 57 unknown bugs, of which 17 were logic bugs and 40 were internal errors.
• Score: 11.164715689334912
• License:
• Abstract: In recent years, a plethora of database management systems have surfaced to meet the demands of various scenarios. Emerging database systems, such as time-series and streaming database systems, are tailored to specific use cases requiring enhanced functionality and performance. However, as they are typically less mature, there can be bugs that either cause incorrect results or errors impacting reliability. To tackle this, we propose enhanced differential testing to uncover various bugs in emerging SQL-like database systems. The challenge is how to deal with differences of these emerging databases. Our insight is that many emerging database systems are conceptually extensions of relational database systems, making it possible to reveal logic bugs leveraging existing relational, known-reliable database systems. However, due to inevitable syntax or semantics gaps, it remains challenging to scale differential testing to various emerging database systems. We enhance differential testing for emerging database systems with three steps: (i) identifying shared clauses; (ii) extending shared clauses via mapping new features back to existing clauses of relational database systems; and (iii) generating differential inputs using extended shared clauses. We implemented our approach in a tool called SQLxDiff and applied it to four popular emerging database systems. In total, we found 57 unknown bugs, of which 17 were logic bugs and 40 were internal errors. Overall, vendors fixed 50 bugs and confirmed 5. Our results demonstrate the practicality and effectiveness of SQLxDiff in detecting bugs in emerging database systems, which has the potential to improve the reliability of their applications.

Related papers

• Constant Optimization Driven Database System Testing [6.246028398098516]
  Logic bugs are bugs that can cause database management systems (DBMSs) to silently produce incorrect results for given queries. We propose Constant-Optimization-Driven Database Testing (CODDTest) as a novel approach for detecting logic bugs in databases.
  arXiv  Detail & Related papers  (2025-01-20T03:32:55Z)
• BabelBench: An Omni Benchmark for Code-Driven Analysis of Multimodal and Multistructured Data [61.936320820180875]
  Large language models (LLMs) have become increasingly pivotal across various domains. BabelBench is an innovative benchmark framework that evaluates the proficiency of LLMs in managing multimodal multistructured data with code execution. Our experimental findings on BabelBench indicate that even cutting-edge models like ChatGPT 4 exhibit substantial room for improvement.
  arXiv  Detail & Related papers  (2024-10-01T15:11:24Z)
• Hybrid Querying Over Relational Databases and Large Language Models [8.926173054003547]
  We present the first cross-domain benchmark, SWAN, containing 120 beyond-Database questions over four real-world databases. We present two solutions: one based on schema expansion and the other based on user defined functions. Our evaluation demonstrates that using GPT-4 Turbo with few-shot prompts, one can achieves up to 40.0% in execution accuracy and 48.2% in data factuality.
  arXiv  Detail & Related papers  (2024-08-01T19:29:18Z)
• Database-Augmented Query Representation for Information Retrieval [59.57065228857247]
  We present a novel retrieval framework called Database-Augmented Query representation (DAQu) DAQu augments the original query with various (query-related) metadata across multiple tables. We validate DAQu in diverse retrieval scenarios that can incorporate metadata from the relational database.
  arXiv  Detail & Related papers  (2024-06-23T05:02:21Z)
• An Empirical Study on the Characteristics of Database Access Bugs in Java Applications [5.844508449542756]
  Database-backed applications rely on the database access code to interact with the underlying database management systems (DBMSs) In this paper, we empirically investigate 423 database access bugs collected from seven large-scale Java source applications.
  arXiv  Detail & Related papers  (2024-05-23T19:26:29Z)
• Federated Neural Graph Databases [53.03085605769093]
  We propose Federated Neural Graph Database (FedNGDB), a novel framework that enables reasoning over multi-source graph-based data while preserving privacy. Unlike existing methods, FedNGDB can handle complex graph structures and relationships, making it suitable for various downstream tasks.
  arXiv  Detail & Related papers  (2024-02-22T14:57:44Z)
• Testing Database Engines via Query Plan Guidance [6.789710498230718]
  We propose the concept of Query Plan Guidance (QPG) for guiding automated testing towards "interesting" test cases. We apply our method to three mature, widely-used, and diverse database systems-DBite, TiDB, and Cockroach-and found 53 unique, previously unknown bugs.
  arXiv  Detail & Related papers  (2023-12-29T08:09:47Z)
• Detecting DBMS Bugs with Context-Sensitive Instantiation and Multi-Plan Execution [11.18715154222032]
  This paper aims to solve the two challenges, including how to generate semantically correctsql queries in a test case, and how to propose effective oracles to capture logic bugs. We have implemented a prototype system called Kangaroo and applied three widely used and well-tested semantic codes. The comparison between our system with the state-of-the-art systems shows that our system outperforms them in terms of the number of generated semantically valid queries, the explored code paths during testing, and the detected bugs.
  arXiv  Detail & Related papers  (2023-12-08T10:15:56Z)
• Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs [89.68522473384522]
  We present Bird, a big benchmark for large-scale database grounded in text-to-efficient tasks. Our emphasis on database values highlights the new challenges of dirty database contents. Even the most effective text-to-efficient models, i.e. ChatGPT, achieves only 40.08% in execution accuracy.
  arXiv  Detail & Related papers  (2023-05-04T19:02:29Z)
• SUN: Exploring Intrinsic Uncertainties in Text-to-SQL Parsers [61.48159785138462]
  This paper aims to improve the performance of text-to-dependence by exploring the intrinsic uncertainties in the neural network based approaches (called SUN) Extensive experiments on five benchmark datasets demonstrate that our method significantly outperforms competitors and achieves new state-of-the-art results.
  arXiv  Detail & Related papers  (2022-09-14T06:27:51Z)
• Robust and Transferable Anomaly Detection in Log Data using Pre-Trained Language Models [59.04636530383049]
  Anomalies or failures in large computer systems, such as the cloud, have an impact on a large number of users. We propose a framework for anomaly detection in log data, as a major troubleshooting source of system information.
  arXiv  Detail & Related papers  (2021-02-23T09:17:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.