RDBench: ML Benchmark for Relational Databases

• URL: http://arxiv.org/abs/2310.16837v2
• Date: Mon, 30 Oct 2023 16:51:30 GMT
• Title: RDBench: ML Benchmark for Relational Databases
• Abstract summary: We introduce RDBench, a standardized benchmark to promote reproducible machine learning (ML) research on relational databases (RDBs) RDBench offers diverse RDB datasets of varying scales, domains, graphs and relational structures, organized into 4 levels. For the first time, RDBench enables meaningful comparisons between ML methods from diverse domains, ranging from XBoostG to Graph Neural Networks, under RDB prediction tasks.
• Score: 38.96429447951739
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Benefiting from high-quality datasets and standardized evaluation metrics, machine learning (ML) has achieved sustained progress and widespread applications. However, while applying machine learning to relational databases (RDBs), the absence of a well-established benchmark remains a significant obstacle to the development of ML. To address this issue, we introduce ML Benchmark For Relational Databases (RDBench), a standardized benchmark that aims to promote reproducible ML research on RDBs that include multiple tables. RDBench offers diverse RDB datasets of varying scales, domains, and relational structures, organized into 4 levels. Notably, to simplify the adoption of RDBench for diverse ML domains, for any given database, RDBench exposes three types of interfaces including tabular data, homogeneous graphs, and heterogeneous graphs, sharing the same underlying task definition. For the first time, RDBench enables meaningful comparisons between ML methods from diverse domains, ranging from XGBoost to Graph Neural Networks, under RDB prediction tasks. We design multiple classification and regression tasks for each RDB dataset and report averaged results over the same dataset, further enhancing the robustness of the experimental findings. RDBench is implemented with DBGym, a user-friendly platform for ML research and application on databases, enabling benchmarking new ML methods with RDBench at ease.

Related papers

• Can LLMs Normalize Databases? A Benchmark and Multi-Agent Framework for Schema Normalization [5.32197677925365]
  Large Language Models (LLMs) are increasingly used to generate structured outputs, but their reliability remains unclear.<n>We study this issue through database normalization, involving reasoning about functional dependencies, join decompositions, and inter-table constraints.<n>We introduce a Database Normalization Benchmark (DNBENCH), comprising 3,275 samples for evaluating LLM-driven database normalization from 1NF to BCNF.
  arXiv  Detail & Related papers  (2026-09-10T06:34:35Z)
• A Fair Benchmarking of Deep Relational Database Learning Models [0.0]
  relational databases (RDBs) are the primary infrastructure in many enterprises.<n>Recent deep learning methods designed for RDBs have been evaluated under inconsistent experimental protocols.<n>We present one of the first systematic benchmarking studies of recently released deep learning methods for RDBs.
  arXiv  Detail & Related papers  (2026-07-04T01:20:21Z)
• RelPrism: A Multi-Faceted Pre-training Framework with Self-Generated Tasks for Relational Databases [49.47033473074821]
  We propose RelPrism, a multi-faceted self-supervised learning framework for RDBs.<n>RDB tasks often require multi-faceted information across different perspectives and granularities.<n>Our experiments show that RelPrism improves ROC-AUC by 4.15% for classification and reduces MAE by 10.75% for regression over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-05-22T05:19:58Z)
• RelBench v2: A Large-Scale Benchmark and Repository for Relational Data [33.795728412518834]
  We introduce RelBench v2, a major expansion of the RelBench benchmark for RDL.<n>RDL models consistently outperform single-table baselines across autocomplete forecasting, and recommendation tasks.
  arXiv  Detail & Related papers  (2026-02-13T04:23:48Z)
• Relational Database Distillation: From Structured Tables to Condensed Graph Data [48.347717300340435]
  We aim to distill large-scale RDBs into compact heterogeneous graphs while retaining the power required for graph-based models.<n>We further design a kernel ridge regression-guided objective with pseudo-labels, which produces quality features for the distilled graph.
  arXiv  Detail & Related papers  (2025-10-08T13:05:31Z)
• REDELEX: A Framework for Relational Deep Learning Exploration [0.0]
  Recently, Deep Deep Learning has emerged as a novel paradigm wherein RDBs are conceptualized as graph structures.<n>There is a lack of analysis into the relationships between various RDL models and the characteristics of the underlying RDBs.<n>We present REDELEX$-$a comprehensive exploration framework for evaluating RDL models of varying complexity on the most diverse collection of over 70 RDBs.
  arXiv  Detail & Related papers  (2025-06-27T13:05:15Z)
• RealHiTBench: A Comprehensive Realistic Hierarchical Table Benchmark for Evaluating LLM-Based Table Analysis [16.572608600078922]
  RealHiTBench is a benchmark designed to evaluate the performance of Large Language Models (LLMs) across a variety of input formats.<n>Our experimental results, using 25 state-of-the-art LLMs, demonstrate that RealHiTBench is indeed a challenging benchmark.<n>We also develop TreeThinker, a tree-based pipeline that organizes hierarchical headers into a tree structure.
  arXiv  Detail & Related papers  (2025-06-16T12:19:08Z)
• RDB2G-Bench: A Comprehensive Benchmark for Automatic Graph Modeling of Relational Databases [23.836665904554426]
  RDB-to-graph modeling helps capture cross-table dependencies, leading to enhanced performance across diverse tasks.<n>Applying a common rule for graph modeling leads to a 10% drop in performance compared to the best-performing graph model.<n>We introduce RDB2G, the first benchmark framework for evaluating such methods.
  arXiv  Detail & Related papers  (2025-06-02T06:34:10Z)
• Learning to Route Queries Across Knowledge Bases for Step-wise Retrieval-Augmented Reasoning [60.84901522792042]
  Multimodal Retrieval-Augmented Generation (MRAG) has shown promise in mitigating hallucinations in Multimodal Large Language Models (MLLMs)<n>We propose R1, a novel MRAG framework that learns to decide when and where to retrieve knowledge based on the evolving reasoning state.<n>R1- can adaptively and effectively leverage diverse KBs, reducing unnecessary retrievals and improving both efficiency and accuracy.
  arXiv  Detail & Related papers  (2025-05-28T08:17:57Z)
• Griffin: Towards a Graph-Centric Relational Database Foundation Model [37.09648739513178]
  Griffin is the first foundation model attemptation designed specifically for Databases (RDBs)<n>We enhance the architecture by incorporating a cross-attention module and a novel aggregator.<n>Griffin is evaluated on large-scale, heterogeneous, and temporal graphs extracted from RDBs across various domains.
  arXiv  Detail & Related papers  (2025-05-08T18:03:43Z)
• SchemaAgent: A Multi-Agents Framework for Generating Relational Database Schema [35.57815867567431]
  Existing efforts are mostly based on customized rules or conventional deep learning models, often producing relational schema. We propose a unified LLM-based multi-agent framework for the automated generation of high-quality database schema.Agent. We incorporate dedicated roles for reflection and inspection, alongside an innovative error detection and correction mechanism to identify rectify issues across various phases.
  arXiv  Detail & Related papers  (2025-03-31T09:39:19Z)
• DB-Explore: Automated Database Exploration and Instruction Synthesis for Text-to-SQL [18.915121803834698]
  We propose DB-Explore, a novel framework for database understanding using large language models (LLMs) Our framework enables comprehensive database understanding through diverse sampling strategies and automated instruction generation. Our open-source implementation, based on Qwen2.5-coder-7B model, outperforms multiple GPT-4-driven text-to-coder systems in comparative evaluations.
  arXiv  Detail & Related papers  (2025-03-06T20:46:43Z)
• CoddLLM: Empowering Large Language Models for Data Analytics [38.23203246023766]
  Large Language Models (LLMs) have the potential to revolutionize data analytics. We unveil a new data recipe for post-Turbo synthesiss. We posttrain a new foundation model, named CoddLLM, based on MistralNeMo-12B.
  arXiv  Detail & Related papers  (2025-02-01T06:03:55Z)
• MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale [66.73529246309033]
  multimodal large language models (MLLMs) have shown significant potential in a broad range of multimodal tasks. Existing instruction-tuning datasets only provide phrase-level answers without any intermediate rationales. We introduce a scalable and cost-effective method to construct a large-scale multimodal instruction-tuning dataset with rich intermediate rationales.
  arXiv  Detail & Related papers  (2024-12-06T18:14:24Z)
• TableRAG: Million-Token Table Understanding with Language Models [53.039560091592215]
  TableRAG is a Retrieval-Augmented Generation (RAG) framework specifically designed for LM-based table understanding. TableRAG leverages query expansion combined with schema and cell retrieval to pinpoint crucial information before providing it to the LMs. Our results demonstrate that TableRAG achieves the highest retrieval quality, leading to the new state-of-the-art performance on large-scale table understanding.
  arXiv  Detail & Related papers  (2024-10-07T04:15:02Z)
• BabelBench: An Omni Benchmark for Code-Driven Analysis of Multimodal and Multistructured Data [61.936320820180875]
  Large language models (LLMs) have become increasingly pivotal across various domains. BabelBench is an innovative benchmark framework that evaluates the proficiency of LLMs in managing multimodal multistructured data with code execution. Our experimental findings on BabelBench indicate that even cutting-edge models like ChatGPT 4 exhibit substantial room for improvement.
  arXiv  Detail & Related papers  (2024-10-01T15:11:24Z)
• RelBench: A Benchmark for Deep Learning on Relational Databases [78.52438155603781]
  We present RelBench, a public benchmark for solving tasks over databases with graph neural networks. We use RelBench to conduct the first comprehensive study of Deep Learning infrastructure. RDL learns better whilst reducing human work needed by more than an order of magnitude.
  arXiv  Detail & Related papers  (2024-07-29T14:46:13Z)
• BERGEN: A Benchmarking Library for Retrieval-Augmented Generation [26.158785168036662]
  Retrieval-Augmented Generation allows to enhance Large Language Models with external knowledge. Inconsistent benchmarking poses a major challenge in comparing approaches and understanding the impact of each component in the pipeline. In this work, we study best practices that lay the groundwork for a systematic evaluation of RAG and present BERGEN, an end-to-end library for reproducible research standardizing RAG experiments.
  arXiv  Detail & Related papers  (2024-07-01T09:09:27Z)
• R-Eval: A Unified Toolkit for Evaluating Domain Knowledge of Retrieval Augmented Large Language Models [51.468732121824125]
  Large language models have achieved remarkable success on general NLP tasks, but they may fall short for domain-specific problems. Existing evaluation tools only provide a few baselines and evaluate them on various domains without mining the depth of domain knowledge. In this paper, we address the challenges of evaluating RALLMs by introducing the R-Eval toolkit, a Python toolkit designed to streamline the evaluation of different RAGs.
  arXiv  Detail & Related papers  (2024-06-17T15:59:49Z)
• TopoBench: A Framework for Benchmarking Topological Deep Learning [48.21297982414672]
  This work introduces TopoBench, an open-source library designed to standardize and accelerate research in topological deep learning (TDL)<n>TopoBench decomposes TDL into a sequence of independent modules for data generation, loading, transforming and processing, as well as model training, optimization and evaluation.<n>A key feature of TopoBench is its support for transformations and lifting across topological domains.
  arXiv  Detail & Related papers  (2024-06-09T18:31:19Z)
• M-RAG: Reinforcing Large Language Model Performance through Retrieval-Augmented Generation with Multiple Partitions [16.188375536066044]
  Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by retrieving relevant memories from an external database. Existing RAG methods typically organize all memories in a whole database. We introduce a multiple partition paradigm for RAG (called M-RAG), where each database partition serves as a basic unit for RAG execution.
  arXiv  Detail & Related papers  (2024-05-26T04:03:13Z)
• 4DBInfer: A 4D Benchmarking Toolbox for Graph-Centric Predictive Modeling on Relational DBs [67.47600679176963]
  RDBs store vast amounts of rich, informative data spread across interconnected tables. The progress of predictive machine learning models falls behind advances in other domains such as computer vision or natural language processing. We explore a class of baseline models predicated on converting multi-table datasets into graphs. We assemble a diverse collection of large-scale RDB datasets and (ii) coincident predictive tasks.
  arXiv  Detail & Related papers  (2024-04-28T15:04:54Z)
• JoinBoost: Grow Trees Over Normalized Data Using Only SQL [10.919507523061888]
  JoinBoost is a Python library that rewrites tree training algorithms over normalized databases into pure SQL. It offers performance competitive with specialized ML libraries, and scales with the underlying capabilities. Our experiment shows that JoinBoost is 3x faster for random forests (gradient boosting) than LightGBM, and over an order magnitude faster than state-of-the-art In-DB ML systems.
  arXiv  Detail & Related papers  (2023-07-01T20:18:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.