Towards Foundation Models for Relational Databases with Language Models and Graph Neural Networks

• URL: http://arxiv.org/abs/2605.16085v1
• Date: Fri, 15 May 2026 15:46:38 GMT
• Title: Towards Foundation Models for Relational Databases with Language Models and Graph Neural Networks
• Abstract summary: We propose a hybrid architecture combining a fine-tuned BART architecture with a GraphS-based GNN over relational REGs.<n>Experiments show that the GNN substantially enriches BART's row inject context, achieving a ROC- embedding of 67.40 on the driverdnf task from the rel-f1 dataset.<n>These results suggest that lightweight hybrid LM-GNN architectures offer a promising and resource-efficient path towards foundation models for relational databases.
• Score: 14.51793414693779
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Relational databases store much of the world's structured information, and they are essential for driving complex predictive applications. However, deep learning progress on relational data remains limited, as conventional approaches flatten databases into single tables via manual feature engineering, discarding relational context. Relational deep learning (RDL) addresses this by modeling databases as relational entity graphs (REGs) for graph neural networks (GNNs), but remains task- and database-specific. To combine the strengths of both paradigms, we propose a hybrid architecture combining a fine-tuned BART encoder to capture intra-row semantics with a GraphSAGE-based GNN over REGs to inject relational context. Experiments on RelBench show that the GNN substantially enriches BART's row embeddings, achieving a ROC-AUC of 67.40 on the driver-dnf task from the rel-f1 dataset. This performance is competitive with supervised baselines such as LightGBM (68.86) and narrows the gap to RDL (72.62) to within 5.22 points, though a substantial gap remains to state-of-the-art foundation models such as KumoRFM (82.63). These results suggest that lightweight hybrid LM-GNN architectures offer a promising and resource-efficient path towards foundation models for relational databases.

Related papers

• Is Fixing Schema Graphs Necessary? Full-Resolution Graph Structure Learning for Relational Deep Learning [34.501460016762756]
  FullResolution Structure and Optimizable Graph framework for RDL formulates learning structure as learnable table role modeling problem.<n>We introduce dependency constraints that regularize functional representations across table entity levels.<n>Experiments demonstrate that our method outperforms existing approaches.
  arXiv  Detail & Related papers  (2026-05-20T17:56:09Z)
• Relational In-Context Learning via Synthetic Pre-training with Structural Prior [60.404256960057545]
  RDB-PFN is the first relational foundation model trained purely via $textbfsynthetic$.<n>Inspired by Prior-Data Fitted Networks (PFNs) where synthetic data generated from Structural Causal Models (SCMs) enables reasoning on single tables.<n>Experiments verify RDB-PFN achieves strong few-shot performance on 19 real-world prediction tasks.
  arXiv  Detail & Related papers  (2026-03-04T07:30:54Z)
• Relational Database Distillation: From Structured Tables to Condensed Graph Data [48.347717300340435]
  We aim to distill large-scale RDBs into compact heterogeneous graphs while retaining the power required for graph-based models.<n>We further design a kernel ridge regression-guided objective with pseudo-labels, which produces quality features for the distilled graph.
  arXiv  Detail & Related papers  (2025-10-08T13:05:31Z)
• A Pre-training Framework for Relational Data with Information-theoretic Principles [57.93973948947743]
  We introduce Task Vector Estimation (TVE), a novel pre-training framework that constructs supervisory signals via set-based aggregation over relational graphs.<n>TVE consistently outperforms traditional pre-training baselines.<n>Our findings advocate for pre-training objectives that encode task heterogeneity and temporal structure as design principles for predictive modeling on relational databases.
  arXiv  Detail & Related papers  (2025-07-14T00:17:21Z)
• Relational Deep Learning: Challenges, Foundations and Next-Generation Architectures [50.46688111973999]
  Graph machine learning has led to a significant increase in the capabilities of models that learn on arbitrary graph-structured data.<n>We present a new blueprint that enables end-to-end representation of'relational entity graphs' without traditional engineering feature.<n>We discuss key challenges including large-scale multi-table integration and the complexities of modeling temporal dynamics and heterogeneous data.
  arXiv  Detail & Related papers  (2025-06-19T23:51:38Z)
• Joint Relational Database Generation via Graph-Conditional Diffusion Models [44.06390394789874]
  Building generative models for databases (RDBs) is important for applications like privacy's data release and real datasets.<n>Most prior either focuses on single-table generation or relies on autoregressive factorizations that impose a fixed table order and generate tables sequentially.<n>We propose a fundamentally different approach: jointly modeling all tables in an RDB without imposing any order.
  arXiv  Detail & Related papers  (2025-05-22T11:12:56Z)
• Boosting Relational Deep Learning with Pretrained Tabular Models [18.34233986830027]
  Graph Neural Networks (GNNs) offer a compelling alternative inherently by modeling these relationships.<n>Our framework achieves up to $33%$ performance improvement and a $526times$ inference speedup compared to GNNs.
  arXiv  Detail & Related papers  (2025-04-07T11:19:04Z)
• RelGNN: Composite Message Passing for Relational Deep Learning [56.48834369525997]
  We introduce RelGNN, a novel GNN framework specifically designed to leverage the unique structural characteristics of the graphs built from relational databases.<n>RelGNN is evaluated on 30 diverse real-world tasks from Relbench (Fey et al., 2024), and achieves state-of-the-art performance on the vast majority tasks, with improvements of up to 25%.
  arXiv  Detail & Related papers  (2025-02-10T18:58:40Z)
• RelBench: A Benchmark for Deep Learning on Relational Databases [78.52438155603781]
  We present RelBench, a public benchmark for solving tasks over databases with graph neural networks. We use RelBench to conduct the first comprehensive study of Deep Learning infrastructure. RDL learns better whilst reducing human work needed by more than an order of magnitude.
  arXiv  Detail & Related papers  (2024-07-29T14:46:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.