OpenRFM: Dissecting Relational In-Context Learning

• URL: http://arxiv.org/abs/2606.04320v1
• Date: Wed, 03 Jun 2026 00:48:44 GMT
• Title: OpenRFM: Dissecting Relational In-Context Learning
• Abstract summary: We show that a pre-trained predictor that returns in one pass via in-context learning (ICL) fails when sparse label-cell relation regression.<n>We propose a dual-stage ICL architecture that combines the relational backbone with a batch-level ICL layer lifted from a pre-trained tabular foundation model to overcome relation-level label scarcity.
• Score: 15.623084178502053
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Relational Foundation Models (RFMs) promise a single pre-trained predictor that, given any relational database, returns predictions in one forward pass via relational in-context learning (ICL). Yet a substantial gap separates open RFMs from their commercial counterparts, and the origin of this gap has not been systematically understood. We dissect a representative framework, the Relational Transformer (RT), from two perspectives. Model side: we show that RT performs relation-level ICL, and a kernel regression view shows it fails when sparse label-cell coverage yields an underdetermined regression. Data side: we ablate RT's pre-training source and find that existing synthetic-only pre-training and in-distribution pre-training drive the same architecture into different regimes, lazy vs. feature-learning. Probing this gap reveals that the missing ingredient is a support-identifiable relational latent in the label-generation process. These two diagnoses translate into (1) a dual-stage ICL architecture that combines the relational backbone with a batch-level ICL layer lifted from a pre-trained tabular foundation model to overcome relation-level label scarcity, and (2) a homophily-aware synthetic plus continual real-data pre-training mixture, augmented with a prototype-based regularization. These choices define OpenRFM, a simple yet effective RFM that improves average task performance by approximately 30% over the RT backbone and surpasses the commercial model KumoRFMv1 on a large set of evaluation tasks.

Related papers

• CONSISTRE: A Unified Consistency-Aware Framework for Document-Level Relation Extraction with Large Language Models [11.083514956613383]
  Document-level relation extraction (DocRE) aims to extract relations among multiple entities across extended contexts.<n>Large language models (LLMs) show remarkable reasoning capabilities in information extraction.<n>LLMs may violate fundamental relational constraints such as transitivity, symmetry, and functional uniqueness, leading to contradictory and unreliable outputs.<n>We propose a unified consistency-aware framework for DocRE that addresses this limitation through two complementary tracks.
  arXiv  Detail & Related papers  (2026-07-27T11:56:29Z)
• RelPrism: A Multi-Faceted Pre-training Framework with Self-Generated Tasks for Relational Databases [49.47033473074821]
  We propose RelPrism, a multi-faceted self-supervised learning framework for RDBs.<n>RDB tasks often require multi-faceted information across different perspectives and granularities.<n>Our experiments show that RelPrism improves ROC-AUC by 4.15% for classification and reduces MAE by 10.75% for regression over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-05-22T05:19:58Z)
• Posterior Augmented Flow Matching [64.1559809786948]
  Posterior-Augmented Flow Matching (PAFM) is a generalization of flow matching (FM)<n>We show that PAFM improves over FM by up to 3.4 FID50K across different model scales.
  arXiv  Detail & Related papers  (2026-05-01T17:59:59Z)
• Rel-MOSS: Towards Imbalanced Relational Deep Learning on Relational Databases [66.24230877709756]
  relational deep learning (RDL) is proposed to structure the relational databases (RDB) as a heterogeneous entity graph.<n>Existing RDL methods neglect the imbalance problem of relational data in RDBs and risk under-representing the minority entities.<n>In this work, we investigate for the first time, class imbalance problem in RDB entity classification and design the relation-centric minority synthetic over-sampling GNN.
  arXiv  Detail & Related papers  (2026-03-09T03:18:26Z)
• Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data [38.656987194921854]
  Transformer (RT) architecture can be pretrained on diverse relational databases.<n>RT churns strong zero-shot, averaging 93% of fully supervised AUROC on binary classification tasks.
  arXiv  Detail & Related papers  (2025-10-07T18:51:51Z)
• Round-trip Reinforcement Learning: Self-Consistent Training for Better Chemical LLMs [51.29260537017623]
  Large Language Models (LLMs) are emerging as versatile foundation models for computational chemistry.<n>These models often lack round-trip consistency.<n>We introduce Round-Trip Reinforcement Learning (RTRL), a novel framework that trains a model to improve its consistency.
  arXiv  Detail & Related papers  (2025-10-01T23:58:58Z)
• Federated Learning Meets LLMs: Feature Extraction From Heterogeneous Clients [0.0]
  Federated learning (FL) enables collaborative model training without sharing raw data.<n>We propose FedLLM-Align, a framework that leverages pre-trained large language models (LLMs) as universal feature extractors.<n>We evaluate FedLLM-Align on coronary heart disease prediction using partitioned datasets with simulated schema divergence.
  arXiv  Detail & Related papers  (2025-09-29T14:06:52Z)
• Task-Agnostic Contrastive Pretraining for Relational Deep Learning [0.0]
  We propose a novel task-agnostic contrastive pretraining approach for RDL that enables database-wide representation learning.<n>We implement the respective pretraining approach through a modular RDL architecture.<n>Our preliminary results demonstrate that finetuning the pretrained models measurably outperforms training from scratch.
  arXiv  Detail & Related papers  (2025-06-27T13:18:13Z)
• Fake It Till Make It: Federated Learning with Consensus-Oriented Generation [52.82176415223988]
  We propose federated learning with consensus-oriented generation (FedCOG) FedCOG consists of two key components at the client side: complementary data generation and knowledge-distillation-based model training. Experiments on classical and real-world FL datasets show that FedCOG consistently outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-12-10T18:49:59Z)
• Continual Contrastive Finetuning Improves Low-Resource Relation Extraction [34.76128090845668]
  Relation extraction has been particularly challenging in low-resource scenarios and domains. Recent literature has tackled low-resource RE by self-supervised learning. We propose to pretrain and finetune the RE model using consistent objectives of contrastive learning.
  arXiv  Detail & Related papers  (2022-12-21T07:30:22Z)
• Robust Finite Mixture Regression for Heterogeneous Targets [70.19798470463378]
  We propose an FMR model that finds sample clusters and jointly models multiple incomplete mixed-type targets simultaneously. We provide non-asymptotic oracle performance bounds for our model under a high-dimensional learning framework. The results show that our model can achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2020-10-12T03:27:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.