Can LLMs Use Relational Transformer Embeddings?

• URL: http://arxiv.org/abs/2609.00457v1
• Date: Mon, 31 Aug 2026 22:51:21 GMT
• Title: Can LLMs Use Relational Transformer Embeddings?
• Abstract summary: Injecting frozen-encoder embeddings as soft tokens into a large language model (LLM) is a conceptually appealing fusion strategy.<n>We test this hypothesis by injecting LLMs from a frozen Transformer (RT) into Qwen3.<n>We report negative results and analyze the failure modes, arguing that soft-token fusion requires stronger alignment objectives and schema-aware design before it can serve as a route to prediction.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Injecting frozen relational-encoder embeddings as soft tokens into a large language model (LLM) is a conceptually appealing fusion strategy: the encoder handles multi-table structure, the LLM handles language and reasoning, and no lossy text serialization is required. We test this hypothesis concretely by injecting embeddings from a frozen Relational Transformer (RT) into Qwen3.5-4B via a learned MLP projection and LoRA adaptation, trained first with supervised fine-tuning (SFT) on chain-of-thought reasoning traces and then with group-based reinforcement learning (GSPO). We evaluate across 10 binary classification tasks on 6 relational databases from RelBench, under four supervision regimes: single-task (ST), within-dataset (WD), cross-dataset (CD), and all-task (ALL). The hybrid model does not consistently outperform standalone RT: it is frequently below random, highly sensitive to serialization format and relational-token budget, and unstable under RL training. We report these negative results and analyze the failure modes, arguing that soft-token fusion requires stronger alignment objectives and schema-aware design before it can serve as a reliable route to relational prediction.

Related papers

• Empowering Compact LLMs with Fusion of Layer-wise Exits for Recommendation [59.216721553953]
  Large language model-based recommender systems (LLM-RSs) have demonstrated remarkable capabilities, but are computationally unsustainable for many real-world applications.<n>We propose a discriminative framework that enhances compact LLMs while retaining scalable full-corpus ranking.<n>Experiments on three real-world datasets with Qwen 3 1.7B and Llama 3.2 3B show that FLEXRec achieves state-of-the-art accuracy while remaining highly efficient.
  arXiv  Detail & Related papers  (2026-08-18T03:14:28Z)
• OpenRFM: Dissecting Relational In-Context Learning [15.623084178502053]
  We show that a pre-trained predictor that returns in one pass via in-context learning (ICL) fails when sparse label-cell relation regression.<n>We propose a dual-stage ICL architecture that combines the relational backbone with a batch-level ICL layer lifted from a pre-trained tabular foundation model to overcome relation-level label scarcity.
  arXiv  Detail & Related papers  (2026-06-03T00:48:44Z)
• Logic-Regularized Verifier Elicits Reasoning from LLMs [63.65875399266337]
  We propose LOVER, an unsupervised verifier regularized by logical rules.<n>Lover treats theverifier as a binary latent variable, utilizinginternal activations and enforcing three logical constraints.<n>Lover significantly outperforms unsupervised baselines.
  arXiv  Detail & Related papers  (2026-05-07T09:03:49Z)
• Matrix as Plan: Structured Logical Reasoning with Feedback-Driven Replanning [9.431480849387595]
  Chain-of-Thought prompting has been shown to enhance the reasoning capabilities of Large Language Models (LLMs)<n>Neuro-symbolic methods address this gap by enforcing formal correctness through external solvers.<n>We propose MatrixCoT, a structured CoT framework with a matrix-based plan.
  arXiv  Detail & Related papers  (2026-01-15T06:12:00Z)
• DAG-Math: Graph-Guided Mathematical Reasoning in LLMs [54.231935013127206]
  Large Language Models (LLMs) demonstrate strong performance on mathematical problems when prompted with Chain-of-Thought (CoT)<n>We propose modeling CoT as a certain rule-based process over directed acyclic graphs (DAGs)<n>We introduce logical closeness, a metric that quantifies how well a model's CoT trajectory adheres to the DAG structure.
  arXiv  Detail & Related papers  (2025-10-19T21:05:17Z)
• Training-Free Time Series Classification via In-Context Reasoning with LLM Agents [29.14242392533328]
  Time series classification (TSC) spans diverse application scenarios, yet labeled data are often scarce.<n>We propose FETA, a multi-agent framework for training-free TSC via exemplar-based in-context reasoning.
  arXiv  Detail & Related papers  (2025-10-07T14:07:43Z)
• Round-trip Reinforcement Learning: Self-Consistent Training for Better Chemical LLMs [51.29260537017623]
  Large Language Models (LLMs) are emerging as versatile foundation models for computational chemistry.<n>These models often lack round-trip consistency.<n>We introduce Round-Trip Reinforcement Learning (RTRL), a novel framework that trains a model to improve its consistency.
  arXiv  Detail & Related papers  (2025-10-01T23:58:58Z)
• Datarus-R1: An Adaptive Multi-Step Reasoning LLM for Automated Data Analysis [0.0]
  We present Datarus-R1-14B, a language model fine-tuned from Qwen 2.5-14B-Instruct to act as a virtual data analyst and graduate-level problem solver.<n>Datarus is trained not on isolated question-answer pairs but on full analytical trajectories including reasoning steps, code execution, error traces, self-corrections, and final conclusions.
  arXiv  Detail & Related papers  (2025-08-18T21:58:18Z)
• Table-r1: Self-supervised and Reinforcement Learning for Program-based Table Reasoning in Small Language Models [52.94091440130039]
  Table reasoning (TR) requires structured reasoning over semi-structured data.<n>Small language models (SLMs) have limited capacity compared to large LMs (LLMs, e.g., GPT-4o)<n>We propose program-based TR (P-TR), which circumvents key limitations of text-based TR (T-TR) by generating executable programs.<n>Experiments on four TR benchmarks demonstrate that Table-r1 outperforms all SLM-based methods.
  arXiv  Detail & Related papers  (2025-06-06T14:52:19Z)
• Self-Training Elicits Concise Reasoning in Large Language Models [23.475414693530965]
  Chain-of-thought (CoT) reasoning has enabled large language models (LLMs) to utilize additional computation through intermediate tokens.<n>We propose simple fine-tuning methods which leverage self-generated concise reasoning paths.<n>Our method achieves a 30% reduction in output tokens, across five model families on GSM8K and MATH, while maintaining average accuracy.
  arXiv  Detail & Related papers  (2025-02-27T14:14:50Z)
• Reliable Text-to-SQL with Adaptive Abstention [21.07332675929629]
  We present a novel framework that enhances query generation reliability by incorporating abstention and human-in-the-loop mechanisms.<n>We validate our approach through comprehensive experiments on the BIRD benchmark, demonstrating significant improvements in robustness and reliability.
  arXiv  Detail & Related papers  (2025-01-18T19:36:37Z)
• TAT-LLM: A Specialized Language Model for Discrete Reasoning over Tabular and Textual Data [73.29220562541204]
  We consider harnessing the amazing power of language models (LLMs) to solve our task. We develop a TAT-LLM language model by fine-tuning LLaMA 2 with the training data generated automatically from existing expert-annotated datasets.
  arXiv  Detail & Related papers  (2024-01-24T04:28:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.