Evaluating Relational Reasoning in LLMs with REL

• URL: http://arxiv.org/abs/2604.12176v1
• Date: Tue, 14 Apr 2026 01:07:15 GMT
• Title: Evaluating Relational Reasoning in LLMs with REL
• Abstract summary: relational reasoning is the ability to infer that jointly bind multiple entities, attributes or variables.<n>We develop a principled way to vary reasoning difficulty while controlling for confounders such as input size, vocabulary, representational choices.<n>Across frontier LLMs degrades consistently and monotonically as RC increases, even when the total number of entities is held fixed.
• Score: 44.475732460699895
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Relational reasoning is the ability to infer relations that jointly bind multiple entities, attributes, or variables. This ability is central to scientific reasoning, but existing evaluations of relational reasoning in large language models often focus on structured inputs such as tables, graphs, or synthetic tasks, and do not isolate the difficulty introduced by higher-arity relational binding. We study this problem through the lens of Relational Complexity (RC), which we define as the minimum number of independent entities or operands that must be simultaneously bound to apply a relation. RC provides a principled way to vary reasoning difficulty while controlling for confounders such as input size, vocabulary, and representational choices. Building on RC, we introduce REL, a generative benchmark framework spanning algebra, chemistry, and biology that varies RC within each domain. Across frontier LLMs, performance degrades consistently and monotonically as RC increases, even when the total number of entities is held fixed. This failure mode persists with increased test-time compute and in-context learning, suggesting a limitation tied to the arity of the required relational binding rather than to insufficient inference steps or lack of exposure to examples. Our results identify a regime of higher-arity reasoning in which current models struggle, and motivate re-examining benchmarks through the lens of relational complexity.

Related papers

• ReaORE: Reasoning-Guided Progressive Open Relation Extraction Empowered by Large Reasoning Models [31.574557765588096]
  Reasoning-guided progressive OpenRE (ReaORE) is a framework for performing relation extraction through coarse-to-fine relation reasoning.<n>ReaORE consists of two key stages: relation filtering and relation prediction.<n>Experiments on two widely used OpenRE datasets demonstrate that ReaORE outperforms existing baselines.
  arXiv  Detail & Related papers  (2026-06-25T12:59:53Z)
• Rel-MOSS: Towards Imbalanced Relational Deep Learning on Relational Databases [66.24230877709756]
  relational deep learning (RDL) is proposed to structure the relational databases (RDB) as a heterogeneous entity graph.<n>Existing RDL methods neglect the imbalance problem of relational data in RDBs and risk under-representing the minority entities.<n>In this work, we investigate for the first time, class imbalance problem in RDB entity classification and design the relation-centric minority synthetic over-sampling GNN.
  arXiv  Detail & Related papers  (2026-03-09T03:18:26Z)
• CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matching [50.65932158912512]
  We propose a new causal reasoning benchmark, CausalFlip, to encourage the development of new large language models.<n>CaulFlip consists of causal judgment questions built over event triples that could form different confounder, chain, and collider relations.<n>We evaluate LLMs under multiple training paradigms, including answer-only training, explicit Chain-of-Thought supervision, and a proposed internalized causal reasoning approach.
  arXiv  Detail & Related papers  (2026-02-23T18:06:15Z)
• CogniLoad: A Synthetic Natural Language Reasoning Benchmark With Tunable Length, Intrinsic Difficulty, and Distractor Density [2.4149105714758545]
  We introduce CogniLoad, a novel synthetic benchmark grounded in Cognitive Load Theory (CLT)<n>CogniLoad generates natural-language logic puzzles with independently tunable parameters that reflect CLT's core dimensions.<n>CogniLoad reveals distinct performance sensitivities, identifying task length as a dominant constraint.
  arXiv  Detail & Related papers  (2025-09-22T22:28:33Z)
• Coherent Multimodal Reasoning with Iterative Self-Evaluation for Vision-Language Models [4.064135211977999]
  Large language models (LLMs) and vision-language models (LVLMs) struggle with complex, multi-step, cross-modal common sense reasoning tasks.<n>We propose the Coherent Multimodal Reasoning Framework (CMRF), a novel approach that enhances LVLMs' common sense reasoning capabilities.<n>CMRF mimics human problem-solving by decomposing complex queries, generating step-by-step inferences, and self-correcting errors.
  arXiv  Detail & Related papers  (2025-08-04T20:33:58Z)
• RelationalFactQA: A Benchmark for Evaluating Tabular Fact Retrieval from Large Language Models [9.211266032947497]
  We demonstrate that fact retrieval is substantially more difficult than isolated point-wise queries.<n>Our experiments reveal that even stateofthe-art LLMs struggle significantly, not exceeding 25% factual accuracy.<n>These findings underscore limitations in current LLMs' ability to synthesize structured factual knowledge.
  arXiv  Detail & Related papers  (2025-05-27T16:33:38Z)
• Relation Extraction with Fine-Tuned Large Language Models in Retrieval Augmented Generation Frameworks [0.0]
  Relation Extraction (RE) is crucial for converting unstructured data into structured formats like Knowledge Graphs (KGs) Recent studies leveraging pre-trained language models (PLMs) have shown significant success in this area. This work explores the performance of fine-tuned LLMs and their integration into the Retrieval Augmented-based (RAG) RE approach.
  arXiv  Detail & Related papers  (2024-06-20T21:27:57Z)
• Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlation [57.351098530477124]
  We consider one critical type of robustness against spurious correlation, where different portions of the state do not have correlations induced by unobserved confounders. A model that learns such useless or even harmful correlation could catastrophically fail when the confounder in the test case deviates from the training one. Existing robust algorithms that assume simple and unstructured uncertainty sets are therefore inadequate to address this challenge.
  arXiv  Detail & Related papers  (2023-07-15T23:53:37Z)
• Modeling Hierarchical Reasoning Chains by Linking Discourse Units and Key Phrases for Reading Comprehension [80.99865844249106]
  We propose a holistic graph network (HGN) which deals with context at both discourse level and word level, as the basis for logical reasoning. Specifically, node-level and type-level relations, which can be interpreted as bridges in the reasoning process, are modeled by a hierarchical interaction mechanism.
  arXiv  Detail & Related papers  (2023-06-21T07:34:27Z)
• Disentangling Observed Causal Effects from Latent Confounders using Method of Moments [67.27068846108047]
  We provide guarantees on identifiability and learnability under mild assumptions. We develop efficient algorithms based on coupled tensor decomposition with linear constraints to obtain scalable and guaranteed solutions.
  arXiv  Detail & Related papers  (2021-01-17T07:48:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.