Memorization, Emergence, and Explaining Reversal Failures: A Controlled Study of Relational Semantics in LLMs

• URL: http://arxiv.org/abs/2601.02931v1
• Date: Tue, 06 Jan 2026 11:20:38 GMT
• Title: Memorization, Emergence, and Explaining Reversal Failures: A Controlled Study of Relational Semantics in LLMs
• Authors: Yihua Zhu, Qianying Liu, Jiaxin Wang, Fei Cheng, Chaoran Liu, Akiko Aizawa, Sadao Kurohashi, Hidetoshi Shimodaira,
• Abstract summary: We propose a synthetic framework that generates text from symmetric/inverse triples, trains GPT-style autoregressive models from scratch, and evaluate memorization, logical inference, and in-context generalization.<n>We find that relational semantics emerge with sufficient logic-bearing supervision, even in shallow (2-3 layer) models, and that successful generalization aligns with stable intermediate-layer signals.
• Score: 43.414287127130684
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autoregressive LLMs perform well on relational tasks that require linking entities via relational words (e.g., father/son, friend), but it is unclear whether they learn the logical semantics of such relations (e.g., symmetry and inversion logic) and, if so, whether reversal-type failures arise from missing relational semantics or left-to-right order bias. We propose a controlled Knowledge Graph-based synthetic framework that generates text from symmetric/inverse triples, train GPT-style autoregressive models from scratch, and evaluate memorization, logical inference, and in-context generalization to unseen entities to address these questions. We find a sharp phase transition in which relational semantics emerge with sufficient logic-bearing supervision, even in shallow (2-3 layer) models, and that successful generalization aligns with stable intermediate-layer signals. Finally, order-matched forward/reverse tests and a diffusion baseline indicate that reversal failures are primarily driven by autoregressive order bias rather than deficient inversion semantics.

Related papers

• CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matching [50.65932158912512]
  We propose a new causal reasoning benchmark, CausalFlip, to encourage the development of new large language models.<n>CaulFlip consists of causal judgment questions built over event triples that could form different confounder, chain, and collider relations.<n>We evaluate LLMs under multiple training paradigms, including answer-only training, explicit Chain-of-Thought supervision, and a proposed internalized causal reasoning approach.
  arXiv  Detail & Related papers  (2026-02-23T18:06:15Z)
• Uncovering Hidden Correctness in LLM Causal Reasoning via Symbolic Verification [56.51953062869371]
  DoVerifier is a symbolic verifier that checks whether causal expressions are derivable from a given causal graph using rules from do-calculus and probability theory.<n>Our evaluations on synthetic data and causal QA benchmarks show that DoVerifier more accurately captures semantic correctness of causal reasoning traces.
  arXiv  Detail & Related papers  (2026-01-29T03:22:58Z)
• VERGE: Formal Refinement and Guidance Engine for Verifiable LLM Reasoning [4.3414302048068745]
  We present a neurosymbolic framework that combines Large Language Models with SMT solvers to produce verification-guided answers.<n>We introduce three key innovations: (1) multi-model consensus via formal semantic equivalence checking, (2) semantic routing that directs different claim types to appropriate verification strategies, and (3) precise logical error localization via Minimal Correction Subsets.<n>With the GPT-OSS-120B model, VERGE demonstrates an average performance uplift of 18.7% at convergence across a set of reasoning benchmarks compared to single-pass approaches.
  arXiv  Detail & Related papers  (2026-01-27T20:59:11Z)
• Matrix as Plan: Structured Logical Reasoning with Feedback-Driven Replanning [9.431480849387595]
  Chain-of-Thought prompting has been shown to enhance the reasoning capabilities of Large Language Models (LLMs)<n>Neuro-symbolic methods address this gap by enforcing formal correctness through external solvers.<n>We propose MatrixCoT, a structured CoT framework with a matrix-based plan.
  arXiv  Detail & Related papers  (2026-01-15T06:12:00Z)
• Directional Attractors in LLM Reasoning: How Similarity Retrieval Steers Iterative Summarization Based Reasoning [0.0]
  We introduce InftyThink with Cross-Chain Memory, an extension that augments iterative reasoning with an embedding-based semantic cache of previously successful reasoning patterns.<n> Experiments show that semantic lemma retrieval improves accuracy in structured domains while exposing failure modes in tests that include heterogeneous domains.
  arXiv  Detail & Related papers  (2025-12-22T00:26:54Z)
• Directional Optimization Asymmetry in Transformers: A Synthetic Stress Test [0.15229257192293197]
  Transformers are theoretically reversal-invariant: their function class does not prefer left-to-right over right-to-left mappings.<n>Recent work on temporal asymmetry in LLMs suggests that real-world corpora carry their own arrow of time.<n>This leaves an unresolved question: do directional failures stem from linguistic statistics, or from the architecture itself?
  arXiv  Detail & Related papers  (2025-11-25T07:03:20Z)
• DAG-Math: Graph-Guided Mathematical Reasoning in LLMs [54.231935013127206]
  Large Language Models (LLMs) demonstrate strong performance on mathematical problems when prompted with Chain-of-Thought (CoT)<n>We propose modeling CoT as a certain rule-based process over directed acyclic graphs (DAGs)<n>We introduce logical closeness, a metric that quantifies how well a model's CoT trajectory adheres to the DAG structure.
  arXiv  Detail & Related papers  (2025-10-19T21:05:17Z)
• More or Less Wrong: A Benchmark for Directional Bias in LLM Comparative Reasoning [10.301985230669684]
  We study the mechanisms by which semantic cues shape reasoning in large language models.<n>We introduce MathComp, a benchmark of 300 comparison scenarios.<n>We find that model errors frequently reflect linguistic steering, systematic shifts toward the comparative term present in the prompt.
  arXiv  Detail & Related papers  (2025-06-04T13:15:01Z)
• How do Transformers Learn Implicit Reasoning? [67.02072851088637]
  We study how implicit multi-hop reasoning emerges by training transformers from scratch in a controlled symbolic environment.<n>We find that training with atomic triples is not necessary but accelerates learning, and that second-hop generalization relies on query-level exposure to specific compositional structures.
  arXiv  Detail & Related papers  (2025-05-29T17:02:49Z)
• Implicit Bias-Like Patterns in Reasoning Models [0.5729426778193398]
  Implicit biases refer to automatic mental processes that shape perceptions, judgments, and behaviors.<n>We present the Reasoning Model Implicit Association Test (RM-IAT) to study implicit bias-like processing in reasoning models.
  arXiv  Detail & Related papers  (2025-03-14T16:40:02Z)
• Order Doesn't Matter, But Reasoning Does: Training LLMs with Order-Centric Augmentation [57.570754504160305]
  We introduce an order-centric data augmentation framework based on commutativity in logical reasoning.<n>By leveraging order-centric augmentations, models can develop a more flexible and generalized reasoning process.
  arXiv  Detail & Related papers  (2025-02-27T09:25:50Z)
• Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language Models [31.558429029429863]
  Large Language Models (LLMs) are expected to be predictable and trustworthy to support reliable decision-making systems.<n>This work examines logical preference consistency as a foundational requirement for building more dependable LLM systems.<n>We show that improving consistency leads to better performance in LLM-driven logic-based algorithms.
  arXiv  Detail & Related papers  (2024-10-03T04:34:04Z)
• Large Language Models as an Indirect Reasoner: Contrapositive and Contradiction for Automated Reasoning [74.90592233107712]
  We propose a Direct-Indirect Reasoning (DIR) method, which considers Direct Reasoning (DR) and Indirect Reasoning (IR) as multiple parallel reasoning paths that are merged to derive the final answer.<n>Our DIR method is simple yet effective and can be straightforwardly integrated with existing variants of CoT methods.
  arXiv  Detail & Related papers  (2024-02-06T03:41:12Z)
• Modeling Hierarchical Reasoning Chains by Linking Discourse Units and Key Phrases for Reading Comprehension [80.99865844249106]
  We propose a holistic graph network (HGN) which deals with context at both discourse level and word level, as the basis for logical reasoning. Specifically, node-level and type-level relations, which can be interpreted as bridges in the reasoning process, are modeled by a hierarchical interaction mechanism.
  arXiv  Detail & Related papers  (2023-06-21T07:34:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.