Thought without systematicity? Evaluating reasoning models on rule induction tasks

• URL: http://arxiv.org/abs/2609.13948v1
• Date: Sat, 12 Sep 2026 13:45:11 GMT
• Title: Thought without systematicity? Evaluating reasoning models on rule induction tasks
• Abstract summary: We extend established rule induction tasks from cognitive science to assess the systematicity of thought in current reasoning models.<n>We find that despite being able to correctly solve a task, models often fail on structurally equivalent variants of the same task.
• Score: 7.063564727972718
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: A central tenet of human cognition is systematicity, the principle that understanding one concept is inherently tied to understanding close variations of that concept. Do reasoning models robustly exhibit such systematicity? If so, we would expect consistent performance on structurally equivalent variants of the same task. Here, we extend established rule induction tasks from cognitive science to assess the systematicity of thought in current reasoning models. Each task family has compositional structure that we use to create structurally equivalent task variations through task isomorphisms such as recombination and substitution. We find that despite being able to correctly solve a task, models often fail on structurally equivalent variants of the same task. These findings suggest that many model behaviors lack systematicity, rendering it difficult to robustly establish the cognitive abilities of reasoning models beyond the particular contexts they were evaluated in.

Related papers

• OrigamiBench: An Interactive Environment to Synthesize Flat-Foldable Origamis [28.493060188882314]
  Building AI systems that can plan, act, and create in the physical world requires more than pattern recognition.<n>Such systems must understand the causal mechanisms and constraints governing physical processes in order to guide sequential decisions.<n>This capability relies on internal representations, analogous to an internal language model, that relate observations, actions, and resulting environmental changes.<n>We introduce OrigamiBench, an interactive benchmark in which models iteratively propose folds and receive feedback on physical validity and similarity to a target configuration.
  arXiv  Detail & Related papers  (2026-03-14T09:33:29Z)
• A Survey of Reasoning in Autonomous Driving Systems: Open Challenges and Emerging Paradigms [49.66022971508878]
  We argue that reasoning should be elevated from a modular component to the system's cognitive core.<n>We derive and systematize seven core reasoning challenges, such as the responsiveness-reasoning trade-off and social-game reasoning.<n>We identify a fundamental and unresolved tension between the high-latency, deliberative nature of LLM-based reasoning and the millisecond-scale, safety-critical demands of vehicle control.
  arXiv  Detail & Related papers  (2026-03-11T07:40:53Z)
• Determinism and Indeterminism as Model Artefacts: Toward a Model-Invariant Ontology of Physics [0.0]
  I argue that the traditional opposition between determinism and indeterminism in physics is representational rather than ontological.<n>I use this model-equivalence to motivate a model-invariance criterion for ontological commitment.<n>I offer a realist for modern physics that combines empirical openness with resistance to metaphysical overreach.
  arXiv  Detail & Related papers  (2025-12-27T09:41:03Z)
• Reasoning Systems as Structured Processes: Foundations, Failures, and Formal Criteria [1.2064681974642195]
  We model reasoning systems as structureds comprising phenomena, explanation space, inference and generation maps, and a principle base.<n>We survey basic internal criteria--including coherence, soundness, and completeness--and catalog typical failure modes such as contradiction, incompleteness, and non-convergence.<n>The goal of this work is to establish a foundational structure for representing and comparing reasoning systems, particularly in contexts where internal failure, adaptation, or fragmentation may arise.
  arXiv  Detail & Related papers  (2025-08-03T14:04:15Z)
• Causality can systematically address the monsters under the bench(marks) [64.36592889550431]
  Benchmarks are plagued by various biases, artifacts, or leakage.<n>Models may behave unreliably due to poorly explored failure modes.<n> causality offers an ideal framework to systematically address these challenges.
  arXiv  Detail & Related papers  (2025-02-07T17:01:37Z)
• Understanding the Language Model to Solve the Symbolic Multi-Step Reasoning Problem from the Perspective of Buffer Mechanism [68.05754701230039]
  We construct a symbolic multi-step reasoning task to investigate the information propagation mechanisms in Transformer models.<n>We propose a random matrix-based algorithm to enhance the model's reasoning ability.
  arXiv  Detail & Related papers  (2024-05-24T07:41:26Z)
• What makes Models Compositional? A Theoretical View: With Supplement [60.284698521569936]
  We propose a general neuro-symbolic definition of compositional functions and their compositional complexity. We show how various existing general and special purpose sequence processing models fit this definition and use it to analyze their compositional complexity.
  arXiv  Detail & Related papers  (2024-05-02T20:10:27Z)
• Hierarchical Invariance for Robust and Interpretable Vision Tasks at Larger Scales [54.78115855552886]
  We show how to construct over-complete invariants with a Convolutional Neural Networks (CNN)-like hierarchical architecture. With the over-completeness, discriminative features w.r.t. the task can be adaptively formed in a Neural Architecture Search (NAS)-like manner. For robust and interpretable vision tasks at larger scales, hierarchical invariant representation can be considered as an effective alternative to traditional CNN and invariants.
  arXiv  Detail & Related papers  (2024-02-23T16:50:07Z)
• On Principles of Emergent Organization [0.0]
  After more than a century of concerted effort, physics still lacks basic principles of spontaneous self-organization. challenges arising from mathematical intractability and the resulting need for computational approaches. An overview of two modern mathematical formulations of organization lays out a way to overcome these challenges.
  arXiv  Detail & Related papers  (2023-11-23T00:39:36Z)
• Ontology Revision based on Pre-trained Language Models [32.92146634065263]
  Ontology revision aims to seamlessly incorporate a new ontology into an existing ontology. Incoherence is a main potential factor to cause inconsistency and reasoning with an inconsistent ontology will obtain meaningless answers. To deal with this problem, various ontology revision approaches have been proposed to define revision operators and design ranking strategies for axioms. In this paper, we study how to apply pre-trained models to revise.
  arXiv  Detail & Related papers  (2023-10-27T00:52:01Z)
• Generative Models as a Complex Systems Science: How can we make sense of large language model behavior? [75.79305790453654]
  Coaxing out desired behavior from pretrained models, while avoiding undesirable ones, has redefined NLP. We argue for a systematic effort to decompose language model behavior into categories that explain cross-task performance.
  arXiv  Detail & Related papers  (2023-07-31T22:58:41Z)
• Neural Causal Models for Counterfactual Identification and Estimation [62.30444687707919]
  We study the evaluation of counterfactual statements through neural models. First, we show that neural causal models (NCMs) are expressive enough. Second, we develop an algorithm for simultaneously identifying and estimating counterfactual distributions.
  arXiv  Detail & Related papers  (2022-09-30T18:29:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.