Why the Valuable Capabilities of LLMs Are Precisely the Unexplainable Ones

• URL: http://arxiv.org/abs/2603.15238v2
• Date: Tue, 17 Mar 2026 11:49:03 GMT
• Title: Why the Valuable Capabilities of LLMs Are Precisely the Unexplainable Ones
• Abstract summary: The paper argues that the truly valuable capabilities of large language models (LLMs) reside precisely in the part that cannot be fully captured by human-readable discrete rules.<n>The paper discusses implications for interpretability research, AI safety, and scientific interpretability.
• Score: 0.786119752211706
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper proposes and argues for a counterintuitive thesis: the truly valuable capabilities of large language models (LLMs) reside precisely in the part that cannot be fully captured by human-readable discrete rules. The core argument is a proof by contradiction via expert system equivalence: if the full capabilities of an LLM could be described by a complete set of human-readable rules, then that rule set would be functionally equivalent to an expert system; but expert systems have been historically and empirically demonstrated to be strictly weaker than LLMs; therefore, a contradiction arises -- the capabilities of LLMs that exceed those of expert systems are exactly the capabilities that cannot be rule-encoded. This thesis is further supported by the Chinese philosophical concept of Wu (sudden insight through practice), the historical failure of expert systems, and a structural mismatch between human cognitive tools and complex systems. The paper discusses implications for interpretability research, AI safety, and scientific epistemology.

Related papers

• Human-AI Co-Interpretation for Responsible AI: A Hermeneutic Perspective [4.832750949021042]
  Hermeneutic understanding is distinguished from token-prediction-based text generation.<n>Hermeneutic understanding is reserved for accountable human interpreters situated in disciplinary historical-linguistic traditions.<n>Hermeneutic understanding is distinguished from token-prediction-based text generation.
  arXiv  Detail & Related papers  (2026-08-02T18:19:13Z)
• Position: agentic AI orchestration should be Bayes-consistent [92.04373485710285]
  This paper argues that coherent decision-making requires Bayesian principles at the orchestration level of the agentic system.<n>It provides examples and design patterns to illustrate how beliefs and utility-aware policies can improve agentic AI orchestration.
  arXiv  Detail & Related papers  (2026-05-01T15:43:43Z)
• Evaluating the Formal Reasoning Capabilities of Large Language Models through Chomsky Hierarchy [62.32144504442516]
  It is unknown whether SOTA LLMs can grasp the structured, hierarchical complexity of formal languages.<n>ChomskyBench is a benchmark for systematically evaluating LLMs through the lens of Chomsky Hierarchy.<n>ChomskyBench is composed of a comprehensive suite of language recognition and generation tasks designed to test capabilities at each level.
  arXiv  Detail & Related papers  (2026-04-03T04:06:39Z)
• Ambiguity Collapse by LLMs: A Taxonomy of Epistemic Risks [5.45414933764086]
  Large language models (LLMs) are increasingly used to make sense of ambiguous, open-textured, value-laden terms.<n>This paper introduces ambiguity collapse: a phenomenon that occurs when an LLM encounters a term that genuinely admits multiple legitimate interpretations, yet produces a singular resolution.
  arXiv  Detail & Related papers  (2026-03-06T01:23:17Z)
• The Epistemological Consequences of Large Language Models: Rethinking collective intelligence and institutional knowledge [0.0]
  We develop a theory of rationality as distributed across human collectives, using dual process theory as background.<n>We distinguish internalist justification, defined as reflective understanding of why a proposition is true, from externalist justification, defined as reliable transmission of truths.<n>We argue that LLMs approximate externalist reliabilism because they can reliably transmit information whose justificatory basis is established elsewhere, but they do not themselves possess reflective justification.
  arXiv  Detail & Related papers  (2025-12-22T16:52:37Z)
• Thinking Longer, Not Always Smarter: Evaluating LLM Capabilities in Hierarchical Legal Reasoning [11.255428720705204]
  We propose a framework that decomposes the process of identifying significant distinctions between cases into three-stage reasoning tasks.<n>Our framework models cases using factual predicates called factors, organizes them into a legal knowledge hierarchy, and defines verifiable rules for identifying distinctions.<n>We find that models consistently expend more computational resources on incorrect responses than correct ones, suggesting that "thinking longer" does not always mean "thinking smarter"
  arXiv  Detail & Related papers  (2025-10-09T18:15:28Z)
• Make LLMs better zero-shot reasoners: Structure-orientated autonomous reasoning [52.83539473110143]
  We introduce a novel structure-oriented analysis method to help Large Language Models (LLMs) better understand a question. To further improve the reliability in complex question-answering tasks, we propose a multi-agent reasoning system, Structure-oriented Autonomous Reasoning Agents (SARA) Extensive experiments verify the effectiveness of the proposed reasoning system. Surprisingly, in some cases, the system even surpasses few-shot methods.
  arXiv  Detail & Related papers  (2024-10-18T05:30:33Z)
• What's in an embedding? Would a rose by any embedding smell as sweet? [0.0]
  Large Language Models (LLMs) are often criticized for lacking true "understanding" and the ability to "reason" with their knowledge. We suggest that LLMs do develop a kind of empirical "understanding" that is "geometry"-like, which seems adequate for a range of applications in NLP. To overcome these limitations, we suggest that LLMs should be integrated with an "algebraic" representation of knowledge that includes symbolic AI elements.
  arXiv  Detail & Related papers  (2024-06-11T01:10:40Z)
• Should We Fear Large Language Models? A Structural Analysis of the Human Reasoning System for Elucidating LLM Capabilities and Risks Through the Lens of Heidegger's Philosophy [0.0]
  This study investigates the capabilities and risks of Large Language Models (LLMs) It uses the innovative parallels between the statistical patterns of word relationships within LLMs and Martin Heidegger's concepts of "ready-to-hand" and "present-at-hand" Our findings reveal that while LLMs possess the capability for Direct Explicative Reasoning and Pseudo Rational Reasoning, they fall short in authentic rational reasoning and have no creative reasoning capabilities.
  arXiv  Detail & Related papers  (2024-03-05T19:40:53Z)
• Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs [87.34281749422756]
  Large language models (LLMs) have achieved impressive human-like performance across various reasoning tasks. However, their mastery of underlying inferential rules still falls short of human capabilities. We propose a logic scaffolding inferential rule generation framework, to construct an inferential rule base, ULogic.
  arXiv  Detail & Related papers  (2024-02-18T03:38:51Z)
• Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners [75.85554779782048]
  Large Language Models (LLMs) have excited the natural language and machine learning community over recent years. Despite of numerous successful applications, the underlying mechanism of such in-context capabilities still remains unclear. In this work, we hypothesize that the learned textitsemantics of language tokens do the most heavy lifting during the reasoning process.
  arXiv  Detail & Related papers  (2023-05-24T07:33:34Z)
• From LSAT: The Progress and Challenges of Complex Reasoning [56.07448735248901]
  We study the three challenging and domain-general tasks of the Law School Admission Test (LSAT), including analytical reasoning, logical reasoning and reading comprehension. We propose a hybrid reasoning system to integrate these three tasks and achieve impressive overall performance on the LSAT tests.
  arXiv  Detail & Related papers  (2021-08-02T05:43:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.