Related papers: ThinkProbe: Beyond Accuracy -- Structural Profiling of Open-Ended LLM Reasoning Traces via Non-Generative Thought Graphs

ThinkProbe: Beyond Accuracy -- Structural Profiling of Open-Ended LLM Reasoning Traces via Non-Generative Thought Graphs

URL: http://arxiv.org/abs/2606.29067v1
Date: Sat, 27 Jun 2026 19:56:55 GMT
Title: ThinkProbe: Beyond Accuracy -- Structural Profiling of Open-Ended LLM Reasoning Traces via Non-Generative Thought Graphs
Abstract summary: ThinkProbe is a framework for structural analysis of reasoning traces.<n>It converts each trace into a Thought Graph with cycles, 8 node types, and 6 edge types.<n>ThinkProbe reveals that reasoning structure is a stable, model-level property.
Score: 4.5194910739268925
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
Abstract: We present ThinkProbe, a framework for structural analysis of LLM reasoning traces. ThinkProbe converts each trace into a Thought Graph a directed graph with cycles, 8 node types, and 6 edge types and derives a 19-metric five-dimensional cognitive profile (5D-CP: Breadth, Depth, Structure, Metacognitive, Efficiency) through a fully non-generative pipeline combining rule-based segmentation and discriminative semantic linking. Applied to 4{,}200 traces from 7 native reasoning models across 200 open-ended questions and 10 cognitive domains, ThinkProbe reveals that reasoning structure is a stable, model-level property: between-model variance exceeds between-domain variance by up to fourfold across four of five cognitive dimensions, with Structure showing genuine sensitivity to question domain, exposing qualitatively distinct cognitive profiles invisible to accuracy-based evaluation.

Related papers

MindTopo: Can Foundation Models Reason in Topological Space? [52.56750995613252]
We introduce MindTopo, a benchmark of topological intuition across five properties grounded in cognitive science and formal topology.<n>MindTopo evaluates each property at two cognitive levels. Reasoning asks a model to identify topological relations or infer how they change.<n>Every MLLM performs better on reasoning than on planning, and the best-performing model remains far below observed human performance.
arXiv Detail & Related papers (2026-09-10T17:54:32Z)
CogArena: A Multimethod Evaluation of Cognitive Ability Structure in Large Language Models [3.991518853474585]
We introduce CogArena, a benchmark for determining when cognitive-task scores warrant dimensional labels.<n>Across 55 open-weight models, nearly all paradigm correlations are positive and a common axis explains about half the variance.<n>The within-grouping advantage is small, scoring-sensitive, and uncertain across model families.
arXiv Detail & Related papers (2026-07-27T18:56:13Z)
RoboPIN: Grounded Embodied Reasoning via Pinned Chain-of-Thought [50.36951914894845]
Embodied reasoning requires models to perceive task-relevant objects and spaces in physical environments.<n>Current vision-language models rely on text-only or coordinate-augmented chain-of-thought.<n>We propose Pinned Chain-of-Thought, a structured reasoning paradigm that pins every reasoning step to visual evidence.
arXiv Detail & Related papers (2026-06-14T11:30:42Z)
ComBench: A Benchmark for Rigorous Proof Reasoning and Constructive Realization in Olympiad-Level Combinatorics [95.15900681087088]
We introduce ComBench, an Olympiad-level benchmark for diagnosing and the reasoning capabilities of large language models.<n>ComBench contains 100 human-annotated competition-level problems organized around two complementary settings.<n>Experiments show that the strongest model reaches 65.4% overall Avg. and 75.3% overall Best@4.
arXiv Detail & Related papers (2026-06-09T06:50:15Z)
AI Mental Models: Learned Intuition and Deliberation in a Bounded Neural Architecture [0.0]
This paper asks whether a bounded neural architecture can exhibit a meaningful division of labor between intuition and deliberation on a syllogistic reasoning benchmark.<n>Experiment 1 evaluates a direct neural baseline for predicting full 9-way human response distributions under 5-fold cross-validation.<n>Experiment 2 introduces a bounded dual-path architecture with separate intuition and deliberation pathways, motivated by computational mental-model theory.
arXiv Detail & Related papers (2026-03-23T20:44:38Z)
I Know What I Don't Know: Latent Posterior Factor Models for Multi-Evidence Probabilistic Reasoning [0.0]
We introduce Latent Posterior Factors (LPF), a framework that transforms latent posteriors into soft likelihood factors for Sum-Product Network (SPN) inference.<n>We instantiate LPF as LPF-SPN (structured factor-based inference) and LPF-Learned (end-to-end learned aggregation)<n>Across eight domains (seven synthetic and the FEVER benchmark), LPF-SPN achieves high accuracy (up to 97.8%), low calibration error (ECE 1.4%), and strong probabilistic fit.
arXiv Detail & Related papers (2026-03-13T10:05:14Z)
TACIT Benchmark: A Programmatic Visual Reasoning Benchmark for Generative and Discriminative Models [0.0]
TACIT Benchmark is a programmatic visual reasoning benchmark comprising 10 tasks across 6 reasoning domains.<n>The benchmark provides dual-track evaluation: a generative track in which models must produce solution images verified through deterministic computer-vision pipelines, and a discriminative track offering five-way multiple choice with structurally plausible near-miss distractors.
arXiv Detail & Related papers (2026-02-27T11:45:26Z)
From Chains to Graphs: Self-Structured Reasoning for General-Domain LLMs [52.0811904328273]
Real-world reasoning requires integrating multiple premises and solving subproblems in parallel.<n>Recent approaches rely on externally provided graphs and do not explore how LLMs can construct and use their own graph-structured reasoning.<n>We propose Self-Graph Reasoning (SGR), a framework that enables LLMs to explicitly represent their reasoning process as a structured graph before producing the final answer.
arXiv Detail & Related papers (2026-01-07T05:27:41Z)
BRIDGE: Building Representations In Domain Guided Program Verification [67.36686119518441]
BRIDGE decomposes verification into three interconnected domains: Code, Specifications, and Proofs.<n>We show that this approach substantially improves both accuracy and efficiency beyond standard error feedback methods.
arXiv Detail & Related papers (2025-11-26T06:39:19Z)
Cognitive Foundations for Reasoning and Their Manifestation in LLMs [63.12951576410617]
Large language models (LLMs) solve complex problems yet fail on simpler variants, suggesting they achieve correct outputs through mechanisms fundamentally different from human reasoning.<n>We synthesize cognitive science research into a taxonomy of 28 cognitive elements spanning reasoning invariants, meta-cognitive controls, representations for organizing reasoning & knowledge, and transformation operations.<n>We develop test-time reasoning guidance that automatically scaffold successful structures, improving performance by up to 66.7% on complex problems.
arXiv Detail & Related papers (2025-11-20T18:59:00Z)
Emergent Cognitive Convergence via Implementation: A Structured Loop Reflecting Four Theories of Mind [0.0]
We report a structural convergence among four influential theories of mind.<n>The convergence emerges unintentionally within a practical AI architecture known as Agentic Flow.<n>This paper proposes that intelligent architectures may evolve toward shared structural patterns shaped by practical constraints.
arXiv Detail & Related papers (2025-07-22T02:54:45Z)
ProtoReasoning: Prototypes as the Foundation for Generalizable Reasoning in LLMs [54.154593699263074]
ProtoReasoning is a framework that enhances the reasoning ability of Large Reasoning Models.<n>ProtoReasoning transforms problems into corresponding prototype representations.<n>ProtoReasoning achieves 4.7% improvement over baseline models on logical reasoning.
arXiv Detail & Related papers (2025-06-18T07:44:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.