Measuring Reasoning Trace Legibility: Can Those Who Understand Teach?

• URL: http://arxiv.org/abs/2603.20508v1
• Date: Fri, 20 Mar 2026 21:23:29 GMT
• Title: Measuring Reasoning Trace Legibility: Can Those Who Understand Teach?
• Abstract summary: We evaluate 90k traces from 12 Reasoning Language Models (RLMs) for the quality of their reasoning traces.<n>We find that the reasoning traces of the highest-performing models rank among the lowest for legibility.
• Score: 14.267421398274571
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language models are increasingly being trained to "reason" before answering users' queries, outputting hundreds or even thousands of tokens worth of deliberation before their final answer. While the main intention of reasoning is to improve models' ability to arrive at a correct answer, we argue that these models should be assessed for the legibility of their reasoning traces in addition to the correctness of their final answers. In this paper, we evaluate 90k traces from 12 Reasoning Language Models (RLMs) for the quality of their reasoning traces. We introduce the concept of transfer utility, which assesses how useful an RLM's reasoning traces are for guiding a weaker, non-reasoning model toward arriving at the correct answer. We find that the reasoning traces of the highest-performing models rank among the lowest for legibility. Furthermore, we uncover tensions between efficiency-based measurements of legibility (such as trace length) and transfer utility. These tensions establish a legibility Pareto frontier, and we demonstrate that an RLM's ability to output highly legible traces can be a task- and audience-dependent goal. Crucially, we find that reward models used to train RLMs do not intrinsically reward legibility. Together, these metrics and the findings they surface chart a path towards scaffolding reasoning traces for a multi-agent future.

Related papers

• Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoning [48.96460246231164]
  Reasoning traces from chain-of-thought models appear to offer a legible window into how a model arrives at its answer.<n>We operationalize the importance of a reasoning step as its advantage.<n>We find that sufficiently capable LLMs can outperform a prevalence baseline but fall well short of a noise ceiling.
  arXiv  Detail & Related papers  (2026-09-03T17:59:08Z)
• Probing the Trajectories of Reasoning Traces in Large Language Models [4.599673637363014]
  We propose a protocol to probe the trajectories of reasoning traces in large language models.<n>We find that accuracy and decision commitment consistently increase as the percentage of provided reasoning tokens grows.<n>We show that trajectory probing provides diagnostics for efficient and safer deployment of reasoning models.
  arXiv  Detail & Related papers  (2026-01-30T16:45:16Z)
• Visual Reasoning Tracer: Object-Level Grounded Reasoning Benchmark [112.46338388724116]
  We introduce the Visual Reasoning Tracer (VRT) task, which requires models to not only localize the target object but also explicitly predict the intermediate objects that form the reasoning path.<n>To advance research in this area, we contribute: (1) VRT-Bench, a human-annotated benchmark for evaluating visual reasoning; (2) a new metric for assessing the quality of reasoning traces; and (3) VRT-80k, a large-scale dataset for reasoning model training.
  arXiv  Detail & Related papers  (2025-12-04T18:55:34Z)
• Reasoning Models Sometimes Output Illegible Chains of Thought [0.0]
  Language models trained via outcome-based reinforcement learning (RL) to reason using chain-of-thought (CoT) have shown remarkable performance.<n>We study CoT legibility across 14 reasoning models, finding that RL often causes reasoning to become illegible to both humans and AI monitors.<n>We show that models use illegible reasoning to reach correct answers (accuracy dropping by 53% when forced to use only legible portions) yet find no correlation between legibility and performance when resampling.
  arXiv  Detail & Related papers  (2025-10-31T10:16:35Z)
• Verifying Large Language Models' Reasoning Paths via Correlation Matrix Rank [71.09032766271493]
  Large language models (LLMs) are prone to errors and hallucinations.<n>How to check their outputs effectively and efficiently has become a critical problem in their applications.
  arXiv  Detail & Related papers  (2025-10-28T11:01:10Z)
• Your Models Have Thought Enough: Training Large Reasoning Models to Stop Overthinking [50.97239453902612]
  Large Reasoning Models (LRMs) have achieved impressive performance on challenging tasks, yet their deep reasoning often incurs substantial computational costs.<n>Inspired by Evidence Accumulation Models, we find that LRMs have accumulated sufficient information early in reasoning, making further reasoning steps redundant.<n>We propose Just-Enough Thinking (JET), which trains models to proactively terminate unnecessary reasoning.
  arXiv  Detail & Related papers  (2025-09-27T16:25:06Z)
• In Their Own Words: Reasoning Traces Tailored for Small Models Make Them Better Reasoners [12.995634497832027]
  Transferring reasoning capabilities from larger language models to smaller ones often fails counterintuitively.<n>We identify that this failure stems from distributional misalignment: reasoning traces from larger models contain tokens that are low probability under the student's distribution.<n>We propose Reverse Speculative Decoding (RSD), a mechanism for generating student-friendly reasoning traces.
  arXiv  Detail & Related papers  (2025-09-26T11:40:32Z)
• Mitigating Spurious Correlations Between Question and Answer via Chain-of-Thought Correctness Perception Distillation [25.195244084313114]
  Chain-of-Thought Correctness Perception Distillation (CoPeD) aims to improve the reasoning quality of the student model.<n>CoPeD encourages the student model to predict answers based on correct rationales and revise them when they are incorrect.
  arXiv  Detail & Related papers  (2025-09-06T05:33:17Z)
• Do Cognitively Interpretable Reasoning Traces Improve LLM Performance? [15.970162088175593]
  Chain-of-Thought (CoT) traces generate intermediate reasoning traces before producing an answer.<n>A common but often implicit assumption is that CoT traces should be semantically meaningful and interpretable to the end user.<n>We investigate this question in the Open Book Question-Answering domain by supervised fine-tuning LLaMA and Qwen models.
  arXiv  Detail & Related papers  (2025-08-21T23:48:50Z)
• Socratic-MCTS: Test-Time Visual Reasoning by Asking the Right Questions [100.41062461003389]
  We show that framing reasoning as a search process helps the model "connect the dots" between fragmented knowledge and produce extended reasoning traces in non-reasoning models.<n>We evaluate our method across three benchmarks and observe consistent improvements.
  arXiv  Detail & Related papers  (2025-06-10T15:51:16Z)
• RM-R1: Reward Modeling as Reasoning [81.50471199906738]
  Reasoning Reward Models (ReasRMs) formulate reward modeling as a reasoning task.<n>We propose a reasoning-oriented training pipeline and train a family of ReasRMs, RM-R1.<n>Our models achieve state-of-the-art performance across three reward model benchmarks on average.
  arXiv  Detail & Related papers  (2025-05-05T06:11:12Z)
• Understanding Reasoning Ability of Language Models From the Perspective of Reasoning Paths Aggregation [110.71955853831707]
  We view LMs as deriving new conclusions by aggregating indirect reasoning paths seen at pre-training time. We formalize the reasoning paths as random walk paths on the knowledge/reasoning graphs. Experiments and analysis on multiple KG and CoT datasets reveal the effect of training on random walk paths.
  arXiv  Detail & Related papers  (2024-02-05T18:25:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.