ReasonOps: Operator Segmentation for LLM Reasoning Traces

• URL: http://arxiv.org/abs/2605.29192v1
• Date: Thu, 28 May 2026 00:08:35 GMT
• Title: ReasonOps: Operator Segmentation for LLM Reasoning Traces
• Abstract summary: Chain-of-thought traces from large reasoning models can span tens of thousands of tokens.<n>We develop ReasonOps, an unsupervised, expressive method for annotating chain-of-thought traces.<n>We analyze 44,662 traces from 12 thinking LLMs spanning 6 families across 8 reasoning benchmarks.
• Score: 29.17171274711201
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Chain-of-thought traces from large reasoning models can span tens of thousands of tokens, yet we lack a vocabulary for describing their internal structure. Previous methods developed to analyze chain-of-thought traces are either too rigid or not expressive enough, failing to capture features across domains and models. To remedy this, we develop ReasonOps, an unsupervised, expressive method for annotating chain-of-thought traces, providing succinct universal operators. Using ReasonOps, we analyze 44,662 traces from 12 thinking LLMs spanning 6 families across 8 reasoning benchmarks and discover that they share a common compositional structure: 7 recurring reasoning operators -- discourse-level moves such as backtracking, inferring, and hypothesizing -- that emerge from unsupervised clustering of sentence-initial 3-token pivots. These operators appear across every model family and benchmark domain, confirmed by three independent LLM judges who classify held-out samples at 70 -76% accuracy. We analyze the structure of operators on easy vs. hard problems, revealing that reflective operators are more helpful on hard problems and harm performance on easy problems. Operator sequences are highly model-identifying: a classifier trained on operator distributions alone recovers the source model with macro-AUC, revealing that each model family has a distinctive reasoning fingerprint. Structural operator features predict within-problem answer correctness well above baselines. Classifiers built on these operators reach WP-AUC and on AIME specifically. ReasonOps further enables early quality estimation well before the trace completes: we predict at WP-AUC for only 50% of the trace. The ReasonOps pipeline is unsupervised and annotation-free, enabling deep insights into LLM reasoning traces as well as strong downstream results on model identification and correctness prediction.

Related papers

• Mechanistic Interpretability of Structure-Aware Numerical Reasoning in LLaMA 3.1 8B [51.10897657510313]
  Large language models (LLMs) exhibit strong numerical sequence modeling capabilities and show promise in time-series prediction.<n>We investigate Llama 3.1-8B from a mechanistic interpretability point of view.<n>We find that Llama displays strong performance on our tasks suggesting that it can pick up on the underlying structure.
  arXiv  Detail & Related papers  (2026-08-19T01:09:51Z)
• The Tell-Tale Trace: Detecting Reasoning Failures in LLMs Using Chain-of-Thought Dynamics [0.9668407688201359]
  Chain-of-thought (CoT) reasoning improves large language model (LLM) performance.<n>Existing approaches that leverage verbalized CoTs to monitor reasoning correctness largely evaluate the semantic correctness or consistency of individual intermediate steps.<n>We show that capability failures can manifest as distributed, task-dependent changes in the structure of visible reasoning.
  arXiv  Detail & Related papers  (2026-08-04T08:04:36Z)
• Training Large Language Models for Self-Explanation Faithfulness [10.933553638236036]
  We propose a Reinforcement Learning (RL) method to directly optimize the faithfulness of self-explanations.<n>We investigate whether models can be trained to accurately detect factors that affect their decisions.<n>We show that models can be trained to implicitly identify influential factors and disclose them.
  arXiv  Detail & Related papers  (2026-07-23T09:20:38Z)
• Shorthand for Thought: Compressing LLM Reasoning via Entropy-Guided Supertokens [4.0704009036918025]
  Reasoning in Large Language Models incurs significant inference-time compute.<n>We show that reasoning tokens split into two functional types: low-entropy textitstructural tokens (recurring phrases that scaffold the reasoning process) and higher-entropy textitorganic tokens (problem-specific content that drives toward a solution)<n>We derive textitsupertokens that capture frequent structural patterns, then teach the model to adopt them via supervised fine-tuning.
  arXiv  Detail & Related papers  (2026-04-29T07:06:43Z)
• CoT-Seg: Rethinking Segmentation with Chain-of-Thought Reasoning and Self-Correction [50.67483317563736]
  This paper aims to explore a system that can think step-by-step, look up information if needed, generate results, self-evaluate its own results, and refine the results.<n>We introduce CoT-Seg, a training-free framework that rethinks reasoning segmentation by combining chain-of-thought reasoning with self-correction.
  arXiv  Detail & Related papers  (2026-01-24T11:41:54Z)
• Reasoning Beyond Chain-of-Thought: A Latent Computational Mode in Large Language Models [39.5490415037017]
  Chain-of-Thought (CoT) prompting has improved the reasoning performance of large language models (LLMs)<n>It remains unclear why it works and whether it is the unique mechanism for triggering reasoning in large language models.
  arXiv  Detail & Related papers  (2026-01-12T23:01:21Z)
• Verifying Large Language Models' Reasoning Paths via Correlation Matrix Rank [71.09032766271493]
  Large language models (LLMs) are prone to errors and hallucinations.<n>How to check their outputs effectively and efficiently has become a critical problem in their applications.
  arXiv  Detail & Related papers  (2025-10-28T11:01:10Z)
• Do Cognitively Interpretable Reasoning Traces Improve LLM Performance? [15.970162088175593]
  Chain-of-Thought (CoT) traces generate intermediate reasoning traces before producing an answer.<n>A common but often implicit assumption is that CoT traces should be semantically meaningful and interpretable to the end user.<n>We investigate this question in the Open Book Question-Answering domain by supervised fine-tuning LLaMA and Qwen models.
  arXiv  Detail & Related papers  (2025-08-21T23:48:50Z)
• SEAL: Steerable Reasoning Calibration of Large Language Models for Free [58.931194824519935]
  Large Language Models (LLMs) have demonstrated compelling capabilities for complex reasoning tasks via the extended chain-of-thought (CoT) reasoning mechanism.<n>Recent studies reveal substantial redundancy in the CoT reasoning traces, which negatively impacts model performance.<n>We introduce SEAL, a training-free approach that seamlessly calibrates the CoT process, improving accuracy while demonstrating significant efficiency gains.
  arXiv  Detail & Related papers  (2025-04-07T02:42:07Z)
• The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models [69.798277882245]
  We introduce Unsupervised Prefix Fine-Tuning (UPFT) to enhance large language models' reasoning efficiency.<n>UPFT removes the need for labeled data or exhaustive sampling.<n> Experiments show that UPFT matches the performance of supervised methods.
  arXiv  Detail & Related papers  (2025-03-04T18:56:03Z)
• Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding [74.31981011985681]
  Large language models (LLMs) have shown impressive capabilities, but still struggle with complex reasoning tasks requiring multiple steps. We introduce LaTent Reasoning Optimization (LaTRO), a principled framework that formulates reasoning as sampling from a latent distribution. We validate LaTRO through experiments on GSM8K and ARC-Challenge datasets using multiple model architectures.
  arXiv  Detail & Related papers  (2024-11-06T22:02:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.