Playing Psychic: Using Thought Trees to Predict Reasoning Models Accuracy on Coding Tasks

• URL: http://arxiv.org/abs/2604.16931v1
• Date: Sat, 18 Apr 2026 09:30:36 GMT
• Title: Playing Psychic: Using Thought Trees to Predict Reasoning Models Accuracy on Coding Tasks
• Abstract summary: We study frontier reasoning models to understand their performance on real-world coding benchmarks.<n>We propose structured thought-trees as means to represent reasoning traces.<n>We show that flagging and retrying structurally anomalous traces based on the extracted features yields consistent gains at lower complexity levels.
• Score: 7.675768582345005
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in large language models (LLMs) have shown that test-time scaling can substantially improve model performance on complex tasks, particularly in the coding domain. Under this paradigm, models use a larger token budget during inference to generate intermediate reasoning traces before producing a final answer. However, current evaluations primarily rely on competitive programming benchmarks, which may not capture the full range of reasoning abilities. In this work, we perform a systematic study of frontier reasoning models to understand their performance on real-world coding benchmarks. To gain more insights into the performance of such models, we devise a programmatic way to {\em automatically generate} coding tasks of arbitrary difficulty and structure from existing benchmarks. Using this framework, our analysis reveals that the structure of a reasoning trace, not just its contents, is a strong predictor of correctness. Motivated by this, we propose structured thought-trees as means to represent reasoning traces. To illustrate their use, we train a lightweight classifier on features extracted from thought-trees to predict trace correctness, and demonstrate that flagging and retrying structurally anomalous traces based on the extracted features yields consistent gains at lower complexity levels.

Related papers

• Forethought: Verifiable Reasoning from Neurosymbolic Primitive Programming [0.764671395172401]
  Forethought is a neurosymbolic reasoning system that treats reasoning as an explicit, verifiable program.<n>It improves base-model accuracy by about 30% relative and outperforms vanilla prompting, reinforcement learning scaffolds, and prompt-evolution methods.
  arXiv  Detail & Related papers  (2026-07-05T03:20:23Z)
• What properties of reasoning supervision are associated with improved downstream model quality? [3.5354214473997225]
  We propose a suite of quantitative measures and evaluate their predictive power by fine-tuning 8B and 11B models on semantically distinct variants of a Polish reasoning dataset.<n>Our analysis reveals that these intrinsic metrics demonstrate strong and significant correlations with downstream model performance.
  arXiv  Detail & Related papers  (2026-05-13T10:04:38Z)
• TFRBench: A Reasoning Benchmark for Evaluating Forecasting Systems [52.91956121737963]
  We introduce TFRBench, the first benchmark designed to evaluate the reasoning capabilities of forecasting systems.<n>We propose a systematic multi-agent framework that utilizes an iterative verification loop to synthesize numerically grounded reasoning traces.<n>Our evaluation confirms that this reasoning is causally effective; useful for evaluation; and prompting LLMs with our generated traces significantly improves forecasting accuracy.
  arXiv  Detail & Related papers  (2026-04-07T03:04:45Z)
• Decoding the Critique Mechanism in Large Reasoning Models [50.821607345799386]
  Large Reasoning Models (LRMs) exhibit backtracking and self-verification mechanisms that enable them to revise intermediate steps and reach correct solutions.<n>We investigate how current LRMs recover from errors by inserting arithmetic mistakes in their intermediate reasoning steps.<n>We find that despite the error propagating through the chain-of-thought, the model still reaches the correct final answer.
  arXiv  Detail & Related papers  (2026-03-17T10:03:30Z)
• Fluid Representations in Reasoning Models [91.77876704697779]
  We present a mechanistic analysis of how QwQ-32B processes abstract structural information.<n>We find that QwQ-32B gradually improves its internal representation of actions and concepts during reasoning.
  arXiv  Detail & Related papers  (2026-02-04T18:34:50Z)
• Combating Spurious Correlations in Graph Interpretability via Self-Reflection [4.81017678027464]
  Interpretable graph learning is a popular research topic in machine learning.<n>One of the most challenging is the Spurious-Motif benchmark, introduced at ICLR 2022.<n>We propose a self-reflection framework that can be integrated with existing interpretable graph learning methods.
  arXiv  Detail & Related papers  (2026-01-16T06:31:16Z)
• Can an LLM Induce a Graph? Investigating Memory Drift and Context Length [11.214847796972705]
  Recently proposed evaluation benchmarks aim to characterize the effective context length and the forgetting tendencies of large language models (LLMs)<n>We argue for evaluating these models on more complex reasoning tasks that require them to induce structured relational knowledge from the text.<n>Our findings reveal that LLMs begin to exhibit memory drift and contextual forgetting at much shorter effective lengths when tasked with this form of relational reasoning.
  arXiv  Detail & Related papers  (2025-10-04T01:56:07Z)
• Adaptive Test-Time Reasoning via Reward-Guided Dual-Phase Search [62.1546099504045]
  We propose a dual-phase test-time scaling framework that separates reasoning into planning and execution.<n>Specifically, we decompose reasoning trajectories and develop reward models for each phase, enabling the search to explore and prune plans and executions separately.<n> Experiments on both mathematical reasoning and code generation benchmarks demonstrate that our approach consistently improves accuracy while reducing computation redundant.
  arXiv  Detail & Related papers  (2025-09-29T19:27:23Z)
• Accelerated Test-Time Scaling with Model-Free Speculative Sampling [58.69141724095398]
  We introduce STAND (STochastic Adaptive N-gram Drafting), a novel model-free speculative decoding approach.<n>We show that STAND reduces inference latency by 60-65% compared to standard autoregressive decoding.<n>As a model-free approach, STAND can be applied to any existing language model without additional training.
  arXiv  Detail & Related papers  (2025-06-05T07:31:18Z)
• Enhancing Performance of Explainable AI Models with Constrained Concept Refinement [10.241134756773228]
  Trade-off between accuracy and interpretability has long been a challenge in machine learning (ML)<n>In this paper, we investigate the impact of deviations in concept representations and propose a novel framework to mitigate these effects.<n>Compared to existing explainable methods, our approach not only improves prediction accuracy while preserving model interpretability across various large-scale benchmarks but also achieves this with significantly lower computational cost.
  arXiv  Detail & Related papers  (2025-02-10T18:53:15Z)
• Causality can systematically address the monsters under the bench(marks) [64.36592889550431]
  Benchmarks are plagued by various biases, artifacts, or leakage.<n>Models may behave unreliably due to poorly explored failure modes.<n> causality offers an ideal framework to systematically address these challenges.
  arXiv  Detail & Related papers  (2025-02-07T17:01:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.