Understanding Chain-of-Thought Effectiveness in Code Generation: An Empirical and Information-Theoretic Analysis

• URL: http://arxiv.org/abs/2512.09679v1
• Date: Wed, 10 Dec 2025 14:25:46 GMT
• Title: Understanding Chain-of-Thought Effectiveness in Code Generation: An Empirical and Information-Theoretic Analysis
• Authors: Naizhu Jin, Zhong Li, Guang Yang, Tian Zhang, Qingkai Zeng,
• Abstract summary: Chain-of-Thought (CoT) prompting helps large language models (LLMs) achieve strong performance on code generation.<n>We present a systematic empirical and information-theoretic study of CoT effectiveness in neural code generation.
• Score: 13.64149870296543
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) achieve strong performance on code generation, but the mechanisms by which Chain-of-Thought (CoT) prompting helps remain unclear. We present a systematic empirical and information-theoretic study of CoT effectiveness in neural code generation, evaluating five paradigms (Zero-Shot, Zero-Shot CoT, Self-Planning, Structured CoT, Reasoning-CoT) across six Python benchmarks, a multilingual benchmark with 12 programming languages, and six models from 7B to 480B parameters, using conditional mutual information $I(Y;C|X)$ as a conceptual lens. Our results show that externally guided CoT consistently outperforms direct generation, with structured methods improving Pass@1 by 5--12\% on average while using substantially fewer tokens than reflective reasoning, and that CoT benefits depend on language type systems and model capacity. We further find that reasoning \emph{quality} is critical: high-quality structured CoT from strong generators yields significantly higher accuracy than lightweight alternatives with the same template, whereas naive Zero-Shot CoT can even degrade performance. These findings provide practical guidance for choosing CoT strategies based on model capacity, language characteristics, and task complexity.

Related papers

• D-COT: Disciplined Chain-of-Thought Learning for Efficient Reasoning in Small Language Models [0.0]
  Disciplined Chain-of-Thought (D-CoT) is a framework that enforces a structured reasoning process using control tags.<n>D-CoT suppresses reasoning drift and simultaneously achieves token reduction and performance improvement.
  arXiv  Detail & Related papers  (2026-02-25T11:08:38Z)
• From Brute Force to Semantic Insight: Performance-Guided Data Transformation Design with LLMs [48.83701310501069]
  Large language models (LLMs) have achieved notable performance in code synthesis.<n>We introduce a performance-aware, closed-loop solution that enables LLMs to autonomously engineer optimal transformations.<n>We fine-tune LLMs with Low-Rank Adaptation on a novel repository of more than 6,000 empirically evaluated PyTorch augmentation functions.
  arXiv  Detail & Related papers  (2026-01-07T11:13:02Z)
• Teaching Language Models to Reason with Tools [73.21700643314917]
  We present emphHint-Engineering, a new data synthesis strategy that strategically injects diverse hints at optimal points within reasoning paths.<n>CoRT significantly enhances efficiency, reducing token usage by approximately 30% for the 32B model and 50% for the 1.5B model.
  arXiv  Detail & Related papers  (2025-10-23T08:41:44Z)
• Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning [65.20602712957725]
  Caco is a novel framework that automates the synthesis of high-quality, verifiable, and diverse instruction-CoT reasoning data.<n>Our work establishes a paradigm for building self-sustaining, trustworthy reasoning systems without human intervention.
  arXiv  Detail & Related papers  (2025-10-05T07:59:24Z)
• CoT Vectors: Transferring and Probing the Reasoning Mechanisms of LLMs [33.63911145333626]
  Chain-of-Thought prompting has emerged as a powerful approach to enhancing the reasoning capabilities of Large Language Models.<n>Existing implementations, such as in-context learning and fine-tuning, remain costly and inefficient.<n>We introduce CoT Vectors, compact representations that encode task-general, multi-step reasoning knowledge.
  arXiv  Detail & Related papers  (2025-10-01T06:58:23Z)
• Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models [9.282278040339138]
  Chain-of-Thought (CoT) technique has proven effective in improving the performance of large language models (LLMs) on complex reasoning tasks.<n>We make a preliminary observation that the monotonicity of token probability distributions may be correlated with the gains achieved through CoT reasoning.<n>We propose two indicators based on the token probability distribution to assess CoT effectiveness across different tasks.
  arXiv  Detail & Related papers  (2025-06-06T11:53:27Z)
• Accelerated Test-Time Scaling with Model-Free Speculative Sampling [58.69141724095398]
  We introduce STAND (STochastic Adaptive N-gram Drafting), a novel model-free speculative decoding approach.<n>We show that STAND reduces inference latency by 60-65% compared to standard autoregressive decoding.<n>As a model-free approach, STAND can be applied to any existing language model without additional training.
  arXiv  Detail & Related papers  (2025-06-05T07:31:18Z)
• Enhancing Large Language Model Efficiencyvia Symbolic Compression: A Formal Approach Towards Interpretability [3.9122242678047456]
  Large language models (LLMs) face significant token efficiency bottlenecks in code generation and logical reasoning tasks.<n>This paper proposes a formal framework based on symbolic compression,integrating logic, information-theoretic optimal encoding, and context-aware inference techniques.
  arXiv  Detail & Related papers  (2025-01-30T06:40:52Z)
• Tensor Product Attention Is All You Need [61.3442269053374]
  Product Attention (TPA) is a novel attention mechanism that uses tensor decompositions to represent queries, keys, and values compactly.<n>TPA achieves improved model quality alongside memory efficiency.<n>Based on TPA, we introduce the ProducT ATTion Transformer (T6), a new model architecture for sequence modeling.
  arXiv  Detail & Related papers  (2025-01-11T03:37:10Z)
• Markovian Transformers for Informative Language Modeling [1.172865818448696]
  Chain-of-Thought (CoT) reasoning often fails to faithfully reflect a language model's underlying decision process.<n>We introduce a Markovian language model framework that can be understood as a reasoning autoencoder.
  arXiv  Detail & Related papers  (2024-04-29T17:36:58Z)
• ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting [124.69672273754144]
  Chain-of-Thought (CoT) prompting can enhance the reasoning capabilities of large language models (LLMs) Existing CoT approaches usually focus on simpler reasoning tasks and thus result in low-quality and inconsistent CoT prompts. We introduce CoTGenius, a novel framework designed for the automatic generation of superior CoT prompts.
  arXiv  Detail & Related papers  (2024-03-21T11:34:26Z)
• Self-Polish: Enhance Reasoning in Large Language Models via Problem Refinement [50.62461749446111]
  Self-Polish (SP) is a novel method that facilitates the model's reasoning by guiding it to progressively refine the given problems to be more comprehensible and solvable. SP is to all other prompting methods of answer/reasoning side like CoT, allowing for seamless integration with state-of-the-art techniques for further improvement.
  arXiv  Detail & Related papers  (2023-05-23T19:58:30Z)
• Automatic Model Selection with Large Language Models for Reasoning [33.93807127935167]
  Chain-of-Thought (CoT) and Program-Aided Language Models (PAL) represent two distinct reasoning methods. We introduce a model selection method to combine the best of both worlds by employing a large language model. Our proposed method demonstrates significant performance improvements across eight reasoning datasets.
  arXiv  Detail & Related papers  (2023-05-23T17:57:59Z)
• Multimodal Chain-of-Thought Reasoning in Language Models [94.70184390935661]
  We propose Multimodal-CoT that incorporates language (text) and vision (images) modalities into a two-stage framework. Experimental results on ScienceQA and A-OKVQA benchmark datasets show the effectiveness of our proposed approach.
  arXiv  Detail & Related papers  (2023-02-02T07:51:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.