Where Do Large Language Models Fail on Competitive Programming? A Taxonomy of Failures by Algorithm Type and Difficulty Rating

• URL: http://arxiv.org/abs/2606.05228v1
• Date: Tue, 02 Jun 2026 18:48:38 GMT
• Title: Where Do Large Language Models Fail on Competitive Programming? A Taxonomy of Failures by Algorithm Type and Difficulty Rating
• Abstract summary: Large language models (LLMs) demonstrate increasing proficiency on competitive programming benchmarks.<n>We present a systematic empirical study of LLM failure patterns using a balanced taxonomy of 315 Codeforces problems.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) demonstrate increasing proficiency on competitive programming benchmarks, yet technical reports predominantly publish aggregate pass rates, obscuring domain-specific vulnerabilities. We present a systematic empirical study of LLM failure patterns using a balanced taxonomy of 315 Codeforces problems across seven algorithm categories and three difficulty tiers. We evaluate GPT-4o and Claude Sonnet 4.6 under strict execution-based conditions, controlling for temperature (T = 0.2). To isolate the impact of reasoning frameworks on algorithmic correctness, we conduct an ablation study comparing direct zero-shot generation against zero-shot Chain-of-Thought (CoT). Our findings reveal a severe divergence from standard NLP benchmarks: forcing CoT aggressively penalizes GPT-4o, dropping its pass rate from 46.0% to 36.8% and exacerbating a critical weakness in Greedy logic. Conversely, while Claude maintains a higher logical baseline (63.5% under CoT), the expanded text generation severely degrades its markdown instruction adherence, causing its Compile Errors to more than triple (from 9 to 31, a 244% increase). Furthermore, failure-mode analysis indicates that Wrong Answer (WA) is the dominant verdict for both models--accounting for over 90% of GPT-4o's and roughly 70% of Claude's unaccepted solutions. These findings empirically demonstrate that standard prompt engineering techniques fail to bridge the algorithmic reasoning gap in competitive programming environments.

Related papers

• State of Thought Enables Endogenous Reasoning [48.902117018115256]
  Test-time reasoning has emerged as a major approach to improving the capabilities of Large Language Models (LLMs)<n>We propose State of Thought, a new reasoning paradigm that enables endogenous reasoning in LLMs.<n>We show that SoT consistently improves mean-baseline accuracy while reducing generated tokens by 62.6% and end-to-end latency by 44.6%.
  arXiv  Detail & Related papers  (2026-09-13T05:55:03Z)
• Diagnosing Fine-Grained Inconsistency Classification in Financial Disclosure Text [3.2712318205295907]
  Financial disclosures contain numerical claims, temporal statements, entity references, policy commitments, and risk descriptions that may conflict in qualitatively different ways.<n>We study this problem as fine-grained inconsistency.<n>A fine-tuned 300M encoder reaches 61.9% accuracy, compared with 61.5% for a LoRA-adapted Qwen3.5-9B model and 61.3% for GPT-adapted generative models.
  arXiv  Detail & Related papers  (2026-07-29T01:03:53Z)
• Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search [50.16356451328644]
  Proving Shannon-type entropy inequalities is a fundamental task in information theory.<n>We investigate whether small-scale large language models, fine-tuned on atomic proof steps, can automate this process.<n>We show that GPT-5.5 solves 1.7% samples under zero-shot prompting while Psitip solves 33.3% samples.
  arXiv  Detail & Related papers  (2026-06-04T05:43:12Z)
• Structured Synthetic Reasoning Data for Arithmetic Fine-Tuning of Small Language Models [0.0]
  We study whether structured synthetic reasoning data can improve arithmetic reasoning in small language models.<n>We generate a 21,250-example corpus of grade-school arithmetic word-problem variants using GPT-5-mini.<n>Fine-tuned models produce shorter reasoning traces, make fewer arithmetic and distractor-use errors, and benefit more consistently from self-consistency sampling.
  arXiv  Detail & Related papers  (2026-05-21T15:50:29Z)
• A-ProS: Towards Reliable Autonomous Programming Through Multi-Model Feedback [4.062253229926774]
  A-ProS is an autonomous AI agent that solves competitive programming problems through a hybrid multi-model feedback framework.<n>A-ProS achieves over 2x greater gains compared to baseline agent loops.
  arXiv  Detail & Related papers  (2026-05-18T08:55:30Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• When the Ruler is Broken: Parsing-Induced Suppression in LLM-Based Security Log Evaluation [0.0]
  We demonstrate a class of silent, systematic evaluation errors that can cause a fully functional model to appear completely non-functional categories.<n>We propose SOC-Bench v0, a benchmark framework comprising a standardized 13 category threat taxonomy, minimum statistical power requirements, fuzzy field extraction specification, and a public scoring script intended to prevent specific accuracy distortion in future SOC research.
  arXiv  Detail & Related papers  (2026-05-08T06:03:11Z)
• On the Role of Reasoning Patterns in the Generalization Discrepancy of Long Chain-of-Thought Supervised Fine-Tuning [63.41902113656453]
  Supervised Fine-Tuning (SFT) on long Chain-of-Thought (CoT) trajectories has become a pivotal phase in building large reasoning models.<n>We conduct a comparative study using two sources of verified CoT trajectories generated by two competing models.<n>We find that SFT on textttDeepSeek-R1-0528 data achieves remarkably lower training loss, yet exhibits significantly worse generalization performance.
  arXiv  Detail & Related papers  (2026-04-02T07:00:54Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness Methods for LLMs [55.20230501807337]
  We present the first systematic evaluation of 5 methods for improving prompt robustness within a unified experimental framework.<n>We benchmark these techniques on 8 models from Llama, Qwen and Gemma families across 52 tasks from Natural Instructions dataset.
  arXiv  Detail & Related papers  (2025-08-15T10:32:50Z)
• Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback [59.078756231841574]
  Critique-GRPO is an online RL framework that integrates both natural language and numerical feedback for effective policy optimization.<n>We show Critique-GRPO consistently outperforms supervised learning and RL-based fine-tuning methods across eight challenging mathematical, STEM, and general reasoning tasks.
  arXiv  Detail & Related papers  (2025-06-03T17:39:02Z)
• Code Generation with Small Language Models: A Codeforces-Based Study [1.728619497446087]
  Large Language Models (LLMs) demonstrate capabilities in code generation, potentially boosting developer productivity.<n>However, their adoption remains limited by high computational costs, among other factors.<n>Small Language Models (SLMs) present a lightweight alternative.
  arXiv  Detail & Related papers  (2025-04-09T23:57:44Z)
• Benchmarking Reasoning Robustness in Large Language Models [76.79744000300363]
  We find significant performance degradation on novel or incomplete data.<n>These findings highlight the reliance on recall over rigorous logical inference.<n>This paper introduces a novel benchmark, termed as Math-RoB, that exploits hallucinations triggered by missing information to expose reasoning gaps.
  arXiv  Detail & Related papers  (2025-03-06T15:36:06Z)
• Preference Optimization for Reasoning with Pseudo Feedback [100.62603571434167]
  We introduce a novel approach to generate pseudo feedback for reasoning tasks by framing the labeling of solutions as an evaluation against associated test cases.<n>We conduct experiments on both mathematical reasoning and coding tasks using pseudo feedback for preference optimization, and observe improvements across both tasks.
  arXiv  Detail & Related papers  (2024-11-25T12:44:02Z)
• Markovian Transformers for Informative Language Modeling [1.172865818448696]
  Chain-of-Thought (CoT) reasoning often fails to faithfully reflect a language model's underlying decision process.<n>We introduce a Markovian language model framework that can be understood as a reasoning autoencoder.
  arXiv  Detail & Related papers  (2024-04-29T17:36:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.