Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming

• URL: http://arxiv.org/abs/2606.12864v1
• Date: Thu, 11 Jun 2026 03:48:18 GMT
• Title: Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming
• Abstract summary: We introduce UOJ-Bench, a benchmark designed to evaluate Large Language Models (LLMs)<n>UOJ-Bench consists of three distinct tasks: code generation, code hacking, and code repair.<n>Our results show that even the strongest models fail to identify errors in more than 50% of a set of submissions.
• Score: 28.162504343040144
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Despite strong performance in competitive programming, the role of Large Language Models (LLMs) in supporting human learning in the same setting remains largely unexplored. In this work, we introduce UOJ-Bench, a benchmark designed to evaluate not only the problem-solving ability of LLMs, but also their ability to identify errors in human-written code -- a crucial educational activity traditionally supported by running test cases over online judge systems. UOJ-Bench consists of three distinct tasks: code generation, code hacking, and code repair, all constructed from real-world code submissions on the Universal Online Judge (UOJ) and evaluated through UOJ's native judging infrastructure. Our results show that under one-shot evaluation, even the strongest models fail to identify errors in more than 50% of a set of submissions that have been found to be incorrect by UOJ users. While test-time scaling improves success rates to above 90%, the substantial computational costs incurred from model inference limit its practicality for large-scale deployment. Despite these limitations, we find that the best-performing models under test-time scaling can uncover errors in over 5% of full-score submissions across roughly 30 problems, suggesting that frontier LLMs can already provide complementary signals beyond standard judging systems.

Related papers

• PROBE: Benchmarking Code Generation in Large Language Models [0.8921166277011344]
  Large Language Models (LLMs) are increasingly being used in everyday software engineering tasks.<n>LLMs are far from perfect, making evaluation essential to understand their strengths and limitations.<n>We introduce PROBE, a benchmark framework that establishes a systematic structure built on diverse and well-defined metrics.
  arXiv  Detail & Related papers  (2026-07-15T13:28:18Z)
• Beyond Blind Spots: Analytic Hints for Mitigating LLM-Based Evaluation Pitfalls [2.4484932263697234]
  Large Language Models are increasingly deployed as judges (LaaJ) in code generation pipelines.<n>LaaJs tend to overlook domain specific issues raising concerns about their reliability in critical evaluation tasks.<n>We develop a lightweight analytic checker tool that flags over 30 domain specific issues observed in practice.<n>We use its outputs as analytic hints, dynamically injecting them into the judges prompt to encourage LaaJ to revisit aspects it may have overlooked.
  arXiv  Detail & Related papers  (2025-12-18T07:43:48Z)
• Scaling Test-Time Compute to Achieve IOI Gold Medal with Open-Weight Models [72.52332895840279]
  GenCluster is a test-time compute framework that attains IOI gold-level performance using open-weight models.<n>We will show that GenCluster can achieve a gold medal at IOI 2025 for the first time with an open-weight model.
  arXiv  Detail & Related papers  (2025-10-16T02:19:25Z)
• COMPASS: A Multi-Dimensional Benchmark for Evaluating Code Generation in Large Language Models [0.0]
  We introduce a comprehensive evaluation framework that assesses code generation across three dimensions: correctness, efficiency, and quality.<n>Our evaluation of three leading reasoning-enhanced models, Anthropic Claude Opus 4, Google Gemini 2.5 Pro, and OpenAI O4-Mini-High, reveals that models achieving high correctness scores do not necessarily produce efficient algorithms or maintainable code.
  arXiv  Detail & Related papers  (2025-08-19T11:55:07Z)
• LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? [88.29001498765629]
  Large language models (LLMs) now outperform elite humans in competitive programming.<n>We revisit this claim, examining how LLMs differ from human experts and where limitations still remain.<n>We introduce LiveCodeBench Pro, a benchmark composed of problems from Codeforces, ICPC, and IOI.<n>A team of Olympiad medalists annotates every problem for algorithmic categories and conducts a line-by-line analysis of failed model-generated submissions.
  arXiv  Detail & Related papers  (2025-06-13T16:29:09Z)
• ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests [85.72404266850982]
  We propose textbfICPC-Eval, a top-level competitive coding benchmark designed to probing the frontiers of reasoning.<n>ICPC-Eval includes 118 carefully curated problems from 11 recent ICPC contests held in various regions of the world.<n>Results underscore the significant challenge in evaluating complex reasoning abilities.
  arXiv  Detail & Related papers  (2025-06-05T11:20:37Z)
• J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning [54.85131761693927]
  We introduce J1, a reinforcement learning framework for teaching LLM judges to think before making decisions.<n>Our core contribution lies in converting all judgment tasks for non-verifiable and verifiable prompts into a unified format with verifiable rewards.<n>We then use RL to train thinking-judges at scales of 8B, 32B, and 70B and show that they obtain state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-05-15T14:05:15Z)
• ProBench: Benchmarking Large Language Models in Competitive Programming [44.09445715541973]
  We propose ProBench to benchmark large language models (LLMs) in competitive programming.<n>ProBench collects a comprehensive set of competitive programming problems from Codeforces, Luogu, and Nowcoder platforms.<n>We assess 9 latest LLMs in competitive programming across multiple dimensions, including thought chain analysis, error type diagnosis, and reasoning depth evaluation.
  arXiv  Detail & Related papers  (2025-02-28T09:12:42Z)
• Learning to Solve and Verify: A Self-Play Framework for Code and Test Generation [69.62857948698436]
  Recent advances in large language models (LLMs) have improved their performance on coding benchmarks.<n>However, improvement is plateauing due to the exhaustion of readily available high-quality data.<n>We propose Sol-Ver, a self-play solver-verifier framework that jointly improves a single model's code and test generation capacity.
  arXiv  Detail & Related papers  (2025-02-20T18:32:19Z)
• CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings [70.95565672516979]
  Existing benchmarks, like LiveCodeBench and USACO, fall short due to the unavailability of private test cases, lack of support for special judges, and misaligned execution environments.<n>CodeElo is a standardized competition-level code generation benchmark that effectively addresses all these challenges for the first time.
  arXiv  Detail & Related papers  (2025-01-02T13:49:00Z)
• What's Wrong with Your Code Generated by Large Language Models? An Extensive Study [92.62952504133926]
  This study evaluated the performance of three leading closed-source LLMs and six popular open-source LLMs on three commonly used benchmarks.<n>We developed a taxonomy of bugs for incorrect codes and analyzed the root cause for common bug types.<n>We propose a novel training-free iterative method that introduces self-critique, enabling LLMs to critique and correct their generated code.
  arXiv  Detail & Related papers  (2024-07-08T17:27:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.