FrontierSmith: Synthesizing Open-Ended Coding Problems at Scale

• URL: http://arxiv.org/abs/2605.14445v1
• Date: Thu, 14 May 2026 06:39:42 GMT
• Title: FrontierSmith: Synthesizing Open-Ended Coding Problems at Scale
• Abstract summary: We introduce FrontierSmith, an automated system for evolving open-ended problems from existing closed-ended coding tasks.<n>On two open-ended coding benchmarks, training on our synthesized data yields substantial gains over the base models.<n>The synthesized problems also make agents take more turns and use more tokens, similar to human-curated ones.
• Score: 73.57353294235223
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many real-world coding challenges are open-ended and admit no known optimal solution. Yet, recent progress in LLM coding has focused on well-defined tasks such as feature implementation, bug fixing, and competitive programming. Open-ended coding remains a weak spot for LLMs, largely because open-ended training problems are scarce and expensive to construct. Our goal is to synthesize open-ended coding problems at scale to train stronger LLM coders. We introduce FrontierSmith, an automated system for iteratively evolving open-ended problems from existing closed-ended coding tasks. Starting from competitive programming problems, FrontierSmith generates candidate open-ended variants by changing the problems'goals, restricting outputs, and generalizing inputs. It then uses a quantitative idea divergence metric to select problems that elicit genuinely diverse approaches from different solvers. Agents then generate test cases and verifiers for the surviving candidates. On two open-ended coding benchmarks, training on our synthesized data yields substantial gains over the base models: Qwen3.5-9B improves by +8.82 score on FrontierCS and +306.36 (Elo-rating-based performance) on ALE-bench; Qwen3.5-27B improves by +12.12 and +309.12, respectively. The synthesized problems also make agents take more turns and use more tokens, similar to human-curated ones, suggesting that closed-ended seeds can be a practical starting point for long-horizon coding data.

Related papers

• Inferring Code Correctness from Specification [0.0]
  Large language models (LLMs) have become integral to modern software development, enabling automated code generation at scale.<n>We propose TRAILS (Targeted Reasoning Agreement via Inputs and Specifications), an approach that grounds LLM reasoning with concrete (input, output) pairs.<n>We evaluate TRAILS on two datasets, LiveCodeBench and CoCoClaNeL, across three LLMs (Qwen3Coder-30B, Devstral-Small-24B, and Olmo3.1-Instruct), comparing against HoarePrompt and a Zero-Shot Chain-of-Thought baseline.
  arXiv  Detail & Related papers  (2026-05-28T12:04:51Z)
• Increasing LLM Coding Capabilities through Diverse Synthetic Coding Tasks [41.75017840131367]
  Large language models (LLMs) have shown impressive promise in code generation.<n>We present a scalable synthetic data generation pipeline that produces nearly 800k instruction-reasoning-code-test quadruplets.
  arXiv  Detail & Related papers  (2025-10-27T10:54:25Z)
• QueST: Incentivizing LLMs to Generate Difficult Problems [77.75835742350644]
  Large Language Models have achieved strong performance on reasoning tasks, solving competition-level coding and math problems.<n>Existing competitive coding datasets contain only thousands to tens of thousands of problems.<n>We propose QueST, a novel framework which combines difficulty-aware graph sampling and difficulty-aware rejection fine-tuning.
  arXiv  Detail & Related papers  (2025-10-20T16:29:53Z)
• SciML Agents: Write the Solver, Not the Solution [69.5021018644143]
  We introduce two new datasets: a diagnostic dataset of adversarial "misleading" problems; and a large-scale benchmark of 1,000 diverse ODE tasks.<n>We evaluate open- and closed-source LLM models along two axes: (i) unguided versus guided prompting with domain-specific knowledge; and (ii) off-the-shelf versus fine-tuned variants.<n>Preliminary results indicate that careful prompting and fine-tuning can yield a specialized LLM agent capable of reliably solving simple ODE problems.
  arXiv  Detail & Related papers  (2025-09-12T02:53:57Z)
• Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedback [52.763620660061115]
  ONI is a distributed architecture that simultaneously learns an RL policy and an intrinsic reward function.<n>We explore a range of algorithmic choices for reward modeling with varying complexity.<n>Our approach achieves state-of-the-art performance across a range of challenging tasks from the NetHack Learning Environment.
  arXiv  Detail & Related papers  (2024-10-30T13:52:43Z)
• An Empirical Study on Self-correcting Large Language Models for Data Science Code Generation [1.335664823620186]
  Large Language Models (LLMs) have recently advanced many applications on software engineering tasks. CoT-SelfEvolve iteratively and automatically refines code through a self-correcting process.
  arXiv  Detail & Related papers  (2024-08-28T09:19:09Z)
• What's Wrong with Your Code Generated by Large Language Models? An Extensive Study [92.62952504133926]
  This study evaluated the performance of three leading closed-source LLMs and six popular open-source LLMs on three commonly used benchmarks.<n>We developed a taxonomy of bugs for incorrect codes and analyzed the root cause for common bug types.<n>We propose a novel training-free iterative method that introduces self-critique, enabling LLMs to critique and correct their generated code.
  arXiv  Detail & Related papers  (2024-07-08T17:27:17Z)
• Code Repair with LLMs gives an Exploration-Exploitation Tradeoff [16.80314690163063]
  Iteratively improving and repairing source code with large language models (LLMs) has emerged as a popular way of generating programs that would be too complex to construct in one shot. We show here that refinement exposes an explore-exploit tradeoff: exploit by refining the program that passes the most test cases, or explore by refining a lesser considered program.
  arXiv  Detail & Related papers  (2024-05-26T04:00:30Z)
• AutoCodeRover: Autonomous Program Improvement [8.66280420062806]
  We propose an automated approach for solving GitHub issues to autonomously achieve program improvement. In our approach called AutoCodeRover, LLMs are combined with sophisticated code search capabilities, ultimately leading to a program modification or patch. Experiments on SWE-bench-lite (300 real-life GitHub issues) show increased efficacy in solving GitHub issues (19% on SWE-bench-lite), which is higher than the efficacy of the recently reported SWE-agent.
  arXiv  Detail & Related papers  (2024-04-08T11:55:09Z)
• InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Models [56.723509505549536]
  InfiBench is the first large-scale freeform question-answering (QA) benchmark for code to our knowledge. It comprises 234 carefully selected high-quality Stack Overflow questions that span across 15 programming languages. We conduct a systematic evaluation for over 100 latest code LLMs on InfiBench, leading to a series of novel and insightful findings.
  arXiv  Detail & Related papers  (2024-03-11T02:06:30Z)
• Measuring Coding Challenge Competence With APPS [54.22600767666257]
  We introduce APPS, a benchmark for code generation. Our benchmark includes 10,000 problems, which range from having simple one-line solutions to being substantial algorithmic challenges. Recent models such as GPT-Neo can pass approximately 15% of the test cases of introductory problems.
  arXiv  Detail & Related papers  (2021-05-20T17:58:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.