Janus: An Algorithm-Evaluator Co-Evolution Framework for LLM-Driven Discovery under Expensive Evaluation Budgets

• URL: http://arxiv.org/abs/2608.08189v1
• Date: Sat, 08 Aug 2026 15:31:12 GMT
• Title: Janus: An Algorithm-Evaluator Co-Evolution Framework for LLM-Driven Discovery under Expensive Evaluation Budgets
• Abstract summary: We introduce Janus, a framework that uses LLMs to co-evolve target programs and executable proxy evaluators.<n>To address label scarcity, Janus leverages domain knowledge encoded in LLMs to generate task-specific evaluator programs.<n>To mitigate distribution shift, Janus evolves evaluators alongside target programs, selects them using a promotion-aligned objective, and maintains region-conditioned portfolios.
• Score: 56.93233495754078
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM-driven program discovery relies on rapid evaluator feedback, but many scientific and engineering tasks require high-fidelity simulations, hardware execution, or physical experiments, making each evaluation expensive. Cheap surrogate evaluators can reduce this cost, yet fixed surrogates are vulnerable to search-induced distribution shift and are difficult to fit reliably from sparse, search-biased labels. We introduce Janus, a framework that uses LLMs to co-evolve target programs and executable proxy evaluators. To address label scarcity, Janus leverages domain knowledge encoded in LLMs to generate task-specific evaluator programs and calibrates them using real outcomes. To mitigate distribution shift, Janus evolves evaluators alongside target programs, selects them using a promotion-aligned objective, and maintains region-conditioned portfolios with online credit updates. Because proxy predictions remain fallible, Janus uses them only to prioritize candidates and requires real validation before candidates can enter the target-program population or update the incumbent. Across five scientific and engineering design tasks, Janus achieves a larger area under the best-so-far improvement curve over the real-evaluation budget and higher final performance than a matched baseline that evolves only target programs. On average, Janus reaches 99/% of the baseline's final improvement with 59.1/% fewer real evaluations. Evolved proxy evaluators also rank promising candidates more accurately than their seed versions. Together, these results extend evaluator-guided LLM discovery from tasks with cheap, scalable feedback to scientific domains where trustworthy evaluation is scarce and expensive.

Related papers

• HarnessOpt-Bench: Evaluating LLMs at Harness Optimization [3.846854817984248]
  We introduce a benchmark for end-to-end harness optimization under expensive evaluation regimes.<n>A trusted execution environment enforces the evaluation boundary, target-agent resource use, and preserves candidate versions for audit.<n>Experiment results show that models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks.
  arXiv  Detail & Related papers  (2026-08-06T17:21:05Z)
• Overcoming the Weakest-Link Effect in LLM-Driven Program Optimization via Heterogeneous Edit Recombination [59.679182823718556]
  HERO is a program that generates diverse, non-overlapping atomic edits and then systematically selects and composes them into coherent program improvements using evaluator scores.<n>We evaluate HERO across algorithmic problems, strategy games, the design of LLM-based agentic systems, and robotic path planning.
  arXiv  Detail & Related papers  (2026-07-31T02:05:45Z)
• LLM-as-a-Verifier: A General-Purpose Verification Framework [74.40111651545979]
  We introduce LLM-as-a-Verifier, a general-purpose verification framework.<n>It provides fine-grained feedback for agentic tasks without requiring additional training.<n>It achieves state-of-the-art performance on several benchmarks.
  arXiv  Detail & Related papers  (2026-07-06T17:59:35Z)
• Property-Guided LLM Program Synthesis for Planning [6.88204255655161]
  Instead of scoring programs after evaluation, we check whether a candidate satisfies a formally defined property.<n>This feedback drastically reduces both the number of program generations and the evaluation cost.
  arXiv  Detail & Related papers  (2026-05-15T16:23:35Z)
• Learning from the Irrecoverable: Error-Localized Policy Optimization for Tool-Integrated LLM Reasoning [36.23268783033404]
  In long-horizon tool-integrated reasoning trajectories, an early irrecoverable mistake can determine success or failure.<n>We propose Error-Localized Policy Optimization (ELPO) to localize the first irrecoverable step and leverage it for fine-grained credit assignment.<n>Our code will be publicly released soon.
  arXiv  Detail & Related papers  (2026-02-10T09:50:24Z)
• FreshBrew: A Benchmark for Evaluating AI Agents on Java Code Migration [2.981397088242044]
  We introduce FreshBrew, a novel benchmark for evaluating AI agents on project-level Java migrations.<n>We benchmark several state-of-the-art LLMs, and compare their performance against established rule-based tools.<n>Our evaluation of AI agents on this benchmark of 228 repositories shows that the top-performing model, 2.5 Gemini Flash, can successfully migrate 52.3 percent of projects to 17.
  arXiv  Detail & Related papers  (2025-10-06T14:39:58Z)
• An LLM-as-Judge Metric for Bridging the Gap with Human Evaluation in SE Tasks [15.820416019287622]
  SE-Jury is the first evaluation metric for LLM-as-Ensemble-Judge.<n>We evaluate SE-Jury across a diverse set of software engineering (SE) benchmarks.
  arXiv  Detail & Related papers  (2025-05-27T08:04:34Z)
• PredictaBoard: Benchmarking LLM Score Predictability [50.47497036981544]
  Large Language Models (LLMs) often fail unpredictably.<n>This poses a significant challenge to ensuring their safe deployment.<n>We present PredictaBoard, a novel collaborative benchmarking framework.
  arXiv  Detail & Related papers  (2025-02-20T10:52:38Z)
• AIME: AI System Optimization via Multiple LLM Evaluators [79.03422337674664]
  AIME is an evaluation protocol that utilizes multiple LLMs that each independently generate an evaluation on separate criteria and then combine them via concatenation. We show AIME outperforming baseline methods in code generation tasks, with up to $62%$ higher error detection rate and up to $16%$ higher success rate than a single LLM evaluation protocol on LeetCodeHard and HumanEval datasets.
  arXiv  Detail & Related papers  (2024-10-04T04:03:24Z)
• VinePPO: Refining Credit Assignment in RL Training of LLMs [66.80143024475635]
  We propose VinePPO, a straightforward approach that leverages the flexibility of language environments to compute unbiased Monte Carlo-based estimates.<n>Our method consistently outperforms PPO and other baselines across MATH and GSM8K datasets in less wall-clock time.
  arXiv  Detail & Related papers  (2024-10-02T15:49:30Z)
• Efficient Budget Allocation for Large-Scale LLM-Enabled Virtual Screening [0.9558392439655016]
  We consider an LLM-as-human-evaluator approach for conducting screening virtually, thereby reducing the cost burden.<n>We propose using a top-$m$ greedy evaluation mechanism, and design the explore-first top-$m$ greedy (EFG-$m$) algorithm.<n>Surprisingly, we uncover a bonus ranking effect, where the algorithm naturally induces an indifference-based ranking within the selected subset.
  arXiv  Detail & Related papers  (2024-08-18T16:44:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.