Ockhamareto: Pareto-Gated Segment-Level Credit Assignment for Concise Unit-Test Generation with Reinforcement Learning
Abstract Overview
Ockhamareto is a single-shot GRPO framework for generating complete unit-test suites while jointly optimizing fault-detection effectiveness and suite conciseness. The method combines a Pareto-gated bonus that rewards only non-dominated rollouts in the quality-versus-test-count space with token-level segment credit that assigns each test’s marginal mutation kills back to the tokens that produced it. The paper evaluates the approach on five held-out Python benchmarks, including ULT, HumanEval+, MBPP+, CodeContests, and TestGenEval-Lite, using mutation score as the primary metric and coverage, correctness, and suite size as supporting measures. The authors also analyze empirical Pareto fronts per function to study how many tests are worth maintaining under diminishing returns.
Novelty
The distinctive contribution is the combination of suite-level Pareto-gated reward shaping with token-level segment credit for single-shot test-suite generation. This gives a one-call RL setup both an explicit conciseness objective and fine-grained intra-suite credit assignment, rather than relying on multi-turn generation for per-test feedback.
Results
On ULT at N=5, Ockhamareto achieves 49.9% mutation with 2.60 tests on average, compared with 31.3% and 4.67 tests for the strongest RL baseline, MIST-RL, thereby improving both effectiveness and conciseness. Across HumanEval+, MBPP+, CodeContests, and TestGenEval-Lite, it reports the highest mutation and coverage metrics among the compared methods while also using the smallest suites. The paper further shows gains of roughly +30 to +35 mutation points over untuned base models at 4B, 9B, and 27B scales, and finds that the median empirical knee point in per-function Pareto fronts is three tests.
Key Points
- Ockhamareto uses Pareto gating to reward only test suites that are non-dominated in effectiveness-versus-size trade-offs, avoiding a fixed scalar penalty for extra tests.
- Its segment-credit mechanism maps each test’s first-kill mutation contribution to the token span of that test, providing fine-grained learning signals within single-shot generation.
- Empirically, the method produces shorter suites that front-load bug-catching value, and its per-function Pareto-front analysis suggests that appropriate suite size cannot be reliably inferred from simple static code metrics.