Reviewer Capability Governs Rejection Targeting, Not Repair Skill: Evidence from LLM Execute-Review-Revise Pipelines

• URL: http://arxiv.org/abs/2609.04270v1
• Date: Wed, 02 Sep 2026 19:21:49 GMT
• Title: Reviewer Capability Governs Rejection Targeting, Not Repair Skill: Evidence from LLM Execute-Review-Revise Pipelines
• Abstract summary: Multi-agent LLM pipelines increasingly assign roles, including execution and verification, to models of different capability tiers.<n>Previous literature has established that verification stages are not always beneficial, but holds reviewer capability roughly fixed relative to the executor.<n>We replace the reviewer with models spanning a capability range down to one that cannot solve the problems at all, and measure the outcome of every individual rejection.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-agent LLM pipelines increasingly assign roles, including execution and verification, to models of different capability tiers. This is done because running a flagship model at every stage is expensive. Previous literature has established that verification stages are not always beneficial, but holds reviewer capability roughly fixed relative to the executor. We vary it. We replace the reviewer with models spanning a capability range down to one that cannot solve the problems at all, and measure the outcome of every individual rejection. This is done across a constant set of 100 olympiad mathematics problems. A cross-family mid-tier reviewer improves final accuracy by 12 percentage points, from 52 to 64 percent (p = 0.0005), with zero damaged answers. Same-model self-review attains the highest error-detection rate of any condition (0.85 recall) yet yields no significant gain: it rejects 2.1 times as often for a third the repair rate (15 against 43 percent, p = 0.0074) and falsely rejects 35 percent of its own correct answers against 2 percent for the cross-family reviewer (paired p = 0.000015). The low damage rate of self-review proves to be an artifact of revision inertia rather than reviewer quality: of 18 falsely rejected correct answers, the three where the executor complied all became wrong, while the fifteen it ignored survived unchanged. Below a capability floor the role becomes inert: our weakest reviewer changed zero of 100 final answers while doubling token cost. These findings describe a single executor-reviewer configuration on 100 problems and should be read as a controlled pilot rather than a general claim about verification stages.

Related papers

• Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets [13.85834524293015]
  We measure the capability that role assumes and find it lacking under the protocol the role is usually deployed with.<n>Models judge whether a candidate belongs far better than they author the set itself.
  arXiv  Detail & Related papers  (2026-08-02T05:00:44Z)
• Where Does Agent Reliability Come From? A Cross-Benchmark Decomposition of Verification Loops, Specialist Models, and Scaffolding in a Production Enterprise Agent [0.0]
  We study one production system whose architecture installs verification loops.<n>We instrument the loop end-to-end, yielding an empirical verifier confusion matrix.
  arXiv  Detail & Related papers  (2026-07-19T03:20:33Z)
• Precise but Uncoupled: Reviewer Precision Does Not Guarantee Critique Uptake in Multi-Agent Math Reasoning [9.345403722854543]
  We test the assumption that a dedicated review stage should help turn wrong candidates into correct ones.<n>We ask whether this gap is explained by reviewer quality or by whether critique changes the next answer the protocol carries forward.
  arXiv  Detail & Related papers  (2026-07-16T18:38:10Z)
• Intelligence Is Not the Bottleneck: Validating an LLM First-Pass Manuscript Score Against Peer-Review Outcomes [0.0]
  Large language model (LLM) systems are increasingly proposed to assist peer review.<n>Most evaluations judge the prose of machine-generated review text, not the validity of the numeric score a system assigns.<n>We validate AIPR, which reads a submitted manuscript and emits five 0-100 quality dimensions and a weighted overall score.
  arXiv  Detail & Related papers  (2026-06-14T16:13:15Z)
• PACE: Anytime-Valid Acceptance Tests for Self-Evolving Agents [0.0]
  Self-evolving agents improve by repeatedly proposing changes to their own prompts, skills, ormodifications.<n>On Qwen2.5 agents (0.5B-3B) self-evolving at the prompt level on GSM8K, SVAMP, and ARC-Challenge, greedy acceptance commits 30-42% false and 10-33% harmful edits.<n> PACE commits the real one and essentially nothing else, matching greedy's held-out accuracy at sharply lower variance and about 18% lower evaluation cost.
  arXiv  Detail & Related papers  (2026-06-06T11:12:11Z)
• CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimization [50.59956036193097]
  A model produces a correct solution under reinforcement learning with verifiable rewards (RLVR)<n>Every token receives the same reward signal regardless of whether it was a decisive reasoning step or a grammatical filler.<n>We propose Contrastive Evidence Policy Optimization (CEPO)<n>CEPO asks a sharper question at every token: "does the correct answer favor this token?" but "does the correct answer favor it while the wrong answer disfavors it?"
  arXiv  Detail & Related papers  (2026-05-19T06:46:19Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam [63.84155758655084]
  Humanity's Last Exam (HLE) has become a widely used benchmark for evaluating frontier large language models.<n>We introduce HLE-Verified, a verified and revised version of HLE with a transparent verification protocol and fine-grained error taxonomy.<n>We evaluate seven state-of-the-art language models on HLE and HLE-Verified, observing an average absolute accuracy gain of 7--10 percentage points.
  arXiv  Detail & Related papers  (2026-02-15T02:50:15Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• RefineBench: Evaluating Refinement Capability of Language Models via Checklists [71.02281792867531]
  We evaluate two refinement modes: guided refinement and self-refinement.<n>In guided refinement, both proprietary LMs and large open-weight LMs can leverage targeted feedback to refine responses to near-perfect levels within five turns.<n>These findings suggest that frontier LMs require breakthroughs to self-refine their incorrect responses.
  arXiv  Detail & Related papers  (2025-11-27T07:20:52Z)
• Large Language Models are not Fair Evaluators [60.27164804083752]
  We find that the quality ranking of candidate responses can be easily hacked by altering their order of appearance in the context. This manipulation allows us to skew the evaluation result, making one model appear considerably superior to the other. We propose a framework with three simple yet effective strategies to mitigate this issue.
  arXiv  Detail & Related papers  (2023-05-29T07:41:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.