Sharding Prevents LLM Oversight Failures and Adversarial Exploitation

• URL: http://arxiv.org/abs/2608.06422v1
• Date: Wed, 05 Aug 2026 18:21:21 GMT
• Title: Sharding Prevents LLM Oversight Failures and Adversarial Exploitation
• Abstract summary: Sharding partitions requirements into smaller groups, assigns each group to a separate call, and aggregates the verdicts.<n>We find that a sharded weaker judge can outperform a more capable holistic judge and match that judge even when the latter receives the panel's full budget.
• Score: 45.380649793616705
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Giving an LLM judge more compute does not necessarily make it check more requirements. When one call must return many verdicts, some decisions become weakly grounded in the evidence, even when that call receives the same token or tool budget as a panel of separate calls. Across expert-graded research replications, legal work, and clinical-trial assessments, agreement with experts falls as the number of verdicts per call grows. We identify sharding as the intervention that mitigates this failure in model-based oversight. Sharding partitions the requirements into smaller groups, assigns each group to a separate call, and aggregates the verdicts. Against a single call with the panel's full budget, sharding improves agreement while holding the model, evidence, total budget, and per-decision budget fixed. Overall, we find that a sharded weaker judge can outperform a more capable holistic judge and match that judge even when the latter receives the panel's full budget. Additionally, we find that sharding exhibits robustness against adversaries. A best-of-N adversary can hold the underlying work fixed, vary only its presentation, and increase an overloaded judge's acceptance of genuinely unmet criteria severalfold. Wherever sharding reduces baseline error, it removes this adversarial advantage, keeping over-acceptance low even as the adversary's search widens. Sharding does not address attacks that persuade the judge separately on each criterion rather than exploiting overload. In that setting, we find that debate-style opposition on top of sharding withstands such adaptive re-optimization.

Related papers

• CABAL: Multi-Agent Simulacra for Tracing the Effects of Collusive Bidding in Peer Review [20.449658717520876]
  alg is an end-to-end multi-agent simulacra framework for studying reviewer assignment integrity.<n>We develop an affinity-guided collusive bidding strategy that uses mutual reviewer-paper affinities to construct collusion rings.
  arXiv  Detail & Related papers  (2026-09-04T14:55:33Z)
• JudgePanel: A Compact Judge with Panel Deliberation via Adaptive Multi-Reward Reinforcement Learning [5.81106044690391]
  Single-model judges are limited by their inherent model biases.<n>Multi-agent evaluation protocols that mitigate this through diverse deliberation are prohibitively expensive at inference time.<n>We propose textbfmodelname, which equips a compact underlineJudge model with multi-agent underlinePanel deliberation capability.
  arXiv  Detail & Related papers  (2026-08-29T09:33:49Z)
• Stopping and Routing LLM Judge Panels [20.569427412312297]
  The deployment question is not only which judge is best, but which examples, and when panel construction should stop.<n>We formulate judge-panel design as a role-conditioned allocation problem.<n>The method is compared with single judges, flat panels, matched diversitys, full-call stacking, reliability juries, and frugal cascades.
  arXiv  Detail & Related papers  (2026-08-20T08:58:00Z)
• Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling [35.945096782147864]
  MLLM judges tend to reward plausible narratives over perceptually correct answers when visual evidence conflicts with textual cues.<n>We introduce the Perceptually Perturbed Judgment dataset, which constructs minimally edited counterfactual responses.<n>We develop a unified training framework that combines a structured GRPO-based reward with a batch-ranking objective, achieving coherent global ordering without explicit pairwise labels.
  arXiv  Detail & Related papers  (2026-06-01T17:59:46Z)
• PReMISE: Policy Rubrics as Measurement Specifications for LLM Judges [33.739528721872844]
  A vague rubric asking for a response to be helpful and factual'' can reward polished answers that invent facts or violate user intent.<n>We introduce PReMISE, a framework that audits any rubric set under LLM-judge use along four axes: structural adequacy, reliability, preference fit, and adversarial robustness.<n>We contribute two audit-targeted repair operations: preference-rank selection raises judge accuracy on paired responses from $65.0%$ to $68.6%$, competitive with the strongest rubric-discovery baselines, and leading on two of three judges in our cross-judge sweep.
  arXiv  Detail & Related papers  (2026-05-29T03:45:11Z)
• Instance-Optimal Estimation with Multiple LLM Judges on a Budget [84.31744861038106]
  We formalize this question as *budgeted heteroskedastic multi-judge estimation*.<n>Given $K$ prompt-response pairs, $J$ judges with known costs, and unknown query-judge variances, the goal is to estimate a bounded score vector while minimizing an $ell_p$-error.<n>We prove that EST-IVWE matches the oracle IVWE rate up to lower-order terms in the budget.
  arXiv  Detail & Related papers  (2026-05-22T08:26:08Z)
• Permutation-Consensus Listwise Judging for Robust Factuality Evaluation [0.8563354084119061]
  We study candidate-order sensitivity in listwise factuality evaluation.<n>We introduce PCFJudge, an inference-time method that reruns the same factuality-first listwise prompt over multiple orderings of the same candidate set.<n>On RewardBench 2 Factuality, PCFJudge improves over direct judging by up to 7 absolute points.
  arXiv  Detail & Related papers  (2026-03-20T23:35:14Z)
• A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness [57.510025257780306]
  We show that existing validation protocols fail to account for substantial distribution shifts inherent to red-teaming.<n>We propose ReliableBench, a benchmark of behaviors that remain more consistently judgeable, and JudgeStressTest, a dataset designed to expose judge failures.
  arXiv  Detail & Related papers  (2026-02-04T15:13:35Z)
• From Sycophancy to Sensemaking: Premise Governance for Human-AI Decision Making [0.0]
  Low-friction assistants can become sycophantic, baking in implicit assumptions and pushing verification costs onto experts.<n>We argue reliable human-AI partnership requires a shift from answer generation to collaborative premise governance.
  arXiv  Detail & Related papers  (2026-02-02T17:42:54Z)
• Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation [76.5533899503582]
  Large language models (LLMs) are increasingly used as judges to evaluate agent performance.<n>We show this paradigm implicitly assumes that the agent's chain-of-thought (CoT) reasoning faithfully reflects both its internal reasoning and the underlying environment state.<n>We demonstrate that manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories spanning diverse web tasks.
  arXiv  Detail & Related papers  (2026-01-21T06:07:43Z)
• CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewards [72.44810390478229]
  CompassJudger-2 is a novel generalist judge model that overcomes limitations via a task-driven, multi-domain data curation strategy.<n> CompassJudger-2 achieves superior results across multiple judge and reward benchmarks.
  arXiv  Detail & Related papers  (2025-07-12T01:34:24Z)
• Confidence-Budget Matching for Sequential Budgeted Learning [69.77435313099366]
  We formalize decision-making problems with querying budget. We consider multi-armed bandits, linear bandits, and reinforcement learning problems. We show that CBM based algorithms perform well in the presence of adversity.
  arXiv  Detail & Related papers  (2021-02-05T19:56:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.