Social Pressure Breaks Majority Voting in LLM Safety Panels

• URL: http://arxiv.org/abs/2608.04415v1
• Date: Wed, 05 Aug 2026 03:51:12 GMT
• Title: Social Pressure Breaks Majority Voting in LLM Safety Panels
• Abstract summary: A common approach is to combine judgments from a panel of models to correct individual mistakes.<n>We study this risk in a controlled two-round experiment.<n>We find that the wrong-label peer message raises the average reviewer false-alarm rate.
• Score: 1.6734502292933282
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are increasingly used to detect unsafe content. A common approach is to combine judgments from a panel of models to correct individual mistakes, but this benefit may disappear when every model sees the same misleading context before voting. We study this risk in a controlled two-round experiment. Each model first judges an item alone, then judges it again after six simulated peers either assert the wrong label or abstain. We combine the final judgments by majority vote. Across six open-weight LLMs and six datasets, we find that the wrong-label peer message raises the average reviewer false-alarm rate from 56.5% under silent peers to 87.5%, and majority voting raises the panel false-alarm rate to 100%. Without an asserted label, the same panel outperforms its average member. The effect is strongly asymmetric: reviewers follow pushes toward "unsafe" far more than pushes toward "safe" (about 75% versus 17%), so the panel's false-alarm rate rises sharply while its harmful-miss rate changes little. The proprietary-model probe shows substantial variation across models. These results identify susceptibility to shared social cues as a failure mode of safety panels and provide a simple pre-deployment diagnostic.

Related papers

• Before You Poll with LLMs: A Deliberative Diagnostic Framework [0.2578242050187029]
  Current evaluations only test whether personas hold the right opinions -- a static snapshot.<n>We introduce the Deliberative Polling Diagnostic Framework, which compares human and LLM belief shifts after identical informational interventions.<n>Applying the framework to five frontier models using data from America in One Room, we find that every model fails.
  arXiv  Detail & Related papers  (2026-09-14T16:41:48Z)
• Conformity Breaks Conformal Prediction [1.3583317564926913]
  A conformal certificate can be valid when an LLM answers alone and invalid when the same LLM sees peers that unanimously assert a wrong answer.<n>We show that this shift silently breaks conformal prediction in multi-agent LLM systems.
  arXiv  Detail & Related papers  (2026-09-03T20:05:47Z)
• TTPO: Test-Time Policy Optimization [53.81524570216593]
  Test-Time Policy Optimization (TTPO) is an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL.<n>We show that TTPO matches label-supervised OPSD on five competition-level benchmarks, raises Qwen3-1.7B from 38.0% to 45.2% in TTT, yields +25.2% to +36.4% without thinking, and shows strong cross-task generalization.
  arXiv  Detail & Related papers  (2026-08-27T17:58:10Z)
• Forged Peer Judgments Mislead Multimodal LLM Judge Panels: Source-Blind Anchoring and Panel-Consensus Verification [9.710464466895521]
  Quoting independent visual judgments creates large anchoring gaps under both self and peer framing.<n>We introduce panel-consensus verification, which cross-checks a quote against independently collected blind votes.
  arXiv  Detail & Related papers  (2026-08-08T04:55:45Z)
• QuantiBias: Benchmarking Quantization-Induced Bias in LLMs [8.970269049715933]
  We find that almost every large language model that reaches a broad audience is quantized.<n>Our benchmark pairs a generative, multilingual stereotype probe with the refusal and multiple-choice controls that isolate open-ended generation.<n>A quantized build must be re-evaluated for open-ended bias, not only on the short-form safeguards it already passes.
  arXiv  Detail & Related papers  (2026-07-23T08:56:11Z)
• How Reliable Is Your Jailbreak Judge? Calibration and Adversarial Robustness of Automated ASR Scoring [7.789295015218435]
  Almost every paper on LLM jailbreaks and prompt injection reports an attack-success rate (ASR)<n>We compare the two judge families against human majority votes and then attack them.<n>We recommend that papers report judge precision and recall on a human-labeled slice, report ASR corrected for judge precision, and include an adversarial check of the judge.
  arXiv  Detail & Related papers  (2026-06-24T07:14:17Z)
• AgentFairBench: Do LLM Agents Discriminate When They Act? [2.3004655342211078]
  AgentFairBench is a cheap, reproducible, multi-domain benchmark for demographic disparity in the actions of LLM agents.<n>It spans three regulator-anchored domains: hiring, lending, and medical triage.<n>A NumPy-only harness computes counterfactual flip rate, mean absolute score difference (MASD), action-rate disparity, and tool-invocation disparity.
  arXiv  Detail & Related papers  (2026-06-15T13:50:26Z)
• Diversity Matters: Revisiting Test-Time Compute in Vision-Language Models [60.9932576020062]
  Test-time compute strategies have emerged as a lightweight approach to boost reasoning in large language models.<n>We propose Entropy-based TTC, which selects the most confident prediction based on predictive entropy.<n>We prove that ETTC outperforms majority voting under mild assumptions and empirically demonstrate that it consistently surpasses both voting and the best individual model.
  arXiv  Detail & Related papers  (2026-05-29T01:06:38Z)
• Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels [0.0]
  LLM-as-a-judge panels aggregate votes from multiple models.<n>We quantify how far their reliability falls short of the independent-voting ideal.
  arXiv  Detail & Related papers  (2026-05-28T11:48:17Z)
• CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution [52.691495954442985]
  CoVerRL is a framework where a single model alternates between generator and verifier roles, with each capability bootstrapping the other.<n> Experiments across Qwen and Llama model families demonstrate that CoVerRL outperforms label-free baselines by 4.7-5.9% on mathematical reasoning benchmarks.<n>Self-verification accuracy improves from around 55% to over 85%, confirming that both capabilities genuinely co-evolve.
  arXiv  Detail & Related papers  (2026-03-18T14:38:55Z)
• A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness [57.510025257780306]
  We show that existing validation protocols fail to account for substantial distribution shifts inherent to red-teaming.<n>We propose ReliableBench, a benchmark of behaviors that remain more consistently judgeable, and JudgeStressTest, a dataset designed to expose judge failures.
  arXiv  Detail & Related papers  (2026-02-04T15:13:35Z)
• Self-Harmony: Learning to Harmonize Self-Supervision and Self-Play in Test-Time Reinforcement Learning [48.14470449860784]
  Test-time reinforcement learning (TTRL) offers a label-free paradigm for adapting models using only synthetic signals at inference.<n>We introduce Self-Harmony, a framework built on a simple intuition.
  arXiv  Detail & Related papers  (2025-11-03T03:34:34Z)
• A Granular Study of Safety Pretraining under Model Abliteration [64.24346997570275]
  We study model abliteration, a lightweight projection technique designed to remove refusal-sensitive directions.<n>We issue 100 prompts with balanced harmful and harmless cases, classify responses as **Refusal** or **Non-Refusal** using multiple judges, and validate judge fidelity.<n>Our study produces a checkpoint-level characterization of which data-centric safety components remain robust under abliteration.
  arXiv  Detail & Related papers  (2025-10-03T07:01:45Z)
• CLUE: Non-parametric Verification from Experience via Hidden-State Clustering [64.50919789875233]
  We show that correctness of a solution is encoded as a geometrically separable signature within the trajectory of hidden activations.<n>ClUE consistently outperforms LLM-as-a-judge baselines and matches or exceeds modern confidence-based methods in reranking candidates.
  arXiv  Detail & Related papers  (2025-10-02T02:14:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.