Conformity Breaks Conformal Prediction

• URL: http://arxiv.org/abs/2609.04445v1
• Date: Thu, 03 Sep 2026 20:05:47 GMT
• Title: Conformity Breaks Conformal Prediction
• Abstract summary: A conformal certificate can be valid when an LLM answers alone and invalid when the same LLM sees peers that unanimously assert a wrong answer.<n>We show that this shift silently breaks conformal prediction in multi-agent LLM systems.
• Score: 1.3583317564926913
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A conformal certificate can be valid when an LLM answers alone and invalid when the same LLM sees peers that unanimously assert a wrong answer. The question is unchanged; the model's score for the correct answer changes. We call this a score-mechanism shift: clean calibration certifies how the model scores answers alone, but not how it scores them under peer pressure. We show that this shift silently breaks conformal prediction in multi-agent LLM systems. Across open-weight models and multiple-choice QA tasks, coverage falls from a calibrated 90% to 74% under unanimous-wrong peers at the standard alpha = 0.10 operating point. The average hides a sharper failure: by targeting the low-confidence items the certificate still covers, an attacker nearly halves coverage on that subgroup, from 87% to 47%, while the monitored average remains much higher. The failure also reaches the decision layer: a system that should escalate when uncertain can instead become confident enough to act on the attacker's wrong answer. Standard conformal fixes do not solve the problem, because the question distribution has not changed; the model's scoring behavior has.

Related papers

• LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails [0.0]
  Self-improving agent pipelines have a problem at their center.<n>An evaluation signal failed in eleven classes: judge bias, harness and metric failures, ground-truth errors, and reward hacking.<n>We describe PROCTOR, a Teacher-Student loop in which a stateful orchestrator holds all tool access, stateless subagents diagnose failures and draft mutations they cannot apply.
  arXiv  Detail & Related papers  (2026-09-02T07:54:23Z)
• Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation [52.58737865652009]
  Multiple-choice question answering (MCQA) benchmarks in NLP use number-right scoring (accuracy)<n>In educational testing, the scoring scheme, the combination of the response mode models follow and the rule for grading responses, is a key design choice that dictates which abilities to reward.<n>We examine how alternatives to number right change what MCQA measures with six education-inspired schemes that assess abilities beyond accuracy.
  arXiv  Detail & Related papers  (2026-08-30T16:35:22Z)
• LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning [6.241883798820154]
  We study cross-model consensus, the degree to which independently trained models, each solving the problem once, agree on a final answer.<n>We treat the panel as an LLM-jury, in which the structure of agreement, not any model's score of another, is the verification signal.<n>We make this precise with a parameter-free law, derived in closed form, that predicts consensus accuracy from three measured panel statistics to a mean absolute error of $0.03$.
  arXiv  Detail & Related papers  (2026-07-11T05:57:54Z)
• PACE: Anytime-Valid Acceptance Tests for Self-Evolving Agents [0.0]
  Self-evolving agents improve by repeatedly proposing changes to their own prompts, skills, ormodifications.<n>On Qwen2.5 agents (0.5B-3B) self-evolving at the prompt level on GSM8K, SVAMP, and ARC-Challenge, greedy acceptance commits 30-42% false and 10-33% harmful edits.<n> PACE commits the real one and essentially nothing else, matching greedy's held-out accuracy at sharply lower variance and about 18% lower evaluation cost.
  arXiv  Detail & Related papers  (2026-06-06T11:12:11Z)
• Empirical Bayes Conformal Prediction for Vision and Language Models [20.40115871314124]
  Conformal prediction (CP) gives distribution-free coverage for modern vision and language models.<n>Standard CP uses one realization, while average-then-calibrate variants smooth multiple realizations into a point estimate.<n>We describe an empirical Bayes conformal prediction framework that uses $r$-values to convert score variability into an uncertainty informed nonconformity score.
  arXiv  Detail & Related papers  (2026-05-22T03:17:14Z)
• The Illusion of Certainty: Decoupling Capability and Calibration in On-Policy Distillation [67.26315138466312]
  We propose a calibration-aware OPD framework, CaOPD, that estimates empirical confidence from model rollouts.<n>Our findings highlight that capability distillation does not imply confidence, and that confidence should be treated as an essential objective in post-training.
  arXiv  Detail & Related papers  (2026-04-18T04:43:40Z)
• From Debate to Decision: Conformal Social Choice for Safe Multi-Agent Deliberation [12.294365308421606]
  We introduce Conformal Social Choice, a post-hoc decision layer that converts debate outputs into calibrated act-versus-escalate decisions.<n>A hierarchical action policy maps singleton sets to autonomous action and larger sets to human escalation.<n>Because the layer refuses to act on cases where debate is confidently wrong, the remaining conformal singletons reach 90.0--96.8% accuracy.
  arXiv  Detail & Related papers  (2026-04-09T00:15:20Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• LEC: Linear Expectation Constraints for False-Discovery Control in Selective Prediction and Routing Systems [95.35293543918762]
  Large language models (LLMs) often generate unreliable answers, while uncertainty methods fail to fully distinguish correct from incorrect predictions.<n>We address this issue through the lens of false discovery rate (FDR) control, ensuring that among all accepted predictions, the proportion of errors does not exceed a target risk level.<n>We propose LEC, which reinterprets selective prediction as a constrained decision problem by enforcing a Linear Expectation Constraint.
  arXiv  Detail & Related papers  (2025-12-01T11:27:09Z)
• Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs [72.82403830490084]
  We argue that the decoding rule should be calibrated by correctness, not confidence alone.<n>We propose simple strategies that achieve this goal: Greedy-Threshold makes sampling greedy at very low confidence steps.<n>Together, our findings challenge prevailings about decoding under uncertainty and show gains across math and general reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-10-07T14:46:12Z)
• LACIE: Listener-Aware Finetuning for Confidence Calibration in Large Language Models [69.68379406317682]
  We introduce a listener-aware finetuning method (LACIE) to calibrate implicit and explicit confidence markers. We show that LACIE models the listener, considering not only whether an answer is right, but whether it will be accepted by a listener. We find that training with LACIE results in 47% fewer incorrect answers being accepted while maintaining the same level of acceptance for correct answers.
  arXiv  Detail & Related papers  (2024-05-31T17:16:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.