Who Flips? Self- and Cross-Model Counterarguments Reveal Answer Instability in LLMs

• URL: http://arxiv.org/abs/2606.16011v1
• Date: Sun, 14 Jun 2026 20:45:30 GMT
• Title: Who Flips? Self- and Cross-Model Counterarguments Reveal Answer Instability in LLMs
• Abstract summary: We introduce a controlled protocol for evaluating answer stability.<n>After a model answers a multiple-choice question correctly, we challenge the model with a coherent argument for an incorrect option.<n>We find that self-attribution consistently increases flip rates.
• Score: 9.465306048183798
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Standard accuracy benchmarks are designed to test how closely large language models (LLMs) approach correct answers, but are not suitable for testing whether LLMs stick with a correct answer when that answer is challenged by a plausible counter-argument. We introduce a controlled protocol for evaluating answer stability: after a model answers a multiple-choice question correctly, we challenge the model's answer with a coherent argument for an incorrect option and measure whether the model flips. The setup a) isolates argumentative content from overt social pressure and b) varies argument length, self-attribution, and cross-model source. Across seven frontier models and 57 MMLU subjects, flip rates range from 17.5% to 97.3%, revealing large differences in stability that are not captured by accuracy metrics alone. We find that self-attribution consistently increases flip rates (mean +7.1pp, up to +18.7pp). Also, pooling wrong-answer arguments across models and selecting the most effective one per question yields stronger adversarial challenges than relying on any single source model. We further construct MaxFlip, a curated challenge set that amplifies flips by up to +23.6pp over standard self-generated challenges. We release the protocol, challenge records, and MaxFlip to support stability evaluation alongside standard accuracy benchmarks. Materials are available at https://github.com/nafisenik/WhoFlips and https://hf.co/datasets/nafisehNik/WhoFlips.

Related papers

• When Models Defer to Wrong Answers: A Robustness Audit of Source-Attributed Cues in Multiple-Choice QA [2.347309464574674]
  Language models often receive a question together with a claim about what another source answered.<n>We audit whether such claims destabilize answers in multiple-choice question answering.<n>We evaluate four instruction-following models on MMLU-Pro and IndicMMLU-Pro in English, Hindi, Bengali, Tamil, and Telugu.
  arXiv  Detail & Related papers  (2026-09-08T15:57:24Z)
• A decodability criterion predicts when hidden-state selection beats majority voting in large language models [10.860308169044679]
  We propose a dynamic selection combiner that trains a linear gate on the answer-token hidden state and selects the highest-scoring candidate.<n>Its main contribution is decodability, a leakage-free measure of how well the gate ranks a question's correct candidates.
  arXiv  Detail & Related papers  (2026-08-17T21:02:10Z)
• Second Guess: Detecting Uncertainty Through Abstention and Answer Stability in Small Language Models [2.5782420501870296]
  Large language models often generate confident but incorrect answers rather than abstaining when uncertain.<n>We propose _Second Guess_, a lightweight, parameter-free prompting technique for abstention in multiple-choice question answering (MCQA)<n>Second Guess achieves the highest composite risk improvement of 10.81%.
  arXiv  Detail & Related papers  (2026-05-25T03:38:54Z)
• Measuring short-form factuality in large language models [50.15055025275888]
  We present SimpleQA, a benchmark that evaluates the ability of language models to answer short, fact-seeking questions. SimpleQA is adversarially collected against GPT-4 responses. Each answer in SimpleQA is graded as either correct, incorrect, or not attempted.
  arXiv  Detail & Related papers  (2024-11-07T01:58:42Z)
• DCR: Divide-and-Conquer Reasoning for Multi-choice Question Answering with LLMs [9.561022942046279]
  We propose Divide and Conquer Reasoning (DCR) to enhance the reasoning capability of large language models (LLMs) We first categorize questions into two subsets based on confidence score ($mathcalCS$), which is estimated by statistical frequency of generated answers. In particular, we first categorize questions into two subsets based on confidence score ($mathcalCS$), which is estimated by statistical frequency of generated answers.
  arXiv  Detail & Related papers  (2024-01-10T14:38:46Z)
• Do Large Language Models have Shared Weaknesses in Medical Question Answering? [1.25828876338076]
  Large language models (LLMs) have made rapid improvement on medical benchmarks, but their unreliability remains a persistent challenge for safe real-world uses. We benchmark a range of top LLMs and identify consistent patterns across models. We found evidence of similarities between models in which questions they answer correctly, as well as similarities with human test takers.
  arXiv  Detail & Related papers  (2023-10-11T06:26:19Z)
• Realistic Conversational Question Answering with Answer Selection based on Calibrated Confidence and Uncertainty Measurement [54.55643652781891]
  Conversational Question Answering (ConvQA) models aim at answering a question with its relevant paragraph and previous question-answer pairs that occurred during conversation multiple times. We propose to filter out inaccurate answers in the conversation history based on their estimated confidences and uncertainties from the ConvQA model. We validate our models, Answer Selection-based realistic Conversation Question Answering, on two standard ConvQA datasets.
  arXiv  Detail & Related papers  (2023-02-10T09:42:07Z)
• Enhancing Self-Consistency and Performance of Pre-Trained Language Models through Natural Language Inference [72.61732440246954]
  Large pre-trained language models often lack logical consistency across test inputs. We propose a framework, ConCoRD, for boosting the consistency and accuracy of pre-trained NLP models. We show that ConCoRD consistently boosts accuracy and consistency of off-the-shelf closed-book QA and VQA models.
  arXiv  Detail & Related papers  (2022-11-21T21:58:30Z)
• Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly [100.60560477391732]
  We promote a problem formulation for reliable visual question answering (VQA) We analyze both their coverage, the portion of questions answered, and risk, the error on that portion. We find that although the best performing models achieve over 71% accuracy on the VQA v2 dataset, introducing the option to abstain limits them to answering less than 8% of the questions to achieve a low risk of error (i.e., 1%) This motivates us to utilize a multimodal selection function to directly estimate the correctness of the predicted answers, which we show can triple the coverage from, for example, 5.0% to 16.7% at
  arXiv  Detail & Related papers  (2022-04-28T16:51:27Z)
• How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering [80.82194311274694]
  We examine the question "how can we know when language models know, with confidence, the answer to a particular query?" We examine three strong generative models -- T5, BART, and GPT-2 -- and study whether their probabilities on QA tasks are well calibrated. We then examine methods to calibrate such models to make their confidence scores correlate better with the likelihood of correctness.
  arXiv  Detail & Related papers  (2020-12-02T03:53:13Z)
• Selective Question Answering under Domain Shift [90.021577320085]
  Abstention policies based solely on the model's softmax probabilities fare poorly, since models are overconfident on out-of-domain inputs. We train a calibrator to identify inputs on which the QA model errs, and abstain when it predicts an error is likely. Our method answers 56% of questions while maintaining 80% accuracy; in contrast, directly using the model's probabilities only answers 48% at 80% accuracy.
  arXiv  Detail & Related papers  (2020-06-16T19:13:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.