When Evidence Conflicts: Uncertainty and Order Effects in Retrieval-Augmented Biomedical Question Answering

• URL: http://arxiv.org/abs/2605.14115v1
• Date: Wed, 13 May 2026 21:02:24 GMT
• Title: When Evidence Conflicts: Uncertainty and Order Effects in Retrieval-Augmented Biomedical Question Answering
• Abstract summary: Using HealthContradict, we evaluate six open-weight large language models (LLMs) under five controlled evidence conditions.<n>Results show that conflicting biomedical evidence is both an uncertainty and robustness problem.
• Score: 2.6393907656700706
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Biomedical retrieval-augmented large language models (LLMs) often face evidence that is incomplete, misleading, or internally contradictory, yet evaluation usually emphasizes answer accuracy under helpful context rather than reliability under conflict. Using HealthContradict, we evaluate six open-weight LLMs under five controlled evidence conditions: no retrieved context, correct-only context, incorrect-only context, and two mixed conditions containing both correct and contradictory documents in opposite orders. In this conflicting-evidence order contrast, where the same two documents are both present and only their order is reversed, accuracy drops for every model and 11.4%--25.2% of predictions flip. To support abstention in these difficult cases, we also evaluate a conflict-aware abstention score that combines model confidence with a detector of evidence conflict. In the two hardest conditions, this score improves selective accuracy over confidence-only, with mean gains of 7.2--33.4 points in incorrect-only (`IC') and 3.6--14.4 points in incorrect-first conflicting (`ICC') conditions across 75%, 50%, and 25% coverage. These results show that conflicting biomedical evidence is both an uncertainty and robustness problem and motivate evaluation and abstention methods that explicitly account for evidence disagreement.

Related papers

• NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment [54.4265627247104]
  Large language models (LLMs) are increasingly used in peer review at major AI conferences.<n>Existing benchmarks assess novelty as a single holistic score.<n>We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis.
  arXiv  Detail & Related papers  (2026-09-10T08:34:56Z)
• ConflictScore: Identifying and Measuring How Language Models Handle Conflicting Evidence [48.938290419832335]
  We introduce ConflictScore, a novel metric that quantifies how well a model's response acknowledges conflicting evidence in its grounding documents.<n>Our framework decomposes responses into atomic claims, labels each claim against each grounding document, and then aggregates these labels into two complementary measures: ConflictScore-Count (CS-C), the proportion of claims exhibiting conflicts, and ConflictScore-Ratio (CS-R), the balance between supporting and contradicting evidence.
  arXiv  Detail & Related papers  (2026-06-24T23:00:09Z)
• Confident but Conflicted: Internal Uncertainty and Cognitive Dissonance Resolution in LLMs [6.968805961712596]
  Large language models (LLMs) frequently encounter inputs that disagree with their prior outputs.<n>Dissonance can be resolved through persuasion, backfire, or immunity.<n>Trust Elasticity (TE) is an econometrics-inspired measure of how readily a model is persuaded toward conflicting evidence.
  arXiv  Detail & Related papers  (2026-06-21T18:39:24Z)
• Signed Evidence Flow: Conflict-Aware and Stability-Calibrated Data Analysis [0.0]
  We propose Signed Evidence Flow (SEF), which combines a fitted prediction rule with signed feature attributions to measure support, opposition, conflict, and stability.<n>We prove that confidence determines conflict exactly when it also determines total evidence mass, derive the remaining conditional variance, and state when conflict can improve loss prediction beyond confidence.
  arXiv  Detail & Related papers  (2026-06-20T04:29:07Z)
• Does RAG Know When Retrieval Is Wrong? Diagnosing Context Compliance under Knowledge Conflict [7.091145002779077]
  Context-Driven Decomposition (CDD) is a belief-decomposition probe that operates at inference time and serves as an intervention mechanism for controlled retrieval conflict.<n>These three patterns identify context-compliance as a structural axis along which standard RAG can be probed and intervened on.
  arXiv  Detail & Related papers  (2026-05-14T07:14:19Z)
• Three Regimes of Context-Parametric Conflict: A Predictive Framework and Empirical Validation [0.0]
  We study how large language models handle conflict between their training knowledge and a contradicting document.<n>We propose a three-regime framework: Regime 1 (single-source updating), Regime 2 (competitive integration), and Regime 3 (task-appropriate selection)<n>We validate the framework across Claude Sonnet 4.6, GPT-5.5, Gemini 2.5 Flash, Llama 4 Maverick, and DeepSeek V3 using 9,970 API calls in three experimental phases.
  arXiv  Detail & Related papers  (2026-05-12T06:00:48Z)
• To Fuse or to Drop? Dual-Path Learning for Resolving Modality Conflicts in Multimodal Emotion Recognition [87.22988227382329]
  Multimodal emotion recognition (MER) benefits from combining text, audio, and vision, yet standard fusion often fails when modalities conflict.<n>We propose Dual-Path Conflict Resolution (DCR), a unified framework that learns when to fuse and when to drop modalities.
  arXiv  Detail & Related papers  (2026-05-06T13:11:33Z)
• Towards Trustworthy Depression Estimation via Disentangled Evidential Learning [50.22167852149165]
  We propose EviDep, an evidential learning framework that jointly quantifies depression severity.<n>EviDep enforces strict information integrity to guarantee robust evidence synthesis.<n>It achieves state-of-the-art predictive accuracy and superior uncertainty calibration, delivering a robust fail-safe mechanism for trustworthy clinical screening.
  arXiv  Detail & Related papers  (2026-04-17T13:27:11Z)
• Evaluating Evidence Grounding Under User Pressure in Instruction-Tuned Language Models [6.431677598656395]
  In contested domains, instruction-tuned language models must balance user-alignment pressures against faithfulness to the in-context evidence.<n>We conduct fine-grained ablations over evidence composition and uncertainty cues across 19 instruction-tuned models spanning 0.27B to 32B parameters.
  arXiv  Detail & Related papers  (2026-03-20T17:38:23Z)
• Certainty robustness: Evaluating LLM stability under self-challenging prompts [0.5156484100374058]
  Large language models (LLMs) often present answers with high apparent confidence despite lacking an explicit mechanism for reasoning about certainty or truth.<n>We introduce the Certainty Robustness Benchmark, a two-turn evaluation framework that measures how LLMs balance stability and adaptability under self-challenging prompts.<n>Using 200 reasoning and mathematics questions from LiveBench, we evaluate four state-of-the-art LLMs and distinguish between justified self-corrections and unjustified answer changes.
  arXiv  Detail & Related papers  (2026-02-10T18:07:51Z)
• Towards Reliable Medical LLMs: Benchmarking and Enhancing Confidence Estimation of Large Language Models in Medical Consultation [97.36081721024728]
  We propose the first benchmark for assessing confidence in multi-turn interaction during realistic medical consultations.<n>Our benchmark unifies three types of medical data for open-ended diagnostic generation.<n>We present MedConf, an evidence-grounded linguistic self-assessment framework.
  arXiv  Detail & Related papers  (2026-01-22T04:51:39Z)
• Parrot: Persuasion and Agreement Robustness Rating of Output Truth -- A Sycophancy Robustness Benchmark for LLMs [0.0]
  PARROT (Persuasion and Agreement Robustness Rating of Output Truth) is a robustness focused framework designed to measure the degradation in accuracy under social pressure exerted on users.<n>We evaluate 22 models using 1,302 MMLU-style multiple-choice questions across 13 domains and domain-specific authority templates.
  arXiv  Detail & Related papers  (2025-11-21T13:01:28Z)
• Zero-shot Faithful Factual Error Correction [53.121642212060536]
  Faithfully correcting factual errors is critical for maintaining the integrity of textual knowledge bases and preventing hallucinations in sequence-to-sequence models. We present a zero-shot framework that formulates questions about input claims, looks for correct answers in the given evidence, and assesses the faithfulness of each correction based on its consistency with the evidence.
  arXiv  Detail & Related papers  (2023-05-13T18:55:20Z)
• AmbiFC: Fact-Checking Ambiguous Claims with Evidence [57.7091560922174]
  We present AmbiFC, a fact-checking dataset with 10k claims derived from real-world information needs. We analyze disagreements arising from ambiguity when comparing claims against evidence in AmbiFC. We develop models for predicting veracity handling this ambiguity via soft labels.
  arXiv  Detail & Related papers  (2021-04-01T17:40:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.