EviScope: Paired Counterfactual Evidence Diagnostics for Faithful and Efficient Grounded Language Models

• URL: http://arxiv.org/abs/2609.17081v1
• Date: Tue, 15 Sep 2026 12:18:19 GMT
• Title: EviScope: Paired Counterfactual Evidence Diagnostics for Faithful and Efficient Grounded Language Models
• Abstract summary: Grounded language-model systems are often evaluated by final answer accuracy, yet a correct answer can be unsupported, drawn from the wrong source, or produced when evidence is insufficient or contradictory.<n>We introduce EviScope, a paired counterfactual benchmark that holds the question fixed while adding, removing, distracting, or contradicting its evidence.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Grounded language-model systems are often evaluated by final answer accuracy, yet a correct answer can be unsupported, drawn from the wrong source, or produced when evidence is insufficient or contradictory. We introduce EviScope, a paired counterfactual benchmark that holds the question fixed while adding, removing, distracting, or contradicting its evidence. EviScope-v1.1 contains 40 four-condition quartets with repaired counterfactual claims and span-level support labels for automatic evaluation. Across 960 gold-blind generations from Qwen2.5-7B, Llama 3.1 8B, and Gemini 3.5 Flash, paired metrics expose model-dependent grounding behavior that answer accuracy hides. On two local open models, an explicit evidence-action gate underperforms vanilla RAG on QCS: 0.15 vs. 0.50 for Qwen and 0.10 vs. 0.375 for Llama. Gemini reaches 0.944 joint success under both prompts, yet still answers 5% of conflict cases after contradiction insertion. EviScope therefore distinguishes unsupported answering, conflict blindness, and wrong non-answer actions rather than scoring answers alone.

Related papers

• When Do Internal Probes Beat Reading the Answer? Miscalibrated Readouts and Behavior-Concealed Knowledge in Language Models [0.0]
  A language model, asked to verify 1,200 logical conclusions, answers YES every time.<n>linear probes on its hidden states read the correct verdict at 0.96 AUC.<n>We ask where the verdict is lost, and find the dominant failure is a single scalar.<n>The diagnosis generalizes: across 90 semantic-label configurations of a five-model, three-family factorial, behavioral accuracy collapses onto a single function of threshold offset.
  arXiv  Detail & Related papers  (2026-09-04T00:26:45Z)
• LOCI: A Locator-Critic with Refinement Loop [65.51043732421378]
  Vision-Language Models (VLMs) still struggle on tasks requiring complex visual understanding.<n>We argue that the core issue is not high-level reasoning, but instead failing to locate critical details in the image.<n>We propose Locator-Critic (LOCI), a training-free framework that decouples visual search from evidence verification.
  arXiv  Detail & Related papers  (2026-08-31T15:26:00Z)
• Same Evidence, Different Target: Decoding How Diagnostic Evidence Bears on Causal Questions from Language-Model States [7.032871370232397]
  We introduce paired prompts that repeat the same diagnostic evidence verbatim while changing the causal target.<n>Each prompt is labeled Favors, Challenges, Unresolved, or Wrong Target according to how the evidence bears on the causal question.<n>We analyze the final-token hidden state from the penultimate transformer block of Qwen2.5-7B-Instruct.
  arXiv  Detail & Related papers  (2026-07-29T13:56:37Z)
• Two Regimes of Chain-of-Thought Unfaithfulness: Behavioral Detection Fails Where Models Are Wrong [1.2744523252873352]
  Black-box (behavioral) detection of unfaithful CoT against FaithCoT-Bench's human annotations.<n>We find answer correctness structures the problem at every level.
  arXiv  Detail & Related papers  (2026-07-26T04:43:53Z)
• Abliteration Is Not a Scalpel: Off-Target Effects of Refusal Removal on Decision Disposition Across Model Families [51.56484100374058]
  Abliteration - deleting a model's refusal direction from its weights - is the standard recipe behind popular "uncensored" open-weight models.<n>As a disposition probe we use 21,600 decisions under uncertainty, replayed through a frozen pipeline so the decision-layer model is the only variable.<n>Three effects replicate across both families (weeks-clustered CIs excluding zero)<n>A fourth effect reverses sign: the same operation makes Gemma-abliterated less confident and Qwen-abliterated more (family CIs non-overlapping)
  arXiv  Detail & Related papers  (2026-07-19T22:27:00Z)
• AdversaBench: Automated LLM Red-Teaming with Multi-Judge Confirmation and Cross-Model Transferability [16.27940179594792]
  AdversaBench is an end-to-end red-teaming pipeline that mutates seed prompts with five structured operators.<n>We report experiments on 45 seeds across three categories: reasoning, instruction-following, and tool use.
  arXiv  Detail & Related papers  (2026-06-23T13:50:51Z)
• Question-Aware Evidence Ledgers for Video Relational Reasoning [49.15616982043187]
  We present a test-time reasoning pipeline built around a strong GPT-5.5 video QA solver and a set of question-aware evidence ledgers.<n>The pipeline achieves 92.95% overall accuracy and 93.79% macro accuracy on the challenge test split.
  arXiv  Detail & Related papers  (2026-06-01T17:18:03Z)
• SURE-RAG: Sufficiency and Uncertainty-Aware Evidence Verification for Selective Retrieval-Augmented Generation [6.604874054866016]
  We present SURE-RAG, a transparent aggregation protocol built on the observation that evidence sufficiency is a set-level property.<n>A shared pair-level claim-evidence verifier produces local relation distributions, which SURE-RAG aggregates into interpretable answer-level signals.<n>We evaluate on HotpotQA-RAG v3, a controlled multi-hop benchmark, under an artifact-aware protocol.
  arXiv  Detail & Related papers  (2026-05-05T09:05:40Z)
• VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification [73.02304272829785]
  VideoBench is a hierarchical benchmark for long-video answering that rigorously verifies evidence.<n>It comprises 500 manually annotated questions across 13 domains, paired with temporal intervals and spatial bounding boxes as evidence.<n>Experiments show that even GeminiPro-3-Pro correctly answers fewer than 17% of questions under the standard end-to-end QA setting.<n>Results expose a significant gap between surface-level answer correctness and genuine evidence-based reasoning.
  arXiv  Detail & Related papers  (2026-04-02T03:29:43Z)
• Probing for Knowledge Attribution in Large Language Models [45.47366023067617]
  Large language models (LLMs) often generate fluent but unfounded claims, or hallucinations.<n> Proper mitigation depends on knowing whether a model's answer is based on the prompt or its internal weights.<n>We show that a probe, a simple linear classifier trained on model hidden representations, can reliably predict contributive attribution.
  arXiv  Detail & Related papers  (2026-02-26T09:21:12Z)
• Decomposed Prompting Does Not Fix Knowledge Gaps, But Helps Models Say "I Don't Know" [47.930782177987446]
  Large language models often struggle to recognize their knowledge limits in closed-book question answering, leading to confident hallucinations.<n>We evaluate three task-equivalent prompting regimes: Direct, Assistive, and Incremental, across different model scales and multi-hop QA benchmarks.<n>Because factual knowledge is stable while hallucinations are agreement, cross-regime provides a precise signal of internal uncertainty.
  arXiv  Detail & Related papers  (2026-02-04T18:39:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.