X-MADAM-RAG: Diagnosing and Handling Chinese-English Evidence Conflict in Retrieval-Augmented Generation

• URL: http://arxiv.org/abs/2606.12903v1
• Date: Thu, 11 Jun 2026 04:59:35 GMT
• Title: X-MADAM-RAG: Diagnosing and Handling Chinese-English Evidence Conflict in Retrieval-Augmented Generation
• Abstract summary: Retrieval-augmented generation (RAG) systems may receive evidence that is not merely noisy but mutually contradictory.<n>We study this problem through X-RAMDocs-ZHEN, a controlled Chinese-English benchmark for diagnosing evidence conflict in RAG.<n>X-MADAM-RAG is an interpretable pipeline that decomposes evidence handling into per-document candidate extraction, visible-evidence repair, deterministic candidate grouping, and conflict-aware aggregation.
• Score: 8.057966659934769
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Retrieval-augmented generation (RAG) systems may receive evidence that is not merely noisy but mutually contradictory. This issue becomes particularly salient in multilingual settings, where retrieved Chinese and English evidence may support incompatible answer candidates. We study this problem through X-RAMDocs-ZHEN, a controlled Chinese-English benchmark derived from RAMDocs for diagnosing evidence conflict in RAG. The benchmark contains 300 examples across six balanced conditions, including monolingual support, bilingual agreement, reversed conflict directions, and conflict with optional noise. We further examine X-MADAM-RAG, an interpretable pipeline that decomposes evidence handling into per-document candidate extraction, visible-evidence repair, deterministic candidate grouping, and conflict-aware aggregation. On the original controlled benchmark with Qwen2.5-7B-Instruct, X-MADAM-RAG achieves 0.9667 strict accuracy and 0.9767 conflict-aware success, outperforming an evidence-normalized single-call baseline. However, a zero-call rule-only extractor reaches 1.0000 on the same benchmark, revealing strong template regularity. To probe this limitation, we construct a deterministic naturalized stress test that removes explicit answer templates while preserving candidate strings. On its 100-sample subset, rule-only extraction falls to 0.0000, but X-MADAM-RAG also drops to 0.3000 strict accuracy, below both naive and evidence-normalized baselines. A privileged oracle remains perfect, indicating that document-level extraction is the main bottleneck. These findings position X-RAMDocs-ZHEN and X-MADAM-RAG as diagnostic tools for controlled evidence conflict rather than as evidence of general hallucination detection or robustness to natural retrieval.

Related papers

• NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment [54.4265627247104]
  Large language models (LLMs) are increasingly used in peer review at major AI conferences.<n>Existing benchmarks assess novelty as a single holistic score.<n>We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis.
  arXiv  Detail & Related papers  (2026-09-10T08:34:56Z)
• VARM-Bench: Benchmarking Verifiable Structured Reasoning in Chinese Abusive Speech Moderation [12.885908330308046]
  VARM-Bench is a benchmark for deterministically verifying the stated basis of a moderation decision.<n>Our protocol evaluates field correctness, target alignment, output validity, complete-record agreement, and hidden record errors.<n>Results show that strong label-level performance can conceal substantial errors in complete moderation records.
  arXiv  Detail & Related papers  (2026-08-16T07:52:31Z)
• HALLMARK: Diagnosing Three Failure Modes in LLM Citation Verifiers [30.02579334273979]
  GPTZero found 53 papers with hallucinated citations among NeurIPS 2025's accepted set.<n>No shared benchmark compares verifiers and gives detailed failure diagnostics.<n>Hallucination benchmark: 2,526 Bib entries spanning 14 types, three difficulty tiers, six diagnostic sub-tests per entry.
  arXiv  Detail & Related papers  (2026-07-20T13:21:27Z)
• When RLVR Shrinks the Reasoning Boundary: Diagnosing Pass@k Inversion [0.0]
  Reinforcement learning with verifiable rewards (RLVR) can improve one-sample accuracy while making a model worse under repeated sampling.<n>We study this pass@k inversion: after training, the policy may solve fewer distinct problems than its base model at large $k$.<n>The failure concentrates on boundary prompts, where the base model contains rare correct trajectories that are recoverable by sampling but too sparse to reliably appear in finite RLVR rollout groups.
  arXiv  Detail & Related papers  (2026-07-12T17:17:18Z)
• AbstRAG: Learning to Abstract for Retrieval Problems [23.379898867224355]
  AbstRAG decomposes the query--evidence gap into expression, conceptual, intent--evidence, and event-type components.<n>It outperforms on nDCG@10 in 18 of 21 paired-bootstrap contrasts and improves generation accuracy by 1.9%, 5.2%, and 4.0%.
  arXiv  Detail & Related papers  (2026-06-08T13:14:01Z)
• Decision-Aware Memory Cards: Counterfactual-Inspired Context Selection and Compression for Tool-Using LLM Agents [3.964533007623828]
  CICL turns instance evidence into a context graph, routes deterministic, Opus-assisted, Qwen, Codex/GPT-5.5, and Qwen-QLoRA judgments.<n> CICL yields a concrete open-benchmark gain while exposing its limits.
  arXiv  Detail & Related papers  (2026-06-06T13:02:28Z)
• Diagnosing Evidence Utilization in Long-Context and Retrieval-Augmented Language Models under Matched Evidence Conditions [0.0]
  A model may answer from parametric priors, fail to use evidence that is present, or cite relevant text without converting it into the final answer.<n>This paper introduces a four-condition diagnostic protocol for evidence-utilization evaluation.
  arXiv  Detail & Related papers  (2026-06-04T22:44:57Z)
• What Are We Actually Decoding? Source Attribution for Non-Invasive Brain-to-Language Retrieval [42.66754319854329]
  We recast stimulus-locked MEG-to-audio retrieval as an auditing framework.<n>We use structural shortcuts, window-level stimulus-locked evidence, and cross-window contextual aggregation.<n>These results suggest that brain-to-language performance should be source-attributed, not merely reported.
  arXiv  Detail & Related papers  (2026-05-23T11:23:39Z)
• SURE-RAG: Sufficiency and Uncertainty-Aware Evidence Verification for Selective Retrieval-Augmented Generation [6.604874054866016]
  We present SURE-RAG, a transparent aggregation protocol built on the observation that evidence sufficiency is a set-level property.<n>A shared pair-level claim-evidence verifier produces local relation distributions, which SURE-RAG aggregates into interpretable answer-level signals.<n>We evaluate on HotpotQA-RAG v3, a controlled multi-hop benchmark, under an artifact-aware protocol.
  arXiv  Detail & Related papers  (2026-05-05T09:05:40Z)
• All Languages Matter: Understanding and Mitigating Language Bias in Multilingual RAG [87.74635133954975]
  We show that current Multilingual Retrieval-Augmented Generation (mRAG) systems suffer from a language bias during reranking.<n>We propose textittextbfLanguage-textbfAgnostic textbfUtility-driven textbfReranker textbfAlignment (LAURA), which aligns multilingual evidence ranking with downstream generative utility.
  arXiv  Detail & Related papers  (2026-04-22T05:33:06Z)
• PAR$^2$-RAG: Planned Active Retrieval and Reasoning for Multi-Hop Question Answering [57.89576196160413]
  Large language models (LLMs) remain brittle on multi-hop question answering (MHQA)<n>We propose textbfPlanned Active Retrieval and Reasoning RAG (PAR$2$-RAG), a two-stage framework that separates emphcoverage from emphcommitment.
  arXiv  Detail & Related papers  (2026-03-30T23:52:54Z)
• HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam [63.84155758655084]
  Humanity's Last Exam (HLE) has become a widely used benchmark for evaluating frontier large language models.<n>We introduce HLE-Verified, a verified and revised version of HLE with a transparent verification protocol and fine-grained error taxonomy.<n>We evaluate seven state-of-the-art language models on HLE and HLE-Verified, observing an average absolute accuracy gain of 7--10 percentage points.
  arXiv  Detail & Related papers  (2026-02-15T02:50:15Z)
• Retrieval is Not Enough: Enhancing RAG Reasoning through Test-Time Critique and Optimization [58.390885294401066]
  Retrieval-augmented generation (RAG) has become a widely adopted paradigm for enabling knowledge-grounded large language models (LLMs)<n>RAG pipelines often fail to ensure that model reasoning remains consistent with the evidence retrieved, leading to factual inconsistencies or unsupported conclusions.<n>We propose AlignRAG, a novel iterative framework grounded in Critique-Driven Alignment (CDA)<n>We introduce AlignRAG-auto, an autonomous variant that dynamically terminates refinement, removing the need to pre-specify the number of critique iterations.
  arXiv  Detail & Related papers  (2025-04-21T04:56:47Z)
• Retrieval-Augmented Generation with Conflicting Evidence [57.66282463340297]
  Large language model (LLM) agents are increasingly employing retrieval-augmented generation (RAG) to improve the factuality of their responses.<n>In practice, these systems often need to handle ambiguous user queries and potentially conflicting information from multiple sources.<n>We propose RAMDocs (Retrieval with Ambiguity and Misinformation in Documents), a new dataset that simulates complex and realistic scenarios for conflicting evidence for a user query.
  arXiv  Detail & Related papers  (2025-04-17T16:46:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.