Faithfulness-QA: A Counterfactual Entity Substitution Dataset for Training Context-Faithful RAG Models

• URL: http://arxiv.org/abs/2604.25313v2
• Date: Wed, 29 Apr 2026 17:00:16 GMT
• Title: Faithfulness-QA: A Counterfactual Entity Substitution Dataset for Training Context-Faithful RAG Models
• Abstract summary: Retrieval-Augmented Generation models produce answers grounded in parametric memory rather than the retrieved context.<n>We introduce Faithfulness-QA, a large-scale dataset of 99,094 samples constructed through counterfactual entity substitution.
• Score: 13.478990954968316
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Retrieval-Augmented Generation (RAG) models frequently produce answers grounded in parametric memory rather than the retrieved context, undermining the core promise of retrieval augmentation. A fundamental obstacle to fixing this unfaithfulness is the lack of training data that explicitly requires models to prefer context over internal knowledge. We introduce Faithfulness-QA, a large-scale dataset of 99,094 samples constructed through counterfactual entity substitution. Starting from two established extractive QA benchmarks--SQuAD and TriviaQA--we automatically identify answer-bearing named entities in each context, replace them with type-consistent alternatives drawn from a curated bank of 76,953 entities, and thereby manufacture controlled knowledge conflicts between context and parametric memory. Rigorous quality filtering ensures 100% pass rates across four automated checks on random 200-sample audits. We release the full dataset, the construction pipeline, and a typed entity bank covering eight named entity categories. Faithfulness-QA is designed as a training resource for attention-based faithfulness objectives and as an evaluation benchmark for measuring context-grounding behavior in RAG systems. Data and code are available at https://github.com/qzhangFDU/faithfulness-qa-dataset.

Related papers

• The Thousand-Graph Hypothesis: A Testable Hypothesis of Task-Conditioned Relation Materialization in Repository-Level Code Reasoning [2.7787094635105185]
  Training repository knowledge into models is costly and quickly stale, while local retrieval can miss scattered requirements.<n>We propose an entity-only external interface with task-conditioned relation materialization during inference.
  arXiv  Detail & Related papers  (2026-08-27T04:33:48Z)
• VeriSciQA: An Auto-Verified Dataset for Scientific Visual Question Answering [53.662676566188175]
  A key bottleneck lies in the lack of public, large-scale, high-quality Scientific Visual Question Answering (SVQA) datasets.<n>We propose a verification-centric Generate-then-Verify framework that first generates QA pairs with figure-associated textual context.<n>We instantiate this framework to curate VeriSciQA, a dataset of 20,351 QA pairs spanning 20 scientific domains and 12 figure types.
  arXiv  Detail & Related papers  (2025-11-25T04:14:52Z)
• Dynamic-KGQA: A Scalable Framework for Generating Adaptive Question Answering Datasets [9.785129730843435]
  We introduce Dynamic-KGQA, a scalable framework for generating adaptive QA datasets from knowledge graphs.<n>Unlike fixed benchmarks, Dynamic-KGQA generates a new dataset variant on every run while preserving the underlying distribution.<n>Dynamic-KGQA produces compact, semantically coherent subgraphs that facilitate both training and evaluation of KGQA models.
  arXiv  Detail & Related papers  (2025-03-06T23:58:01Z)
• Chain-of-Skills: A Configurable Model for Open-domain Question Answering [79.8644260578301]
  The retrieval model is an indispensable component for real-world knowledge-intensive tasks. Recent work focuses on customized methods, limiting the model transferability and scalability. We propose a modular retriever where individual modules correspond to key skills that can be reused across datasets.
  arXiv  Detail & Related papers  (2023-05-04T20:19:39Z)
• Zero-shot Fact Verification by Claim Generation [85.27523983027471]
  We develop QACG, a framework for training a robust fact verification model. We use automatically generated claims that can be supported, refuted, or unverifiable from evidence from Wikipedia. In a zero-shot scenario, QACG improves a RoBERTa model's F1 from 50% to 77%, equivalent in performance to 2K+ manually-curated examples.
  arXiv  Detail & Related papers  (2021-05-31T03:13:52Z)
• Generating Diverse and Consistent QA pairs from Contexts with Information-Maximizing Hierarchical Conditional VAEs [62.71505254770827]
  We propose a conditional variational autoencoder (HCVAE) for generating QA pairs given unstructured texts as contexts. Our model obtains impressive performance gains over all baselines on both tasks, using only a fraction of data for training.
  arXiv  Detail & Related papers  (2020-05-28T08:26:06Z)
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks [133.93803565077337]
  retrieval-augmented generation models combine pre-trained parametric and non-parametric memory for language generation. We show that RAG models generate more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline.
  arXiv  Detail & Related papers  (2020-05-22T21:34:34Z)
• Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering [98.48363619128108]
  We propose an unsupervised approach to training QA models with generated pseudo-training data. We show that generating questions for QA training by applying a simple template on a related, retrieved sentence rather than the original context sentence improves downstream QA performance.
  arXiv  Detail & Related papers  (2020-04-24T17:57:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.