DoGMaTiQ: Automated Generation of Question-and-Answer Nuggets for Report Evaluation

• URL: http://arxiv.org/abs/2605.04458v1
• Date: Wed, 06 May 2026 03:34:46 GMT
• Title: DoGMaTiQ: Automated Generation of Question-and-Answer Nuggets for Report Evaluation
• Abstract summary: DoGMaTiQ is a pipeline for generating high-quality QA-based nugget sets.<n>We integrate DoGMaTiQ with AutoArgue to enable fully automatic evaluation of generated reports.
• Score: 41.38684270729433
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evaluation of long-form, citation-backed reports has lately received significant attention due to the wide-scale adoption of retrieval-augmented generation (RAG) systems. Core to many evaluation frameworks is the use of atomic facts, or nuggets, to assess a report's coverage of query-relevant information attested in the underlying collection. While nuggets have traditionally been represented as short statements, recent work has used question-answer (QA) representations, enabling fine-grained evaluations that decouple the information need (i.e. the question) from the potentially diverse content that satisfies it (i.e. its answers). A persistent challenge for nugget-based evaluation is the need to manually curate sets of nuggets for each topic in a test collection -- a laborious process that scales poorly to novel information needs. This challenge is acute in cross-lingual settings, where information is found in multilingual source documents. Accordingly, we introduce DoGMaTiQ, a pipeline for generating high-quality QA-based nugget sets in three stages: (1) document-grounded nugget generation, (2) paraphrase clustering, and (3) nugget subselection based on principled quality criteria. We integrate DoGMaTiQ nuggets with AutoArgue -- a recent nugget-based evaluation framework -- to enable fully automatic evaluation of generated reports. We conduct extensive experiments on two cross-lingual TREC shared tasks, NeuCLIR and RAGTIME, showing strong rank correlations with both human-in-the-loop and fully manual judgments. Finally, detailed analysis of our pipeline reveals that a strong LLM nugget generator is key, and that the system rankings induced by DoGMaTiQ are robust to outlier systems. We facilitate future research in report evaluation by publicly releasing our code and artifacts at https://github.com/manestay/dogmatiq.

Related papers

• Incorporating Q&A Nuggets into Retrieval-Augmented Generation [23.32167679162754]
  Crucible is a Nugget-Augmented Generation System that preserves explicit citation provenance by constructing a bank of Q&A nuggets from retrieved documents.<n> Reasoning on nuggets avoids repeated information through clear and interpretable Q&A semantics.<n>Our system substantially outperforms Ginger, a recent nugget-based RAG system, in nugget recall, density, and citation grounding.
  arXiv  Detail & Related papers  (2026-01-19T16:57:33Z)
• ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering [54.72902502486611]
  ReAG is a Reasoning-Augmented Multimodal RAG approach that combines coarse- and fine-grained retrieval with a critic model that filters irrelevant passages.<n>ReAG significantly outperforms prior methods, improving answer accuracy and providing interpretable reasoning grounded in retrieved evidence.
  arXiv  Detail & Related papers  (2025-11-27T19:01:02Z)
• Chatbot Arena Meets Nuggets: Towards Explanations and Diagnostics in the Evaluation of LLM Responses [45.2769075498271]
  We apply our AutoNuggetizer framework to analyze data from roughly 7K Search Arena battles provided by LMArena.<n>Our results show a significant correlation between nugget scores and human preferences.
  arXiv  Detail & Related papers  (2025-04-28T17:24:36Z)
• The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models [53.12387628636912]
  We propose an automatic evaluation framework that is validated against human annotations.<n>This approach was originally developed for the TREC Question Answering (QA) Track in 2003.<n>We observe strong agreement at the run level between scores derived from fully automatic nugget evaluation and human-based variants.
  arXiv  Detail & Related papers  (2025-04-21T12:55:06Z)
• Conversational Gold: Evaluating Personalized Conversational Search System using Gold Nuggets [8.734527090842139]
  We introduce a new resource for assessing the retrieval effectiveness and relevance of response generated by RAG systems.<n>Our dataset extends to the TREC iKAT 2024 collection, which includes 17 conversations and 20,575 relevance passage assessments.
  arXiv  Detail & Related papers  (2025-03-12T23:44:10Z)
• Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines [17.803396998387665]
  Retrieval-augmented generation (RAG) has emerged to address the knowledge-intensive visual question answering (VQA) task.<n>We propose ReAuSE, an alternative to the previous RAG model for the knowledge-based VQA task.<n>Our model functions both as a generative retriever and an accurate answer generator.
  arXiv  Detail & Related papers  (2025-02-23T16:39:39Z)
• Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework [53.12387628636912]
  This report provides an initial look at partial results from the TREC 2024 Retrieval-Augmented Generation (RAG) Track. We have identified RAG evaluation as a barrier to continued progress in information access.
  arXiv  Detail & Related papers  (2024-11-14T17:25:43Z)
• PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models [72.57329554067195]
  ProxyQA is an innovative framework dedicated to assessing longtext generation. It comprises in-depth human-curated meta-questions spanning various domains, each accompanied by specific proxy-questions with pre-annotated answers. It assesses the generated content's quality through the evaluator's accuracy in addressing the proxy-questions.
  arXiv  Detail & Related papers  (2024-01-26T18:12:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.