Unstructured Evidence Attribution for Long Context Query Focused Summarization

• URL: http://arxiv.org/abs/2502.14409v2
• Date: Thu, 30 Oct 2025 15:05:42 GMT
• Title: Unstructured Evidence Attribution for Long Context Query Focused Summarization
• Authors: Dustin Wright, Zain Muhammad Mujahid, Lu Wang, Isabelle Augenstein, David Jurgens,
• Abstract summary: We propose to extract unstructured (i.e., spans of any length) evidence in order to acquire more relevant and consistent evidence than in the fixed granularity case.<n>We show how existing systems struggle to copy and properly cite unstructured evidence, which also tends to be "lost-in-the-middle"
• Score: 53.08341620504465
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are capable of generating coherent summaries from very long contexts given a user query, and extracting and citing evidence spans helps improve the trustworthiness of these summaries. Whereas previous work has focused on evidence citation with fixed levels of granularity (e.g. sentence, paragraph, document, etc.), we propose to extract unstructured (i.e., spans of any length) evidence in order to acquire more relevant and consistent evidence than in the fixed granularity case. We show how existing systems struggle to copy and properly cite unstructured evidence, which also tends to be "lost-in-the-middle". To help models perform this task, we create the Summaries with Unstructured Evidence Text dataset (SUnsET), a synthetic dataset generated using a novel pipeline, which can be used as training supervision for unstructured evidence summarization. We demonstrate across 5 LLMs and 4 datasets spanning human written, synthetic, single, and multi-document settings that LLMs adapted with SUnsET generate more relevant and factually consistent evidence with their summaries, extract evidence from more diverse locations in their context, and can generate more relevant and consistent summaries than baselines with no fine-tuning and fixed granularity evidence. We release SUnsET and our generation code to the public.

Related papers

• Multilingual, Multimodal Pipeline for Creating Authentic and Structured Fact-Checked Claim Dataset [3.1256048031872425]
  This paper introduces a comprehensive data collection and processing pipeline that constructs multimodal fact-checking datasets in French and German languages.<n>We used state-of-the-art large language models (LLMs) and multimodal LLMs for (i) evidence extraction under predefined evidence categories and (ii) justification generation that links evidence to verdicts.
  arXiv  Detail & Related papers  (2026-01-12T20:33:46Z)
• DeepSynth-Eval: Objectively Evaluating Information Consolidation in Deep Survey Writing [53.85037373860246]
  We introduce Deep Synth-Eval, a benchmark designed to objectively evaluate information consolidation capabilities.<n>We propose a fine-grained evaluation protocol using General Checklists (for factual coverage) and Constraint Checklists (for structural organization)<n>Our results demonstrate that agentic plan-and-write significantly outperform single-turn generation.
  arXiv  Detail & Related papers  (2026-01-07T03:07:52Z)
• VeriCite: Towards Reliable Citations in Retrieval-Augmented Generation via Rigorous Verification [107.75781898355562]
  We introduce a novel framework, called VeriCite, designed to rigorously validate supporting evidence and enhance answer attribution.<n>We conduct experiments across five open-source LLMs and four datasets, demonstrating that VeriCite can significantly improve citation quality while maintaining the correctness of the answers.
  arXiv  Detail & Related papers  (2025-10-13T13:38:54Z)
• Attribution, Citation, and Quotation: A Survey of Evidence-based Text Generation with Large Language Models [9.664217498808338]
  We introduce a unified taxonomy of evidence-based text generation with large language models.<n>We investigate 300 evaluation metrics across seven key dimensions.<n>We highlight open challenges and outline promising directions for future work.
  arXiv  Detail & Related papers  (2025-08-21T09:36:35Z)
• Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction [28.47810405584841]
  Arranged and Organized Extraction Benchmark designed to evaluate ability of large language models to comprehend fragmented documents.<n>AOE includes 11 carefully crafted tasks across three diverse domains, requiring models to generate context-specific schema tailored to varied input queries.<n>Results show that even the most advanced models struggled significantly.
  arXiv  Detail & Related papers  (2025-07-22T06:37:51Z)
• Large Language Models are Good Relational Learners [55.40941576497973]
  We introduce Rel-LLM, a novel architecture that utilizes a graph neural network (GNN)- based encoder to generate structured relational prompts for large language models (LLMs)<n>Unlike traditional text-based serialization approaches, our method preserves the inherent relational structure of databases while enabling LLMs to process and reason over complex entity relationships.
  arXiv  Detail & Related papers  (2025-06-06T04:07:55Z)
• Latent Factor Models Meets Instructions: Goal-conditioned Latent Factor Discovery without Task Supervision [50.45597801390757]
  Instruct-LF is a goal-oriented latent factor discovery system. It integrates instruction-following ability with statistical models to handle noisy datasets.
  arXiv  Detail & Related papers  (2025-02-21T02:03:08Z)
• Context-Aware Hierarchical Merging for Long Document Summarization [56.96619074316232]
  We propose different approaches to enrich hierarchical merging with context from the source document.<n> Experimental results on datasets representing legal and narrative domains show that contextual augmentation consistently outperforms zero-shot and hierarchical merging baselines.
  arXiv  Detail & Related papers  (2025-02-03T01:14:31Z)
• HERA: Improving Long Document Summarization using Large Language Models with Context Packaging and Reordering [6.876612430571396]
  We propose a novel summary generation framework, called HERA.<n>We first segment a long document by its semantic structure and retrieve text segments about the same event, and finally reorder them to form the input context.<n>The experimental results show that HERA outperforms foundation models in ROUGE, BERTScore and faithfulness metrics.
  arXiv  Detail & Related papers  (2025-02-01T14:55:06Z)
• GEGA: Graph Convolutional Networks and Evidence Retrieval Guided Attention for Enhanced Document-level Relation Extraction [15.246183329778656]
  Document-level relation extraction (DocRE) aims to extract relations between entities from unstructured document text. To overcome these challenges, we propose GEGA, a novel model for DocRE. We evaluate the GEGA model on three widely used benchmark datasets: DocRED, Re-DocRED, and Revisit-DocRED.
  arXiv  Detail & Related papers  (2024-07-31T07:15:33Z)
• Multimodal Misinformation Detection using Large Vision-Language Models [7.505532091249881]
  Large language models (LLMs) have shown remarkable performance in various tasks. Few approaches consider evidence retrieval as part of misinformation detection. We propose a novel re-ranking approach for multimodal evidence retrieval.
  arXiv  Detail & Related papers  (2024-07-19T13:57:11Z)
• Attribute or Abstain: Large Language Models as Long Document Assistants [58.32043134560244]
  LLMs can help humans working with long documents, but are known to hallucinate. Existing approaches to attribution have only been evaluated in RAG settings, where the initial retrieval confounds LLM performance. This is crucially different from the long document setting, where retrieval is not needed, but could help. We present LAB, a benchmark of 6 diverse long document tasks with attribution, and experiments with different approaches to attribution on 5 LLMs of different sizes.
  arXiv  Detail & Related papers  (2024-07-10T16:16:02Z)
• Retrieval Augmented Fact Verification by Synthesizing Contrastive Arguments [23.639378586798884]
  We propose retrieval augmented fact verification through the synthesis of contrasting arguments. Our method effectively retrieves relevant documents as evidence and evaluates arguments from varying perspectives. We demonstrate the effectiveness of our method through extensive experiments, where RAFTS can outperform GPT-based methods with a significantly smaller 7B LLM.
  arXiv  Detail & Related papers  (2024-06-14T08:13:34Z)
• Learning to Reduce: Optimal Representations of Structured Data in Prompting Large Language Models [42.16047343029512]
  Large Language Models (LLMs) have been widely used as general-purpose AI agents. We propose a framework, Learning to Reduce, that fine-tunes a language model to generate a reduced version of an input context. We show that our model achieves comparable accuracies in selecting the relevant evidence from an input context.
  arXiv  Detail & Related papers  (2024-02-22T00:41:23Z)
• Blinded by Generated Contexts: How Language Models Merge Generated and Retrieved Contexts When Knowledge Conflicts? [45.233517779029334]
  We identify whether responses are attributed to generated or retrieved contexts. Experiments reveal a significant bias in several LLMs to favor generated contexts, even when they provide incorrect information.
  arXiv  Detail & Related papers  (2024-01-22T12:54:04Z)
• Effective Large Language Model Adaptation for Improved Grounding and Citation Generation [48.07830615309543]
  This paper focuses on improving large language models (LLMs) by grounding their responses in retrieved passages and by providing citations. We propose a new framework, AGREE, that improves the grounding from a holistic perspective. Our framework tunes LLMs to selfground the claims in their responses and provide accurate citations to retrieved documents.
  arXiv  Detail & Related papers  (2023-11-16T03:22:25Z)
• On Context Utilization in Summarization with Large Language Models [83.84459732796302]
  Large language models (LLMs) excel in abstractive summarization tasks, delivering fluent and pertinent summaries. Recent advancements have extended their capabilities to handle long-input contexts, exceeding 100k tokens. We conduct the first comprehensive study on context utilization and position bias in summarization.
  arXiv  Detail & Related papers  (2023-10-16T16:45:12Z)
• Description-Based Text Similarity [59.552704474862004]
  We identify the need to search for texts based on abstract descriptions of their content. We propose an alternative model that significantly improves when used in standard nearest neighbor search.
  arXiv  Detail & Related papers  (2023-05-21T17:14:31Z)
• Text Summarization with Latent Queries [60.468323530248945]
  We introduce LaQSum, the first unified text summarization system that learns Latent Queries from documents for abstractive summarization with any existing query forms. Under a deep generative framework, our system jointly optimize a latent query model and a conditional language model, allowing users to plug-and-play queries of any type at test time. Our system robustly outperforms strong comparison systems across summarization benchmarks with different query types, document settings, and target domains.
  arXiv  Detail & Related papers  (2021-05-31T21:14:58Z)
• Hierarchical Evidence Set Modeling for Automated Fact Extraction and Verification [5.836068916903788]
  Hierarchical Evidence Set Modeling (HESM) is a framework to extract evidence sets and verify a claim to be supported, refuted or not enough info. Our experimental results show that HESM outperforms 7 state-of-the-art methods for fact extraction and claim verification.
  arXiv  Detail & Related papers  (2020-10-10T22:27:17Z)
• StructSum: Summarization via Structured Representations [27.890477913486787]
  Abstractive text summarization aims at compressing the information of a long source document into a condensed summary. Despite advances in modeling techniques, abstractive summarization models still suffer from several key challenges. We propose a framework based on document-level structure induction for summarization to address these challenges.
  arXiv  Detail & Related papers  (2020-03-01T20:32:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.