Think Before You Attribute: Improving the Performance of LLMs Attribution Systems

• URL: http://arxiv.org/abs/2505.12621v1
• Date: Mon, 19 May 2025 02:08:20 GMT
• Title: Think Before You Attribute: Improving the Performance of LLMs Attribution Systems
• Abstract summary: We propose a sentence-level pre-attribution step for Retrieve-Augmented Generation (RAG) systems.<n>By separating sentences before attribution, a proper attribution method can be selected for the type of sentence, or the attribution can be skipped altogether.
• Score: 2.527698260421756
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) are increasingly applied in various science domains, yet their broader adoption remains constrained by a critical challenge: the lack of trustworthy, verifiable outputs. Current LLMs often generate answers without reliable source attribution, or worse, with incorrect attributions, posing a barrier to their use in scientific and high-stakes settings, where traceability and accountability are non-negotiable. To be reliable, attribution systems need high accuracy and retrieve data with short lengths, i.e., attribute to a sentence within a document rather than a whole document. We propose a sentence-level pre-attribution step for Retrieve-Augmented Generation (RAG) systems that classify sentences into three categories: not attributable, attributable to a single quote, and attributable to multiple quotes. By separating sentences before attribution, a proper attribution method can be selected for the type of sentence, or the attribution can be skipped altogether. Our results indicate that classifiers are well-suited for this task. In this work, we propose a pre-attribution step to reduce the computational complexity of attribution, provide a clean version of the HAGRID dataset, and provide an end-to-end attribution system that works out of the box.

Related papers

• Align-then-Slide: A complete evaluation framework for Ultra-Long Document-Level Machine Translation [26.418216341998953]
  We introduce textittextbfAlign-then-Slide, a complete evaluation framework for ultra-long doc-mt.<n>In the Align stage, we automatically infer sentence-level source-target correspondences and rebuild the target to match the source sentence number.<n>In the n-Chunk Sliding Evaluate stage, we calculate averaged metric scores under 1-, 2-, 3- and 4-chunk for multi-granularity assessment.
  arXiv  Detail & Related papers  (2025-09-04T01:50:20Z)
• RepreGuard: Detecting LLM-Generated Text by Revealing Hidden Representation Patterns [50.401907401444404]
  Large language models (LLMs) are crucial for preventing misuse and building trustworthy AI systems.<n>We propose RepreGuard, an efficient statistics-based detection method.<n> Experimental results show that RepreGuard outperforms all baselines with average 94.92% AUROC on both in-distribution (ID) and OOD scenarios.
  arXiv  Detail & Related papers  (2025-08-18T17:59:15Z)
• LAQuer: Localized Attribution Queries in Content-grounded Generation [69.60308443863606]
  Grounded text generation models often produce content that deviates from their source material, requiring user verification to ensure accuracy.<n>Existing attribution methods associate entire sentences with source documents, which can be overwhelming for users seeking to fact-check specific claims.<n>We introduce Localized Attribution Queries (LAQuer), a new task that localizes selected spans of generated output to their corresponding source spans, allowing fine-grained and user-directed attribution.
  arXiv  Detail & Related papers  (2025-06-01T21:46:23Z)
• Document Attribution: Examining Citation Relationships using Large Language Models [62.46146670035751]
  We propose a zero-shot approach that frames attribution as a straightforward textual entailment task.<n>We also explore the role of the attention mechanism in enhancing the attribution process.
  arXiv  Detail & Related papers  (2025-05-09T04:40:11Z)
• ClaimTrust: Propagation Trust Scoring for RAG Systems [7.7690689135107425]
  ClaimTrust is a propagation-based trust scoring framework that dynamically evaluates the reliability of documents in a RAG system.<n>We preprocess and analyze 814 political news articles to extract 2,173 unique claims and classify 965 meaningful relationships.<n>ClaimTrust iteratively updates trust scores until convergence, effectively differentiating trustworthy articles from unreliable ones.
  arXiv  Detail & Related papers  (2025-03-12T07:52:24Z)
• Attribute or Abstain: Large Language Models as Long Document Assistants [58.32043134560244]
  LLMs can help humans working with long documents, but are known to hallucinate. Existing approaches to attribution have only been evaluated in RAG settings, where the initial retrieval confounds LLM performance. This is crucially different from the long document setting, where retrieval is not needed, but could help. We present LAB, a benchmark of 6 diverse long document tasks with attribution, and experiments with different approaches to attribution on 5 LLMs of different sizes.
  arXiv  Detail & Related papers  (2024-07-10T16:16:02Z)
• Ground Every Sentence: Improving Retrieval-Augmented LLMs with Interleaved Reference-Claim Generation [51.8188846284153]
  Attributed Text Generation (ATG) is proposed to enhance credibility and verifiability in RAG systems.<n>This paper proposes ReClaim, a fine-grained ATG method that alternates the generation of references and answers step by step.<n>With extensive experiments, we verify the effectiveness of ReClaim in extensive settings, achieving a citation accuracy rate of 90%.
  arXiv  Detail & Related papers  (2024-07-01T20:47:47Z)
• WebCiteS: Attributed Query-Focused Summarization on Chinese Web Search Results with Citations [34.99831757956635]
  We formulate the task of attributed query-focused summarization (AQFS) and present WebCiteS, a Chinese dataset featuring 7k human-annotated summaries with citations. We tackle these issues by developing detailed metrics and enabling the automatic evaluator to decompose the sentences into sub-claims for fine-grained verification.
  arXiv  Detail & Related papers  (2024-03-04T07:06:41Z)
• Rescue: Ranking LLM Responses with Partial Ordering to Improve Response Generation [28.89786334298637]
  We develop a novel method to optimize LLMs using ranking metrics. Rather than a traditional full ordering, we advocate for a partial ordering. We test our system's improved response generation ability using benchmark datasets.
  arXiv  Detail & Related papers  (2023-11-15T17:27:14Z)
• SLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Window [24.524282909076767]
  We present a metric named SLIDE (SLIding Document Evaluator) which operates on blocks of sentences. We find that SLIDE obtains significantly higher pairwise system accuracy than its sentence-level baseline. This suggests that source context may provide the same information as a human reference in disambiguating source ambiguities.
  arXiv  Detail & Related papers  (2023-09-16T01:30:58Z)
• RARR: Researching and Revising What Language Models Say, Using Language Models [31.057495176599502]
  We propose RARR (Retrofit Attribution using Research and Revision), a system that automatically finds attribution for the output of any text generation model. We find that RARR significantly improves attribution while otherwise preserving the original input to a much greater degree than previously explored edit models.
  arXiv  Detail & Related papers  (2022-10-17T03:44:30Z)
• Factual Error Correction for Abstractive Summaries Using Entity Retrieval [57.01193722520597]
  We propose an efficient factual error correction system RFEC based on entities retrieval post-editing process. RFEC retrieves the evidence sentences from the original document by comparing the sentences with the target summary. Next, RFEC detects the entity-level errors in the summaries by considering the evidence sentences and substitutes the wrong entities with the accurate entities from the evidence sentences.
  arXiv  Detail & Related papers  (2022-04-18T11:35:02Z)
• GERE: Generative Evidence Retrieval for Fact Verification [57.78768817972026]
  We propose GERE, the first system that retrieves evidences in a generative fashion. The experimental results on the FEVER dataset show that GERE achieves significant improvements over the state-of-the-art baselines.
  arXiv  Detail & Related papers  (2022-04-12T03:49:35Z)
• Text Summarization with Latent Queries [60.468323530248945]
  We introduce LaQSum, the first unified text summarization system that learns Latent Queries from documents for abstractive summarization with any existing query forms. Under a deep generative framework, our system jointly optimize a latent query model and a conditional language model, allowing users to plug-and-play queries of any type at test time. Our system robustly outperforms strong comparison systems across summarization benchmarks with different query types, document settings, and target domains.
  arXiv  Detail & Related papers  (2021-05-31T21:14:58Z)
• Context-Based Quotation Recommendation [60.93257124507105]
  We propose a novel context-aware quote recommendation system. It generates a ranked list of quotable paragraphs and spans of tokens from a given source document. We conduct experiments on a collection of speech transcripts and associated news articles.
  arXiv  Detail & Related papers  (2020-05-17T17:49:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.