Attention with Dependency Parsing Augmentation for Fine-Grained Attribution

• URL: http://arxiv.org/abs/2412.11404v1
• Date: Mon, 16 Dec 2024 03:12:13 GMT
• Title: Attention with Dependency Parsing Augmentation for Fine-Grained Attribution
• Authors: Qiang Ding, Lvzhou Luo, Yixuan Cao, Ping Luo,
• Abstract summary: We develop a fine-grained attribution mechanism that provides supporting evidence from retrieved documents for every answer span.<n>Existing attribution methods rely on model-internal similarity metrics between responses and documents, such as saliency scores and hidden state similarity.<n>We propose two techniques applicable to all model-internals-based methods. First, we aggregate token-wise evidence through set union operations, preserving the granularity of representations.<n>Second, we enhance the attributor by integrating dependency parsing to enrich the semantic completeness of target spans.
• Score: 26.603281615221505
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To assist humans in efficiently validating RAG-generated content, developing a fine-grained attribution mechanism that provides supporting evidence from retrieved documents for every answer span is essential. Existing fine-grained attribution methods rely on model-internal similarity metrics between responses and documents, such as saliency scores and hidden state similarity. However, these approaches suffer from either high computational complexity or coarse-grained representations. Additionally, a common problem shared by the previous works is their reliance on decoder-only Transformers, limiting their ability to incorporate contextual information after the target span. To address the above problems, we propose two techniques applicable to all model-internals-based methods. First, we aggregate token-wise evidence through set union operations, preserving the granularity of representations. Second, we enhance the attributor by integrating dependency parsing to enrich the semantic completeness of target spans. For practical implementation, our approach employs attention weights as the similarity metric. Experimental results demonstrate that the proposed method consistently outperforms all prior works.

Related papers

• Knowledge Graph Completion with Relation-Aware Anchor Enhancement [50.50944396454757]
  We propose a relation-aware anchor enhanced knowledge graph completion method (RAA-KGC) We first generate anchor entities within the relation-aware neighborhood of the head entity. Then, by pulling the query embedding towards the neighborhoods of the anchors, it is tuned to be more discriminative for target entity matching.
  arXiv  Detail & Related papers  (2025-04-08T15:22:08Z)
• Unified Semantic and ID Representation Learning for Deep Recommenders [28.709935854073535]
  We propose a Unified Semantic and ID Representation Learning framework. In our framework, ID tokens capture unique item attributes, while semantic tokens represent shared, transferable characteristics. Our framework integrates cosine similarity in earlier layers and Euclidean distance in the final layer to optimize representation learning.
  arXiv  Detail & Related papers  (2025-02-23T07:17:28Z)
• Enhanced Retrieval of Long Documents: Leveraging Fine-Grained Block Representations with Large Language Models [24.02950598944251]
  We introduce a novel, fine-grained approach aimed at enhancing the accuracy of relevance scoring for long documents. Our methodology firstly segments a long document into blocks, each of which is embedded using an LLM. We aggregate the query-block relevance scores through a weighted sum method, yielding a comprehensive score for the query with the entire document.
  arXiv  Detail & Related papers  (2025-01-28T16:03:52Z)
• Towards Optimal Aggregation of Varying Range Dependencies in Haze Removal [17.29370328189668]
  Haze removal aims to restore a clear image from a hazy input. Existing methods have shown significant efficacy by capturing either short-range dependencies for local detail preservation or long-range dependencies for global context modeling. We propose bfDehazeMatic, which captures both short- and long-range dependencies through dual-path design for improved restoration.
  arXiv  Detail & Related papers  (2024-08-22T11:51:50Z)
• Fusion Makes Perfection: An Efficient Multi-Grained Matching Approach for Zero-Shot Relation Extraction [9.657656957845987]
  We propose an efficient multi-grained matching approach that uses virtual entity matching to reduce manual annotation cost. Experimental results show that our approach outperforms the previous State Of The Art (SOTA) methods.
  arXiv  Detail & Related papers  (2024-06-17T11:31:48Z)
• Enhancing Retrieval-Augmented LMs with a Two-stage Consistency Learning Compressor [4.35807211471107]
  This work proposes a novel two-stage consistency learning approach for retrieved information compression in retrieval-augmented language models. The proposed method is empirically validated across multiple datasets, demonstrating notable enhancements in precision and efficiency for question-answering tasks.
  arXiv  Detail & Related papers  (2024-06-04T12:43:23Z)
• Unifying Feature and Cost Aggregation with Transformers for Semantic and Visual Correspondence [51.54175067684008]
  This paper introduces a Transformer-based integrative feature and cost aggregation network designed for dense matching tasks. We first show that feature aggregation and cost aggregation exhibit distinct characteristics and reveal the potential for substantial benefits stemming from the judicious use of both aggregation processes. Our framework is evaluated on standard benchmarks for semantic matching, and also applied to geometric matching, where we show that our approach achieves significant improvements compared to existing methods.
  arXiv  Detail & Related papers  (2024-03-17T07:02:55Z)
• Generalized Correspondence Matching via Flexible Hierarchical Refinement and Patch Descriptor Distillation [13.802788788420175]
  Correspondence matching plays a crucial role in numerous robotics applications. This paper addresses the limitations of deep feature matching (DFM), a state-of-the-art (SoTA) plug-and-play correspondence matching approach. Our proposed method achieves an overall performance in terms of mean matching accuracy of 0.68, 0.92, and 0.95 with respect to the tolerances of 1, 3, and 5 pixels, respectively.
  arXiv  Detail & Related papers  (2024-03-08T15:32:18Z)
• Revisiting Sparse Retrieval for Few-shot Entity Linking [33.15662306409253]
  We propose an ELECTRA-based keyword extractor to denoise the mention context and construct a better query expression. For training the extractor, we propose a distant supervision method to automatically generate training data based on overlapping tokens between mention contexts and entity descriptions. Experimental results on the ZESHEL dataset demonstrate that the proposed method outperforms state-of-the-art models by a significant margin across all test domains.
  arXiv  Detail & Related papers  (2023-10-19T03:51:10Z)
• BERM: Training the Balanced and Extractable Representation for Matching to Improve Generalization Ability of Dense Retrieval [54.66399120084227]
  We propose a novel method to improve the generalization of dense retrieval via capturing matching signal called BERM. Dense retrieval has shown promise in the first-stage retrieval process when trained on in-domain labeled datasets.
  arXiv  Detail & Related papers  (2023-05-18T15:43:09Z)
• PIE: a Parameter and Inference Efficient Solution for Large Scale Knowledge Graph Embedding Reasoning [24.29409958504209]
  We propose PIE, a textbfparameter and textbfinference textbfefficient solution. Inspired from tensor decomposition methods, we find that decompose entity embedding matrix into low rank matrices can reduce more than half of the parameters. To accelerate model inference, we propose a self-supervised auxiliary task, which can be seen as fine-grained entity typing.
  arXiv  Detail & Related papers  (2022-04-29T09:06:56Z)
• Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detection [60.88952532574564]
  This paper conducts a thorough comparison of out-of-domain intent detection methods. We evaluate multiple contextual encoders and methods, proven to be efficient, on three standard datasets for intent classification. Our main findings show that fine-tuning Transformer-based encoders on in-domain data leads to superior results.
  arXiv  Detail & Related papers  (2021-01-11T09:10:58Z)
• Weakly-Supervised Aspect-Based Sentiment Analysis via Joint Aspect-Sentiment Topic Embedding [71.2260967797055]
  We propose a weakly-supervised approach for aspect-based sentiment analysis. We learn sentiment, aspect> joint topic embeddings in the word embedding space. We then use neural models to generalize the word-level discriminative information.
  arXiv  Detail & Related papers  (2020-10-13T21:33:24Z)
• Generation-Augmented Retrieval for Open-domain Question Answering [134.27768711201202]
  Generation-Augmented Retrieval (GAR) for answering open-domain questions. We show that generating diverse contexts for a query is beneficial as fusing their results consistently yields better retrieval accuracy. GAR achieves state-of-the-art performance on Natural Questions and TriviaQA datasets under the extractive QA setup when equipped with an extractive reader.
  arXiv  Detail & Related papers  (2020-09-17T23:08:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.