Context Quality Matters in Training Fusion-in-Decoder for Extractive Open-Domain Question Answering

• URL: http://arxiv.org/abs/2403.14197v1
• Date: Thu, 21 Mar 2024 07:47:57 GMT
• Title: Context Quality Matters in Training Fusion-in-Decoder for Extractive Open-Domain Question Answering
• Authors: Kosuke Akimoto, Kunihiro Takeoka, Masafumi Oyamada,
• Abstract summary: Retrieval-augmented generation models augment knowledge encoded in a language model by providing additional relevant external knowledge (context) during generation. This paper explores how context quantity and quality during model training affect the performance of Fusion-in-Decoder (FiD), the state-of-the-art retrieval-augmented generation model.
• Score: 5.259846811078731
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Retrieval-augmented generation models augment knowledge encoded in a language model by providing additional relevant external knowledge (context) during generation. Although it has been shown that the quantity and quality of context impact the performance of retrieval-augmented generation models during inference, limited research explores how these characteristics affect model training. This paper explores how context quantity and quality during model training affect the performance of Fusion-in-Decoder (FiD), the state-of-the-art retrieval-augmented generation model, in extractive open-domain question answering tasks. Experimental results suggest that FiD models overfit to context quality during training and show suboptimal performance when evaluated on different context quality. Through the experimental results, we also reveal FiD models trained with different context quality have different cross-attention distribution patterns. Specifically, as context quality during training increases, FiD models tend to attend more uniformly to each passage in context. Finally, based on these observations, we propose a method to mitigate overfitting to specific context quality by introducing bias to the cross-attention distribution, which we demonstrate to be effective in improving the performance of FiD models on different context quality.

Related papers

• Analyzing Persuasive Strategies in Meme Texts: A Fusion of Language Models with Paraphrase Enrichment [0.23020018305241333]
  This paper describes our approach to hierarchical multi-label detection of persuasion techniques in meme texts. The scope of the study encompasses enhancing model performance through innovative training techniques and data augmentation strategies.
  arXiv  Detail & Related papers  (2024-07-01T20:25:20Z)
• Multi-Modal Prompt Learning on Blind Image Quality Assessment [65.0676908930946]
  Image Quality Assessment (IQA) models benefit significantly from semantic information, which allows them to treat different types of objects distinctly. Traditional methods, hindered by a lack of sufficiently annotated data, have employed the CLIP image-text pretraining model as their backbone to gain semantic awareness. Recent approaches have attempted to address this mismatch using prompt technology, but these solutions have shortcomings. This paper introduces an innovative multi-modal prompt-based methodology for IQA.
  arXiv  Detail & Related papers  (2024-04-23T11:45:32Z)
• Exploring Precision and Recall to assess the quality and diversity of LLMs [82.21278402856079]
  We introduce a novel evaluation framework for Large Language Models (LLMs) such as textscLlama-2 and textscMistral. This approach allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora.
  arXiv  Detail & Related papers  (2024-02-16T13:53:26Z)
• A Cross-Attention Augmented Model for Event-Triggered Context-Aware Story Generation [28.046803293933213]
  We introduce a novel neural generation model, EtriCA, that enhances the relevance and coherence of generated stories. We employ a post-training framework for knowledge enhancement (KeEtriCA) on a large-scale book corpus. This results in approximately 5% improvement in automatic metrics and over 10% improvement in human evaluation.
  arXiv  Detail & Related papers  (2023-11-19T08:54:47Z)
• CONVIQT: Contrastive Video Quality Estimator [63.749184706461826]
  Perceptual video quality assessment (VQA) is an integral component of many streaming and video sharing platforms. Here we consider the problem of learning perceptually relevant video quality representations in a self-supervised manner. Our results indicate that compelling representations with perceptual bearing can be obtained using self-supervised learning.
  arXiv  Detail & Related papers  (2022-06-29T15:22:01Z)
• FUNQUE: Fusion of Unified Quality Evaluators [42.41484412777326]
  Fusion-based quality assessment has emerged as a powerful method for developing high-performance quality models. We propose FUNQUE, a quality model that fuses unified quality evaluators.
  arXiv  Detail & Related papers  (2022-02-23T00:21:43Z)
• Learning Transformer Features for Image Quality Assessment [53.51379676690971]
  We propose a unified IQA framework that utilizes CNN backbone and transformer encoder to extract features. The proposed framework is compatible with both FR and NR modes and allows for a joint training scheme.
  arXiv  Detail & Related papers  (2021-12-01T13:23:00Z)
• NoiER: An Approach for Training more Reliable Fine-TunedDownstream Task Models [54.184609286094044]
  We propose noise entropy regularisation (NoiER) as an efficient learning paradigm that solves the problem without auxiliary models and additional data. The proposed approach improved traditional OOD detection evaluation metrics by 55% on average compared to the original fine-tuned models.
  arXiv  Detail & Related papers  (2021-08-29T06:58:28Z)
• Training Speech Recognition Models with Federated Learning: A Quality/Cost Framework [4.125187280299247]
  We propose using federated learning, a decentralized on-device learning paradigm, to train speech recognition models. By performing epochs of training on a per-user basis, federated learning must incur the cost of dealing with non-IID data distributions.
  arXiv  Detail & Related papers  (2020-10-29T22:01:37Z)
• Improving the Reconstruction of Disentangled Representation Learners via Multi-Stage Modeling [54.94763543386523]
  Current autoencoder-based disentangled representation learning methods achieve disentanglement by penalizing the ( aggregate) posterior to encourage statistical independence of the latent factors. We present a novel multi-stage modeling approach where the disentangled factors are first learned using a penalty-based disentangled representation learning method. Then, the low-quality reconstruction is improved with another deep generative model that is trained to model the missing correlated latent variables.
  arXiv  Detail & Related papers  (2020-10-25T18:51:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.