Enhancing Pre-Trained Generative Language Models with Question Attended Span Extraction on Machine Reading Comprehension
- URL: http://arxiv.org/abs/2404.17991v2
- Date: Wed, 12 Jun 2024 04:47:01 GMT
- Title: Enhancing Pre-Trained Generative Language Models with Question Attended Span Extraction on Machine Reading Comprehension
- Authors: Lin Ai, Zheng Hui, Zizhou Liu, Julia Hirschberg,
- Abstract summary: Integrated during the fine-tuning phase of pre-trained generative language models (PLMs), QASE significantly enhances their performance.
The efficacy of the QASE module has been rigorously tested across various datasets.
- Score: 6.602323571343169
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Machine Reading Comprehension (MRC) poses a significant challenge in the field of Natural Language Processing (NLP). While mainstream MRC methods predominantly leverage extractive strategies using encoder-only models such as BERT, generative approaches face the issue of out-of-control generation -- a critical problem where answers generated are often incorrect, irrelevant, or unfaithful to the source text. To address these limitations in generative models for MRC, we introduce the Question-Attended Span Extraction (QASE) module. Integrated during the fine-tuning phase of pre-trained generative language models (PLMs), QASE significantly enhances their performance, allowing them to surpass the extractive capabilities of advanced Large Language Models (LLMs) such as GPT-4 in few-shot settings. Notably, these gains in performance do not come with an increase in computational demands. The efficacy of the QASE module has been rigorously tested across various datasets, consistently achieving or even surpassing state-of-the-art (SOTA) results, thereby bridging the gap between generative and extractive models in extractive MRC tasks.
Related papers
- QASE Enhanced PLMs: Improved Control in Text Generation for MRC [6.602323571343169]
We introduce the Question-Attended Span Extraction (QASE) module to address the challenges of out-of-control generation in generative models for machine reading comprehension.
integrated during the fine-tuning of pre-trained generative language models (PLMs), QASE enables these PLMs to match SOTA extractive methods.
arXiv Detail & Related papers (2024-02-26T05:34:16Z) - Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts
for Instruction Tuning on General Tasks [6.048370838631722]
We introduce.
-Efficient Sparsity Crafting (PESC), which transitions dense models to sparse models.
PESC integrates adapters into the MoE layers of sparse models, differentiating experts without altering individual weights within these layers.
Our sparse models, dubbed Camelidae, outperform all other opensource sparse models and exhibit superior general capabilities compared to GPT3.5.
arXiv Detail & Related papers (2024-01-05T09:58:09Z) - Simultaneous Machine Translation with Large Language Models [51.470478122113356]
We investigate the possibility of applying Large Language Models to SimulMT tasks.
We conducted experiments using the textttLlama2-7b-chat model on nine different languages from the MUST-C dataset.
The results show that LLM outperforms dedicated MT models in terms of BLEU and LAAL metrics.
arXiv Detail & Related papers (2023-09-13T04:06:47Z) - Instruction Position Matters in Sequence Generation with Large Language
Models [67.87516654892343]
Large language models (LLMs) are capable of performing conditional sequence generation tasks, such as translation or summarization.
We propose enhancing the instruction-following capability of LLMs by shifting the position of task instructions after the input sentences.
arXiv Detail & Related papers (2023-08-23T12:36:57Z) - Extrapolating Multilingual Understanding Models as Multilingual
Generators [82.1355802012414]
This paper explores methods to empower multilingual understanding models the generation abilities to get a unified model.
We propose a textbfSemantic-textbfGuided textbfAlignment-then-Denoising (SGA) approach to adapt an encoder to a multilingual generator with a small number of new parameters.
arXiv Detail & Related papers (2023-05-22T15:33:21Z) - Mixture of Soft Prompts for Controllable Data Generation [21.84489422361048]
Mixture of Soft Prompts (MSP) is proposed as a tool for data augmentation rather than direct prediction.
Our method achieves state-of-the-art results on three benchmarks when compared against strong baselines.
arXiv Detail & Related papers (2023-03-02T21:13:56Z) - Maximizing Efficiency of Language Model Pre-training for Learning
Representation [6.518508607788086]
ELECTRA is a novel approach for improving the compute efficiency of pre-trained language models.
Our work proposes adaptive early exit strategy to maximize the efficiency of the pre-training process.
arXiv Detail & Related papers (2021-10-13T10:25:06Z) - Unsupervised Paraphrasing with Pretrained Language Models [85.03373221588707]
We propose a training pipeline that enables pre-trained language models to generate high-quality paraphrases in an unsupervised setting.
Our recipe consists of task-adaptation, self-supervision, and a novel decoding algorithm named Dynamic Blocking.
We show with automatic and human evaluations that our approach achieves state-of-the-art performance on both the Quora Question Pair and the ParaNMT datasets.
arXiv Detail & Related papers (2020-10-24T11:55:28Z) - Multi-span Style Extraction for Generative Reading Comprehension [90.6069071495214]
We propose a new framework which enables generative MRC to be smoothly solved as multi-span extraction.
Thorough experiments demonstrate that this novel approach can alleviate the dilemma between generative models and single-span models.
arXiv Detail & Related papers (2020-09-15T23:06:48Z) - Improve Variational Autoencoder for Text Generationwith Discrete Latent
Bottleneck [52.08901549360262]
Variational autoencoders (VAEs) are essential tools in end-to-end representation learning.
VAEs tend to ignore latent variables with a strong auto-regressive decoder.
We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
arXiv Detail & Related papers (2020-04-22T14:41:37Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.