Related papers: Assessing Discourse Relations in Language Generation from GPT-2

Assessing Discourse Relations in Language Generation from GPT-2

URL: http://arxiv.org/abs/2004.12506v3
Date: Sat, 31 Oct 2020 05:53:21 GMT
Title: Assessing Discourse Relations in Language Generation from GPT-2
Authors: Wei-Jen Ko, Junyi Jessy Li
Abstract summary: GPT-2 is suited for generation tasks given its left-to-right language modeling objective. We study the validity of explicit discourse relations in GPT-2's outputs under both organic generation and fine-tuned scenarios.
Score: 37.30382375828105
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: Recent advances in NLP have been attributed to the emergence of large-scale pre-trained language models. GPT-2, in particular, is suited for generation tasks given its left-to-right language modeling objective, yet the linguistic quality of its generated text has largely remain unexplored. Our work takes a step in understanding GPT-2's outputs in terms of discourse coherence. We perform a comprehensive study on the validity of explicit discourse relations in GPT-2's outputs under both organic generation and fine-tuned scenarios. Results show GPT-2 does not always generate text containing valid discourse relations; nevertheless, its text is more aligned with human expectation in the fine-tuned scenario. We propose a decoupled strategy to mitigate these problems and highlight the importance of explicitly modeling discourse information.

Related papers

Wav2Gloss: Generating Interlinear Glossed Text from Speech [78.64412090339044]
We propose Wav2Gloss, a task in which four linguistic annotation components are extracted automatically from speech. We provide various baselines to lay the groundwork for future research on Interlinear Glossed Text generation from speech.
arXiv Detail & Related papers (2024-03-19T21:45:29Z)
SOUL: Towards Sentiment and Opinion Understanding of Language [96.74878032417054]
We propose a new task called Sentiment and Opinion Understanding of Language (SOUL) SOUL aims to evaluate sentiment understanding through two subtasks: Review (RC) and Justification Generation (JG)
arXiv Detail & Related papers (2023-10-27T06:48:48Z)
Improving Mandarin Prosodic Structure Prediction with Multi-level Contextual Information [68.89000132126536]
This work proposes to use inter-utterance linguistic information to improve the performance of prosodic structure prediction (PSP) Our method achieves better F1 scores in predicting prosodic word (PW), prosodic phrase (PPH) and intonational phrase (IPH)
arXiv Detail & Related papers (2023-08-31T09:19:15Z)
Adapting GPT, GPT-2 and BERT Language Models for Speech Recognition [14.82259273703819]
We present results using fine-tuned GPT, GPT-2 and their combination for automatic speech recognition (ASR) A conversion method is proposed to compute the correct language prior probability based on bidirectional LM outputs. The proposed conversion for language prior probabilities enables BERT to receive an extra 3% relative WERR.
arXiv Detail & Related papers (2021-07-29T16:53:37Z)
As good as new. How to successfully recycle English GPT-2 to make models for other languages [3.6042575355093907]
We describe the adaptation of English GPT-2 to Italian and Dutch by retraining lexical embeddings without tuning the Transformer layers. We show how to scale up complexity by transforming relearned lexical embeddings of GPT-2 small to the GPT-2 medium embedding space. English GPT-2 models with relearned lexical embeddings can generate realistic sentences in Italian and Dutch, but on average these sentences are still identifiable as artificial by humans.
arXiv Detail & Related papers (2020-12-10T12:27:16Z)
Infusing Finetuning with Semantic Dependencies [62.37697048781823]
We show that, unlike syntax, semantics is not brought to the surface by today's pretrained models. We then use convolutional graph encoders to explicitly incorporate semantic parses into task-specific finetuning.
arXiv Detail & Related papers (2020-12-10T01:27:24Z)
Topic-Preserving Synthetic News Generation: An Adversarial Deep Reinforcement Learning Approach [40.254715367640635]
GPT-2 can generate readable text and can be fine-tuned to generate text for a specific domain. In this paper, we study the novel problem of topic-preserving synthetic news generation. We propose a novel deep reinforcement learning-based method to control the output of GPT-2 with respect to a given news topic.
arXiv Detail & Related papers (2020-10-30T15:29:16Z)
Video-Grounded Dialogues with Pretrained Generation Language Models [88.15419265622748]
We leverage the power of pre-trained language models for improving video-grounded dialogue. We propose a framework by formulating sequence-to-grounded dialogue tasks as a sequence-to-grounded task. Our framework allows fine-tuning language models to capture dependencies across multiple modalities.
arXiv Detail & Related papers (2020-06-27T08:24:26Z)
Labeling Explicit Discourse Relations using Pre-trained Language Models [0.0]
State-of-the-art models achieve slightly above 45% of F-score by using hand-crafted features. We find that the pre-trained language models, when finetuned, are powerful enough to replace the linguistic features. This is the first time when a model outperforms the knowledge intensive models without employing any linguistic features.
arXiv Detail & Related papers (2020-06-21T17:18:01Z)
Unsupervised Paraphrase Generation using Pre-trained Language Models [0.0]
OpenAI's GPT-2 is notable for its capability to generate fluent, well formulated, grammatically consistent text. We leverage this generation capability of GPT-2 to generate paraphrases without any supervision from labelled data. Our experiments show that paraphrases generated with our model are of good quality, are diverse and improves the downstream task performance when used for data augmentation.
arXiv Detail & Related papers (2020-06-09T19:40:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.