Naturalness Evaluation of Natural Language Generation in Task-oriented Dialogues using BERT

• URL: http://arxiv.org/abs/2109.02938v1
• Date: Tue, 7 Sep 2021 08:40:14 GMT
• Title: Naturalness Evaluation of Natural Language Generation in Task-oriented Dialogues using BERT
• Authors: Ye Liu, Wolfgang Maier, Wolfgang Minker and Stefan Ultes
• Abstract summary: This paper presents an automatic method to evaluate the naturalness of natural language generation in dialogue systems. By fine-tuning the BERT model, our proposed naturalness evaluation method shows robust results and outperforms the baselines.
• Score: 6.1478669848771546
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents an automatic method to evaluate the naturalness of natural language generation in dialogue systems. While this task was previously rendered through expensive and time-consuming human labor, we present this novel task of automatic naturalness evaluation of generated language. By fine-tuning the BERT model, our proposed naturalness evaluation method shows robust results and outperforms the baselines: support vector machines, bi-directional LSTMs, and BLEURT. In addition, the training speed and evaluation performance of naturalness model are improved by transfer learning from quality and informativeness linguistic knowledge.

Related papers

• Injecting linguistic knowledge into BERT for Dialogue State Tracking [60.42231674887294]
  This paper proposes a method that extracts linguistic knowledge via an unsupervised framework. We then utilize this knowledge to augment BERT's performance and interpretability in Dialogue State Tracking (DST) tasks. We benchmark this framework on various DST tasks and observe a notable improvement in accuracy.
  arXiv  Detail & Related papers  (2023-11-27T08:38:42Z)
• Context-Tuning: Learning Contextualized Prompts for Natural Language Generation [52.835877179365525]
  We propose a novel continuous prompting approach, called Context-Tuning, to fine-tuning PLMs for natural language generation. Firstly, the prompts are derived based on the input text, so that they can elicit useful knowledge from PLMs for generation. Secondly, to further enhance the relevance of the generated text to the inputs, we utilize continuous inverse prompting to refine the process of natural language generation.
  arXiv  Detail & Related papers  (2022-01-21T12:35:28Z)
• NATURE: Natural Auxiliary Text Utterances for Realistic Spoken Language Evaluation [7.813460653362095]
  Slot-filling and intent detection are the backbone of conversational agents such as voice assistants, and are active areas of research. Even though state-of-the-art techniques on publicly available benchmarks show impressive performance, their ability to generalize to realistic scenarios is yet to be demonstrated. We present, a set of simple spoken-language oriented transformations, applied to the evaluation set of datasets to introduce human spoken language variations while preserving the semantics of an utterance.
  arXiv  Detail & Related papers  (2021-11-09T15:09:06Z)
• Evaluating the Morphosyntactic Well-formedness of Generated Texts [88.20502652494521]
  We propose L'AMBRE -- a metric to evaluate the morphosyntactic well-formedness of text. We show the effectiveness of our metric on the task of machine translation through a diachronic study of systems translating into morphologically-rich languages.
  arXiv  Detail & Related papers  (2021-03-30T18:02:58Z)
• TextFlint: Unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing [73.16475763422446]
  We propose a multilingual robustness evaluation platform for NLP tasks (TextFlint) It incorporates universal text transformation, task-specific transformation, adversarial attack, subpopulation, and their combinations to provide comprehensive robustness analysis. TextFlint generates complete analytical reports as well as targeted augmented data to address the shortcomings of the model's robustness.
  arXiv  Detail & Related papers  (2021-03-21T17:20:38Z)
• Self-Paced Learning for Neural Machine Translation [55.41314278859938]
  We propose self-paced learning for neural machine translation (NMT) training. We show that the proposed model yields better performance than strong baselines.
  arXiv  Detail & Related papers  (2020-10-09T11:33:16Z)
• Automating Text Naturalness Evaluation of NLG Systems [0.0]
  We present an attempt to automate the evaluation of text naturalness. Instead of relying on human participants for scoring or labeling the text samples, we propose to automate the process. We analyze the text probability fractions and observe how they are influenced by the size of the generative and discriminative models involved in the process.
  arXiv  Detail & Related papers  (2020-06-23T18:48:33Z)
• Exploring Software Naturalness through Neural Language Models [56.1315223210742]
  The Software Naturalness hypothesis argues that programming languages can be understood through the same techniques used in natural language processing. We explore this hypothesis through the use of a pre-trained transformer-based language model to perform code analysis tasks.
  arXiv  Detail & Related papers  (2020-06-22T21:56:14Z)
• A Hybrid Natural Language Generation System Integrating Rules and Deep Learning Algorithms [13.288402527470591]
  This paper proposes an enhanced natural language generation system combining the merits of both rule-based approaches and modern deep learning algorithms. We also come up with a novel approach called HMCU to measure the performance of the natural language processing comprehensively and precisely.
  arXiv  Detail & Related papers  (2020-06-15T00:50:41Z)
• Boosting Naturalness of Language in Task-oriented Dialogues via Adversarial Training [29.468502787886813]
  We propose to integrate adversarial training to produce more human-like responses. In the RNN-LG Restaurant dataset, our model AdvNLG outperforms the previous state-of-the-art result by 3.6% in BLEU.
  arXiv  Detail & Related papers  (2020-04-30T03:35:20Z)
• Stochastic Natural Language Generation Using Dependency Information [0.7995360025953929]
  This article presents a corpus-based model for generating natural language text. Our model encodes dependency relations from training data through a feature set, then produces a new dependency tree for a given meaning representation. We show that our model produces high-quality utterances in aspects of informativeness and naturalness as well as quality.
  arXiv  Detail & Related papers  (2020-01-12T09:40:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.