Generating Repetitions with Appropriate Repeated Words

• URL: http://arxiv.org/abs/2207.00929v1
• Date: Sun, 3 Jul 2022 01:21:49 GMT
• Title: Generating Repetitions with Appropriate Repeated Words
• Authors: Toshiki Kawamoto, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura
• Abstract summary: Repetitions are essential in communication to build trust with others. To the best of our knowledge, this is the first neural approach to address repetition generation. We propose Weighted Label Smoothing, a smoothing method for explicitly learning which words to repeat during fine-tuning, and a repetition scoring method that can output more appropriate repetitions during decoding.
• Score: 30.10429353715689
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A repetition is a response that repeats words in the previous speaker's utterance in a dialogue. Repetitions are essential in communication to build trust with others, as investigated in linguistic studies. In this work, we focus on repetition generation. To the best of our knowledge, this is the first neural approach to address repetition generation. We propose Weighted Label Smoothing, a smoothing method for explicitly learning which words to repeat during fine-tuning, and a repetition scoring method that can output more appropriate repetitions during decoding. We conducted automatic and human evaluations involving applying these methods to the pre-trained language model T5 for generating repetitions. The experimental results indicate that our methods outperformed baselines in both evaluations.

Related papers

• Repetition Neurons: How Do Language Models Produce Repetitions? [25.430820735194768]
  This paper introduces repetition neurons, regarded as skill neurons responsible for the repetition problem in text generation tasks. We identify these repetition neurons by comparing activation values before and after the onset of repetition in texts generated by recent pre-trained language models.
  arXiv  Detail & Related papers  (2024-10-17T12:43:47Z)
• Continuously Learning New Words in Automatic Speech Recognition [56.972851337263755]
  We propose an self-supervised continual learning approach to recognize new words. We use a memory-enhanced Automatic Speech Recognition model from previous work. We show that with this approach, we obtain increasing performance on the new words when they occur more frequently.
  arXiv  Detail & Related papers  (2024-01-09T10:39:17Z)
• Lexical Repetitions Lead to Rote Learning: Unveiling the Impact of Lexical Overlap in Train and Test Reference Summaries [131.80860903537172]
  Ideal summarization models should generalize to novel summary-worthy content without remembering reference training summaries by rote. We propose a fine-grained evaluation protocol by partitioning a test set based on the lexical similarity of reference test summaries with training summaries.
  arXiv  Detail & Related papers  (2023-11-15T23:47:53Z)
• Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective [91.14291142262262]
  This work presents a straightforward and fundamental explanation from the data perspective. Our preliminary investigation reveals a strong correlation between the degeneration issue and the presence of repetitions in training data. Our experiments reveal that penalizing the repetitions in training data remains critical even when considering larger model sizes and instruction tuning.
  arXiv  Detail & Related papers  (2023-10-16T09:35:42Z)
• Mitigating the Learning Bias towards Repetition by Self-Contrastive Training for Open-Ended Generation [92.42032403795879]
  We show that pretrained language models (LMs) such as GPT2 still tend to generate repetitive texts. We attribute their overestimation of token-level repetition probabilities to the learning bias. We find that LMs use longer-range dependencies to predict repetitive tokens than non-repetitive ones, which may be the cause of sentence-level repetition loops.
  arXiv  Detail & Related papers  (2023-07-04T07:53:55Z)
• Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation [41.3948101212288]
  We study the relationship between the probabilities of the repetitive tokens and their previous repetitions in the context. We propose a training method where the model learns to penalize probabilities of sentence-level repetitions from pseudo repetitive data.
  arXiv  Detail & Related papers  (2022-06-06T05:51:12Z)
• Towards Robust Online Dialogue Response Generation [62.99904593650087]
  We argue that this can be caused by a discrepancy between training and real-world testing. We propose a hierarchical sampling-based method consisting of both utterance-level sampling and semi-utterance-level sampling.
  arXiv  Detail & Related papers  (2022-03-07T06:51:41Z)
• Taming Repetition in Dialogue Generation [1.851321027703742]
  Inappropriate repetition of words can significantly degrade the quality of the generated texts. We design a context-aware classifier to explicitly decide when to allow repetition and when to employ penalized sampling. Our method can generate higher quality and more authentic dialogues.
  arXiv  Detail & Related papers  (2021-12-16T06:25:46Z)
• Refine and Imitate: Reducing Repetition and Inconsistency in Persuasion Dialogues via Reinforcement Learning and Human Demonstration [45.14559188965439]
  We propose to apply reinforcement learning to refine an MLE-based language model without user simulators. We distill sentence-level information about repetition, inconsistency and task relevance through rewards. Experiments show that our model outperforms previous state-of-the-art dialogue models on both automatic metrics and human evaluation results.
  arXiv  Detail & Related papers  (2020-12-31T00:02:51Z)
• PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation [92.7366819044397]
  Self-supervised pre-training has emerged as a powerful technique for natural language understanding and generation. This work presents PALM with a novel scheme that jointly pre-trains an autoencoding and autoregressive language model on a large unlabeled corpus. An extensive set of experiments show that PALM achieves new state-of-the-art results on a variety of language generation benchmarks.
  arXiv  Detail & Related papers  (2020-04-14T06:25:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.