Neural Text Generation with Part-of-Speech Guided Softmax

• URL: http://arxiv.org/abs/2105.03641v1
• Date: Sat, 8 May 2021 08:53:16 GMT
• Title: Neural Text Generation with Part-of-Speech Guided Softmax
• Authors: Zhixian Yang, Xiaojun Wan
• Abstract summary: We propose using linguistic annotation, i.e., part-of-speech (POS), to guide the text generation. We show that our proposed methods can generate more diverse text while maintaining comparable quality.
• Score: 82.63394952538292
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural text generation models are likely to suffer from the low-diversity problem. Various decoding strategies and training-based methods have been proposed to promote diversity only by exploiting contextual features, but rarely do they consider incorporating syntactic structure clues. In this work, we propose using linguistic annotation, i.e., part-of-speech (POS), to guide the text generation. In detail, we introduce POS Guided Softmax (POSG-Softmax) to explicitly model two posterior probabilities: (i) next-POS, and (ii) next-token from the vocabulary of the target POS. A POS guided sampling strategy is further proposed to address the low-diversity problem by enriching the diversity of POS. Extensive experiments and human evaluations demonstrate that, compared with existing state-of-the-art methods, our proposed methods can generate more diverse text while maintaining comparable quality.

Related papers

• Beyond Coarse-Grained Matching in Video-Text Retrieval [50.799697216533914]
  We introduce a new approach for fine-grained evaluation. Our approach can be applied to existing datasets by automatically generating hard negative test captions. Experiments on our fine-grained evaluations demonstrate that this approach enhances a model's ability to understand fine-grained differences.
  arXiv  Detail & Related papers  (2024-10-16T09:42:29Z)
• Detecting Machine-Generated Long-Form Content with Latent-Space Variables [54.07946647012579]
  Existing zero-shot detectors primarily focus on token-level distributions, which are vulnerable to real-world domain shifts. We propose a more robust method that incorporates abstract elements, such as event transitions, as key deciding factors to detect machine versus human texts.
  arXiv  Detail & Related papers  (2024-10-04T18:42:09Z)
• Beat: Bi-directional One-to-Many Embedding Alignment for Text-based Person Retrieval [66.61856014573742]
  Text-based person retrieval (TPR) is a challenging task that involves retrieving a specific individual based on a textual description. Previous methods have attempted to align text and image samples in a modal-shared space. We propose an effective bi-directional one-to-many embedding paradigm that offers a clear optimization direction for each sample.
  arXiv  Detail & Related papers  (2024-06-09T03:06:55Z)
• ToBlend: Token-Level Blending With an Ensemble of LLMs to Attack AI-Generated Text Detection [6.27025292177391]
  ToBlend is a novel token-level ensemble text generation method to challenge the robustness of current AI-content detection approaches. We find ToBlend significantly drops the performance of most mainstream AI-content detection methods.
  arXiv  Detail & Related papers  (2024-02-17T02:25:57Z)
• Detecting Textual Adversarial Examples Based on Distributional Characteristics of Data Representations [11.93653349589025]
  adversarial examples are constructed by adding small non-random perturbations to correctly classified inputs. Approaches to adversarial attacks in natural language tasks have boomed in the last five years using character-level, word-level, or phrase-level perturbations. We propose two new reactive methods for NLP to fill this gap. Adapted LID and MDRE obtain state-of-the-art results on character-level, word-level, and phrase-level attacks on the IMDB dataset.
  arXiv  Detail & Related papers  (2022-04-29T02:32:02Z)
• A Contrastive Framework for Neural Text Generation [46.845997620234265]
  We show that an underlying reason for model degeneration is the anisotropic distribution of token representations. We present a contrastive solution: (i) SimCTG, a contrastive training objective to calibrate the model's representation space, and (ii) a decoding method -- contrastive search -- to encourage diversity while maintaining coherence in the generated text.
  arXiv  Detail & Related papers  (2022-02-13T21:46:14Z)
• Few-Shot Text Generation with Pattern-Exploiting Training [12.919486518128734]
  In this paper, we show that the underlying idea can also be applied to text generation tasks. We adapt Pattern-Exploiting Training (PET), a recently proposed few-shot approach, for finetuning generative language models on text generation tasks.
  arXiv  Detail & Related papers  (2020-12-22T10:53:07Z)
• Intrinsic Probing through Dimension Selection [69.52439198455438]
  Most modern NLP systems make use of pre-trained contextual representations that attain astonishingly high performance on a variety of tasks. Such high performance should not be possible unless some form of linguistic structure inheres in these representations, and a wealth of research has sprung up on probing for it. In this paper, we draw a distinction between intrinsic probing, which examines how linguistic information is structured within a representation, and the extrinsic probing popular in prior work, which only argues for the presence of such information by showing that it can be successfully extracted.
  arXiv  Detail & Related papers  (2020-10-06T15:21:08Z)
• Informed Sampling for Diversity in Concept-to-Text NLG [8.883733362171034]
  We propose an Imitation Learning approach to explore the level of diversity that a language generation model can reliably produce. Specifically, we augment the decoding process with a meta-classifier trained to distinguish which words at any given timestep will lead to high-quality output.
  arXiv  Detail & Related papers  (2020-04-29T17:43:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.