TegFormer: Topic-to-Essay Generation with Good Topic Coverage and High Text Coherence

• URL: http://arxiv.org/abs/2212.13456v1
• Date: Tue, 27 Dec 2022 11:50:14 GMT
• Title: TegFormer: Topic-to-Essay Generation with Good Topic Coverage and High Text Coherence
• Authors: Wang Qi and Rui Liu and Yuan Zuo and Yong Chen and Dell Zhang
• Abstract summary: We propose a novel approach to topic-to-essay generation called TegFormer. A emphTopic-Extension layer captures the interaction between the given topics and their domain-specific contexts. An emphEmbedding-Fusion module combines the domain-specific word embeddings learnt from the given corpus and the general-purpose word embeddings provided by a GPT-2 model pre-trained on massive text data.
• Score: 8.422108048684215
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Creating an essay based on a few given topics is a challenging NLP task. Although several effective methods for this problem, topic-to-essay generation, have appeared recently, there is still much room for improvement, especially in terms of the coverage of the given topics and the coherence of the generated text. In this paper, we propose a novel approach called TegFormer which utilizes the Transformer architecture where the encoder is enriched with domain-specific contexts while the decoder is enhanced by a large-scale pre-trained language model. Specifically, a \emph{Topic-Extension} layer capturing the interaction between the given topics and their domain-specific contexts is plugged into the encoder. Since the given topics are usually concise and sparse, such an additional layer can bring more topic-related semantics in to facilitate the subsequent natural language generation. Moreover, an \emph{Embedding-Fusion} module that combines the domain-specific word embeddings learnt from the given corpus and the general-purpose word embeddings provided by a GPT-2 model pre-trained on massive text data is integrated into the decoder. Since GPT-2 is at a much larger scale, it contains a lot more implicit linguistic knowledge which would help the decoder to produce more grammatical and readable text. Extensive experiments have shown that the pieces of text generated by TegFormer have better topic coverage and higher text coherence than those from SOTA topic-to-essay techniques, according to automatic and human evaluations. As revealed by ablation studies, both the Topic-Extension layer and the Embedding-Fusion module contribute substantially to TegFormer's performance advantage.

Related papers

• TopicGPT: A Prompt-based Topic Modeling Framework [77.72072691307811]
  We introduce TopicGPT, a prompt-based framework that uses large language models to uncover latent topics in a text collection. It produces topics that align better with human categorizations compared to competing methods. Its topics are also interpretable, dispensing with ambiguous bags of words in favor of topics with natural language labels and associated free-form descriptions.
  arXiv  Detail & Related papers  (2023-11-02T17:57:10Z)
• Story Visualization by Online Text Augmentation with Context Memory [64.86944645907771]
  We propose a novel memory architecture for the Bi-directional Transformer framework with an online text augmentation. The proposed method significantly outperforms the state of the arts in various metrics including FID, character F1, frame accuracy, BLEU-2/3, and R-precision.
  arXiv  Detail & Related papers  (2023-08-15T05:08:12Z)
• TextFormer: A Query-based End-to-End Text Spotter with Mixed Supervision [61.186488081379]
  We propose TextFormer, a query-based end-to-end text spotter with Transformer architecture. TextFormer builds upon an image encoder and a text decoder to learn a joint semantic understanding for multi-task modeling. It allows for mutual training and optimization of classification, segmentation, and recognition branches, resulting in deeper feature sharing.
  arXiv  Detail & Related papers  (2023-06-06T03:37:41Z)
• Uncovering the Potential of ChatGPT for Discourse Analysis in Dialogue: An Empirical Study [51.079100495163736]
  This paper systematically inspects ChatGPT's performance in two discourse analysis tasks: topic segmentation and discourse parsing. ChatGPT demonstrates proficiency in identifying topic structures in general-domain conversations yet struggles considerably in specific-domain conversations. Our deeper investigation indicates that ChatGPT can give more reasonable topic structures than human annotations but only linearly parses the hierarchical rhetorical structures.
  arXiv  Detail & Related papers  (2023-05-15T07:14:41Z)
• MGDoc: Pre-training with Multi-granular Hierarchy for Document Image Understanding [53.03978356918377]
  spatial hierarchical relationships between content at different levels of granularity are crucial for document image understanding tasks. Existing methods learn features from either word-level or region-level but fail to consider both simultaneously. We propose MGDoc, a new multi-modal multi-granular pre-training framework that encodes page-level, region-level, and word-level information at the same time.
  arXiv  Detail & Related papers  (2022-11-27T22:47:37Z)
• A Benchmark Corpus for the Detection of Automatically Generated Text in Academic Publications [0.02578242050187029]
  This paper presents two datasets comprised of artificially generated research content. In the first case, the content is completely generated by the GPT-2 model after a short prompt extracted from original papers. The partial or hybrid dataset is created by replacing several sentences of abstracts with sentences that are generated by the Arxiv-NLP model. We evaluate the quality of the datasets comparing the generated texts to aligned original texts using fluency metrics such as BLEU and ROUGE.
  arXiv  Detail & Related papers  (2022-02-04T08:16:56Z)
• Topical Change Detection in Documents via Embeddings of Long Sequences [4.13878392637062]
  We formulate the task of text segmentation as an independent supervised prediction task. By fine-tuning on paragraphs of similar sections, we are able to show that learned features encode topic information. Unlike previous approaches, which mostly operate on sentence-level, we consistently use a broader context.
  arXiv  Detail & Related papers  (2020-12-07T12:09:37Z)
• Robust Conversational AI with Grounded Text Generation [77.56950706340767]
  GTG is a hybrid model which uses a large-scale Transformer neural network as its backbone. It generates responses grounded in dialog belief state and real-world knowledge for task completion.
  arXiv  Detail & Related papers  (2020-09-07T23:49:28Z)
• Efficient text generation of user-defined topic using generative adversarial networks [0.32228025627337864]
  We propose a User-Defined GAN (UD-GAN) with two-level discriminators to solve this problem. The proposed method is capable of generating texts with less time than others.
  arXiv  Detail & Related papers  (2020-06-22T04:49:47Z)
• Diversifying Dialogue Generation with Non-Conversational Text [38.03510529185192]
  We propose a new perspective to diversify dialogue generation by leveraging non-conversational text. We collect a large-scale non-conversational corpus from multi sources including forum comments, idioms and book snippets. The resulting model is tested on two conversational datasets and is shown to produce significantly more diverse responses without sacrificing the relevance with context.
  arXiv  Detail & Related papers  (2020-05-09T02:16:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.