Substance over Style: Document-Level Targeted Content Transfer
- URL: http://arxiv.org/abs/2010.08618v1
- Date: Fri, 16 Oct 2020 20:26:10 GMT
- Title: Substance over Style: Document-Level Targeted Content Transfer
- Authors: Allison Hegel, Sudha Rao, Asli Celikyilmaz and Bill Dolan
- Abstract summary: We introduce the task of document-level targeted content transfer and address it in the recipe domain.
We propose a novel model for this task based on the generative pre-trained language model (GPT-2)
Both automatic and human evaluations show that our model out-performs existing methods.
- Score: 42.18770674148932
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Existing language models excel at writing from scratch, but many real-world
scenarios require rewriting an existing document to fit a set of constraints.
Although sentence-level rewriting has been fairly well-studied, little work has
addressed the challenge of rewriting an entire document coherently. In this
work, we introduce the task of document-level targeted content transfer and
address it in the recipe domain, with a recipe as the document and a dietary
restriction (such as vegan or dairy-free) as the targeted constraint. We
propose a novel model for this task based on the generative pre-trained
language model (GPT-2) and train on a large number of roughly-aligned recipe
pairs (https://github.com/microsoft/document-level-targeted-content-transfer).
Both automatic and human evaluations show that our model out-performs existing
methods by generating coherent and diverse rewrites that obey the constraint
while remaining close to the original document. Finally, we analyze our model's
rewrites to assess progress toward the goal of making language generation more
attuned to constraints that are substantive rather than stylistic.
Related papers
- Dr Genre: Reinforcement Learning from Decoupled LLM Feedback for Generic Text Rewriting [15.796381427671681]
We introduce a generic model proficient in factuality, stylistic, and conversational rewriting tasks.
To simulate real-world user rewrite requests, we construct a conversational rewrite dataset, ChatRewrite, that presents natural''-sounding instructions.
To align with task-specific objectives, we propose Dr Genre, a Decoupled-reward learning framework for Generic rewriting.
arXiv Detail & Related papers (2025-03-09T21:23:52Z) - Peek Across: Improving Multi-Document Modeling via Cross-Document
Question-Answering [49.85790367128085]
We pre-training a generic multi-document model from a novel cross-document question answering pre-training objective.
This novel multi-document QA formulation directs the model to better recover cross-text informational relations.
Unlike prior multi-document models that focus on either classification or summarization tasks, our pre-training objective formulation enables the model to perform tasks that involve both short text generation and long text generation.
arXiv Detail & Related papers (2023-05-24T17:48:40Z) - Towards Document-Level Paraphrase Generation with Sentence Rewriting and
Reordering [88.08581016329398]
We propose CoRPG (Coherence Relationship guided Paraphrase Generation) for document-level paraphrase generation.
We use graph GRU to encode the coherence relationship graph and get the coherence-aware representation for each sentence.
Our model can generate document paraphrase with more diversity and semantic preservation.
arXiv Detail & Related papers (2021-09-15T05:53:40Z) - Automatic Document Sketching: Generating Drafts from Analogous Texts [44.626645471195495]
We introduce a new task, document sketching, which involves generating entire draft documents for the writer to review and revise.
These drafts are built from sets of documents that overlap in form - sharing large segments of potentially reusable text - while diverging in content.
We investigate the application of weakly supervised methods, including use of a transformer-based mixture of experts, together with reinforcement learning.
arXiv Detail & Related papers (2021-06-14T06:46:06Z) - LAWDR: Language-Agnostic Weighted Document Representations from
Pre-trained Models [8.745407715423992]
Cross-lingual document representations enable language understanding in multilingual contexts.
Large pre-trained language models such as BERT, XLM and XLM-RoBERTa have achieved great success when fine-tuned on sentence-level downstream tasks.
arXiv Detail & Related papers (2021-06-07T07:14:00Z) - DRAG: Director-Generator Language Modelling Framework for Non-Parallel
Author Stylized Rewriting [9.275464023441227]
Author stylized rewriting is the task of rewriting an input text in a particular author's style.
We propose a Director-Generator framework to rewrite content in the target author's style.
arXiv Detail & Related papers (2021-01-28T06:52:40Z) - Robust Document Representations using Latent Topics and Metadata [17.306088038339336]
We propose a novel approach to fine-tuning a pre-trained neural language model for document classification problems.
We generate document representations that capture both text and metadata artifacts in a task manner.
Our solution also incorporates metadata explicitly rather than just augmenting them with text.
arXiv Detail & Related papers (2020-10-23T21:52:38Z) - Pre-training via Paraphrasing [96.79972492585112]
We introduce MARGE, a pre-trained sequence-to-sequence model learned with an unsupervised multi-lingual paraphrasing objective.
We show it is possible to jointly learn to do retrieval and reconstruction, given only a random initialization.
For example, with no additional task-specific training we achieve BLEU scores of up to 35.8 for document translation.
arXiv Detail & Related papers (2020-06-26T14:43:43Z) - Pre-training for Abstractive Document Summarization by Reinstating
Source Text [105.77348528847337]
This paper presents three pre-training objectives which allow us to pre-train a Seq2Seq based abstractive summarization model on unlabeled text.
Experiments on two benchmark summarization datasets show that all three objectives can improve performance upon baselines.
arXiv Detail & Related papers (2020-04-04T05:06:26Z) - Learning to Select Bi-Aspect Information for Document-Scale Text Content
Manipulation [50.01708049531156]
We focus on a new practical task, document-scale text content manipulation, which is the opposite of text style transfer.
In detail, the input is a set of structured records and a reference text for describing another recordset.
The output is a summary that accurately describes the partial content in the source recordset with the same writing style of the reference.
arXiv Detail & Related papers (2020-02-24T12:52:10Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.