Copy that! Editing Sequences by Copying Spans

• URL: http://arxiv.org/abs/2006.04771v2
• Date: Mon, 14 Dec 2020 10:03:21 GMT
• Title: Copy that! Editing Sequences by Copying Spans
• Authors: Sheena Panthaplackel, Miltiadis Allamanis, Marc Brockschmidt
• Abstract summary: We present an extension of seq2seq models capable of copying entire spans of the input to the output in one step. In experiments on a range of editing tasks of natural language and source code, we show that our new model consistently outperforms simpler baselines.
• Score: 40.23377412674599
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural sequence-to-sequence models are finding increasing use in editing of documents, for example in correcting a text document or repairing source code. In this paper, we argue that common seq2seq models (with a facility to copy single tokens) are not a natural fit for such tasks, as they have to explicitly copy each unchanged token. We present an extension of seq2seq models capable of copying entire spans of the input to the output in one step, greatly reducing the number of decisions required during inference. This extension means that there are now many ways of generating the same output, which we handle by deriving a new objective for training and a variation of beam search for inference that explicitly handles this problem. In our experiments on a range of editing tasks of natural language and source code, we show that our new model consistently outperforms simpler baselines.

Related papers

• Seq2seq is All You Need for Coreference Resolution [26.551602768015986]
  We finetune a pretrained seq2seq transformer to map an input document to a tagged sequence encoding the coreference annotation. Our model outperforms or closely matches the best coreference systems in the literature on an array of datasets.
  arXiv  Detail & Related papers  (2023-10-20T19:17:22Z)
• Copy Is All You Need [66.00852205068327]
  We formulate text generation as progressively copying text segments from an existing text collection. Our approach achieves better generation quality according to both automatic and human evaluations. Our approach attains additional performance gains by simply scaling up to larger text collections.
  arXiv  Detail & Related papers  (2023-07-13T05:03:26Z)
• Hierarchical Phrase-based Sequence-to-Sequence Learning [94.10257313923478]
  We describe a neural transducer that maintains the flexibility of standard sequence-to-sequence (seq2seq) models while incorporating hierarchical phrases as a source of inductive bias during training and as explicit constraints during inference. Our approach trains two models: a discriminative derivation based on a bracketing grammar whose tree hierarchically aligns source and target phrases, and a neural seq2seq model that learns to translate the aligned phrases one-by-one.
  arXiv  Detail & Related papers  (2022-11-15T05:22:40Z)
• Thutmose Tagger: Single-pass neural model for Inverse Text Normalization [76.87664008338317]
  Inverse text normalization (ITN) is an essential post-processing step in automatic speech recognition. We present a dataset preparation method based on the granular alignment of ITN examples. One-to-one correspondence between tags and input words improves the interpretability of the model's predictions.
  arXiv  Detail & Related papers  (2022-07-29T20:39:02Z)
• Text Generation with Text-Editing Models [78.03750739936956]
  This tutorial provides a comprehensive overview of text-editing models and current state-of-the-art approaches. We discuss challenges related to productionization and how these models can be used to mitigate hallucination and bias.
  arXiv  Detail & Related papers  (2022-06-14T17:58:17Z)
• May the Force Be with Your Copy Mechanism: Enhanced Supervised-Copy Method for Natural Language Generation [1.2453219864236247]
  We propose a novel supervised approach of a copy network that helps the model decide which words need to be copied and which need to be generated. Specifically, we re-define the objective function, which leverages source sequences and target vocabularies as guidance for copying. The experimental results on data-to-text generation and abstractive summarization tasks verify that our approach enhances the copying quality and improves the degree of abstractness.
  arXiv  Detail & Related papers  (2021-12-20T06:54:28Z)
• Unsupervised Learning of General-Purpose Embeddings for Code Changes [6.652641137999891]
  We propose an approach for obtaining embeddings of code changes during pre-training. We evaluate them on two different downstream tasks - applying changes to code and commit message generation. Our model outperforms the model that uses full edit sequences by 5.9 percentage points in accuracy.
  arXiv  Detail & Related papers  (2021-06-03T19:08:53Z)
• CopyNext: Explicit Span Copying and Alignment in Sequence to Sequence Models [31.832217465573503]
  We present a model with an explicit token-level copy operation and extend it to copying entire spans. Our model provides hard alignments between spans in the input and output, allowing for nontraditional applications of seq2seq, like information extraction.
  arXiv  Detail & Related papers  (2020-10-28T22:45:16Z)
• Pre-training for Abstractive Document Summarization by Reinstating Source Text [105.77348528847337]
  This paper presents three pre-training objectives which allow us to pre-train a Seq2Seq based abstractive summarization model on unlabeled text. Experiments on two benchmark summarization datasets show that all three objectives can improve performance upon baselines.
  arXiv  Detail & Related papers  (2020-04-04T05:06:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.