BERT4SO: Neural Sentence Ordering by Fine-tuning BERT

• URL: http://arxiv.org/abs/2103.13584v1
• Date: Thu, 25 Mar 2021 03:32:32 GMT
• Title: BERT4SO: Neural Sentence Ordering by Fine-tuning BERT
• Authors: Yutao Zhu, Jian-Yun Nie, Kun Zhou, Shengchao Liu, Pan Du
• Abstract summary: Recent work frames it as a ranking problem and applies deep neural networks to it. We propose a new method, named BERT4SO, by fine-tuning BERT for sentence ordering.
• Score: 26.050527288844005
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sentence ordering aims to arrange the sentences of a given text in the correct order. Recent work frames it as a ranking problem and applies deep neural networks to it. In this work, we propose a new method, named BERT4SO, by fine-tuning BERT for sentence ordering. We concatenate all sentences and compute their representations by using multiple special tokens and carefully designed segment (interval) embeddings. The tokens across multiple sentences can attend to each other which greatly enhances their interactions. We also propose a margin-based listwise ranking loss based on ListMLE to facilitate the optimization process. Experimental results on five benchmark datasets demonstrate the effectiveness of our proposed method.

Related papers

• Text Summarization with Oracle Expectation [88.39032981994535]
  Extractive summarization produces summaries by identifying and concatenating the most important sentences in a document. Most summarization datasets do not come with gold labels indicating whether document sentences are summary-worthy. We propose a simple yet effective labeling algorithm that creates soft, expectation-based sentence labels.
  arXiv  Detail & Related papers  (2022-09-26T14:10:08Z)
• Pyramid-BERT: Reducing Complexity via Successive Core-set based Token Selection [23.39962989492527]
  Transformer-based language models such as BERT have achieved the state-of-the-art on various NLP tasks, but are computationally prohibitive. We present Pyramid-BERT where we replace previously useds with a em core-set based token selection method justified by theoretical results. The core-set based token selection technique allows us to avoid expensive pre-training, gives a space-efficient fine tuning, and thus makes it suitable to handle longer sequence lengths.
  arXiv  Detail & Related papers  (2022-03-27T19:52:01Z)
• Pruned Graph Neural Network for Short Story Ordering [0.7087237546722617]
  Organizing sentences into an order that maximizes coherence is known as sentence ordering. We propose a new method for constructing sentence-entity graphs of short stories to create the edges between sentences. We also observe that replacing pronouns with their referring entities effectively encodes sentences in sentence-entity graphs.
  arXiv  Detail & Related papers  (2022-03-13T22:25:17Z)
• PromptBERT: Improving BERT Sentence Embeddings with Prompts [95.45347849834765]
  We propose a prompt based sentence embeddings method which can reduce token embeddings biases and make the original BERT layers more effective. We also propose a novel unsupervised training objective by the technology of template denoising, which substantially shortens the performance gap between the supervised and unsupervised setting. Our fine-tuned method outperforms the state-of-the-art method SimCSE in both unsupervised and supervised settings.
  arXiv  Detail & Related papers  (2022-01-12T06:54:21Z)
• Using BERT Encoding and Sentence-Level Language Model for Sentence Ordering [0.9134244356393667]
  We propose an algorithm for sentence ordering in a corpus of short stories. Our proposed method uses a language model based on Universal Transformers (UT) that captures sentences' dependencies by employing an attention mechanism. The proposed model includes three components: Sentence, Language Model, and Sentence Arrangement with Brute Force Search.
  arXiv  Detail & Related papers  (2021-08-24T23:03:36Z)
• Three Sentences Are All You Need: Local Path Enhanced Document Relation Extraction [54.95848026576076]
  We present an embarrassingly simple but effective method to select evidence sentences for document-level RE. We have released our code at https://github.com/AndrewZhe/Three-Sentences-Are-All-You-Need.
  arXiv  Detail & Related papers  (2021-06-03T12:29:40Z)
• Reformulating Sentence Ordering as Conditional Text Generation [17.91448517871621]
  We present Reorder-BART (RE-BART), a sentence ordering framework. We reformulate the task as a conditional text-to-marker generation setup. Our framework achieves the state-of-the-art performance across six datasets in Perfect Match Ratio (PMR) and Kendall's tau ($tau$) metric.
  arXiv  Detail & Related papers  (2021-04-14T18:16:47Z)
• Neural Sentence Ordering Based on Constraint Graphs [32.14555157902546]
  Sentence ordering aims at arranging a list of sentences in the correct order. We devise a new approach based on multi-granular orders between sentences. These orders form multiple constraint graphs, which are then encoded by Graph Isomorphism Networks and fused into sentence representations.
  arXiv  Detail & Related papers  (2021-01-27T02:53:10Z)
• Topological Sort for Sentence Ordering [133.05105352571715]
  We propose a new framing of this task as a constraint solving problem and introduce a new technique to solve it. The results on both automatic and human metrics across four different datasets show that this new technique is better at capturing coherence in documents.
  arXiv  Detail & Related papers  (2020-05-01T15:07:59Z)
• Incorporating BERT into Neural Machine Translation [251.54280200353674]
  We propose a new algorithm named BERT-fused model, in which we first use BERT to extract representations for an input sequence. We conduct experiments on supervised (including sentence-level and document-level translations), semi-supervised and unsupervised machine translation, and achieve state-of-the-art results on seven benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-17T08:13:36Z)
• Fact-aware Sentence Split and Rephrase with Permutation Invariant Training [93.66323661321113]
  Sentence Split and Rephrase aims to break down a complex sentence into several simple sentences with its meaning preserved. Previous studies tend to address the issue by seq2seq learning from parallel sentence pairs. We introduce Permutation Training to verifies the effects of order variance in seq2seq learning for this task.
  arXiv  Detail & Related papers  (2020-01-16T07:30:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.