A New Sentence Ordering Method Using BERT Pretrained Model
- URL: http://arxiv.org/abs/2108.11994v1
- Date: Thu, 26 Aug 2021 18:47:15 GMT
- Title: A New Sentence Ordering Method Using BERT Pretrained Model
- Authors: Melika Golestani, Seyedeh Zahra Razavi, and Heshaam Faili
- Abstract summary: We propose a method for sentence ordering which does not need a training phase and consequently a large corpus for learning.
Our proposed method outperformed other baselines on ROCStories, a corpus of 5-sentence human-made stories.
Among other advantages of this method are its interpretability and needlessness to linguistic knowledge.
- Score: 2.1793134762413433
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Building systems with capability of natural language understanding (NLU) has
been one of the oldest areas of AI. An essential component of NLU is to detect
logical succession of events contained in a text. The task of sentence ordering
is proposed to learn succession of events with applications in AI tasks. The
performance of previous works employing statistical methods is poor, while the
neural networks-based approaches are in serious need of large corpora for model
learning. In this paper, we propose a method for sentence ordering which does
not need a training phase and consequently a large corpus for learning. To this
end, we generate sentence embedding using BERT pre-trained model and measure
sentence similarity using cosine similarity score. We suggest this score as an
indicator of sequential events' level of coherence. We finally sort the
sentences through brute-force search to maximize overall similarities of the
sequenced sentences. Our proposed method outperformed other baselines on
ROCStories, a corpus of 5-sentence human-made stories. The method is
specifically more efficient than neural network-based methods when no huge
corpus is available. Among other advantages of this method are its
interpretability and needlessness to linguistic knowledge.
Related papers
- Efficient and Flexible Topic Modeling using Pretrained Embeddings and
Bag of Sentences [1.8592384822257952]
We propose a novel topic modeling and inference algorithm.
We leverage pre-trained sentence embeddings by combining generative process models and clustering.
TheTailor evaluation shows that our method yields state-of-the art results with relatively little computational demands.
arXiv Detail & Related papers (2023-02-06T20:13:11Z) - Ensemble Transfer Learning for Multilingual Coreference Resolution [60.409789753164944]
A problem that frequently occurs when working with a non-English language is the scarcity of annotated training data.
We design a simple but effective ensemble-based framework that combines various transfer learning techniques.
We also propose a low-cost TL method that bootstraps coreference resolution models by utilizing Wikipedia anchor texts.
arXiv Detail & Related papers (2023-01-22T18:22:55Z) - Reweighting Strategy based on Synthetic Data Identification for Sentence
Similarity [30.647497555295974]
We train a classifier that identifies machine-written sentences, and observe that the linguistic features of the sentences identified as written by a machine are significantly different from those of human-written sentences.
The distilled information from the classifier is then used to train a reliable sentence embedding model.
Our model trained on synthetic data generalizes well and outperforms the existing baselines.
arXiv Detail & Related papers (2022-08-29T05:42:22Z) - Using BERT Encoding and Sentence-Level Language Model for Sentence
Ordering [0.9134244356393667]
We propose an algorithm for sentence ordering in a corpus of short stories.
Our proposed method uses a language model based on Universal Transformers (UT) that captures sentences' dependencies by employing an attention mechanism.
The proposed model includes three components: Sentence, Language Model, and Sentence Arrangement with Brute Force Search.
arXiv Detail & Related papers (2021-08-24T23:03:36Z) - InsertGNN: Can Graph Neural Networks Outperform Humans in TOEFL Sentence
Insertion Problem? [66.70154236519186]
Sentence insertion is a delicate but fundamental NLP problem.
Current approaches in sentence ordering, text coherence, and question answering (QA) are neither suitable nor good at solving it.
We propose InsertGNN, a model that represents the problem as a graph and adopts the graph Neural Network (GNN) to learn the connection between sentences.
arXiv Detail & Related papers (2021-03-28T06:50:31Z) - Learning a Word-Level Language Model with Sentence-Level Noise
Contrastive Estimation for Contextual Sentence Probability Estimation [3.1040192682787415]
Inferring the probability distribution of sentences or word sequences is a key process in natural language processing.
While word-level language models (LMs) have been widely adopted for computing the joint probabilities of word sequences, they have difficulty capturing a context long enough for sentence probability estimation (SPE)
Recent studies introduced training methods using sentence-level noise-contrastive estimation (NCE) with recurrent neural networks (RNNs)
We apply our method to a simple word-level RNN LM to focus on the effect of the sentence-level NCE training rather than on the network architecture.
arXiv Detail & Related papers (2021-03-14T09:17:37Z) - Narrative Incoherence Detection [76.43894977558811]
We propose the task of narrative incoherence detection as a new arena for inter-sentential semantic understanding.
Given a multi-sentence narrative, decide whether there exist any semantic discrepancies in the narrative flow.
arXiv Detail & Related papers (2020-12-21T07:18:08Z) - MC-BERT: Efficient Language Pre-Training via a Meta Controller [96.68140474547602]
Large-scale pre-training is computationally expensive.
ELECTRA, an early attempt to accelerate pre-training, trains a discriminative model that predicts whether each input token was replaced by a generator.
We propose a novel meta-learning framework, MC-BERT, to achieve better efficiency and effectiveness.
arXiv Detail & Related papers (2020-06-10T09:22:19Z) - Toward Better Storylines with Sentence-Level Language Models [54.91921545103256]
We propose a sentence-level language model which selects the next sentence in a story from a finite set of fluent alternatives.
We demonstrate the effectiveness of our approach with state-of-the-art accuracy on the unsupervised Story Cloze task.
arXiv Detail & Related papers (2020-05-11T16:54:19Z) - Progressively Pretrained Dense Corpus Index for Open-Domain Question
Answering [87.32442219333046]
We propose a simple and resource-efficient method to pretrain the paragraph encoder.
Our method outperforms an existing dense retrieval method that uses 7 times more computational resources for pretraining.
arXiv Detail & Related papers (2020-04-30T18:09:50Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.