Lacking the embedding of a word? Look it up into a traditional dictionary

• URL: http://arxiv.org/abs/2109.11763v1
• Date: Fri, 24 Sep 2021 06:27:58 GMT
• Title: Lacking the embedding of a word? Look it up into a traditional dictionary
• Authors: Elena Sofia Ruzzetti, Leonardo Ranaldi, Michele Mastromattei, Francesca Fallucchi, Fabio Massimo Zanzotto
• Abstract summary: We propose to use definitions retrieved in traditional dictionaries to produce word embeddings for rare words. DefiNNet and DefBERT significantly outperform state-of-the-art as well as baseline methods for producing embeddings of unknown words.
• Score: 0.2624902795082451
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Word embeddings are powerful dictionaries, which may easily capture language variations. However, these dictionaries fail to give sense to rare words, which are surprisingly often covered by traditional dictionaries. In this paper, we propose to use definitions retrieved in traditional dictionaries to produce word embeddings for rare words. For this purpose, we introduce two methods: Definition Neural Network (DefiNNet) and Define BERT (DefBERT). In our experiments, DefiNNet and DefBERT significantly outperform state-of-the-art as well as baseline methods devised for producing embeddings of unknown words. In fact, DefiNNet significantly outperforms FastText, which implements a method for the same task-based on n-grams, and DefBERT significantly outperforms the BERT method for OOV words. Then, definitions in traditional dictionaries are useful to build word embeddings for rare words.

Related papers

• LitMind Dictionary: An Open-Source Online Dictionary [5.2221935174520056]
  We introduce the LitMind Dictionary, an open-source online generative dictionary. It takes a word and context containing the word as input and automatically generates a definition as output. It supports not only Chinese and English, but also Chinese-English cross-lingual queries.
  arXiv  Detail & Related papers  (2022-04-23T15:10:40Z)
• Pretraining without Wordpieces: Learning Over a Vocabulary of Millions of Words [50.11559460111882]
  We explore the possibility of developing BERT-style pretrained model over a vocabulary of words instead of wordpieces. Results show that, compared to standard wordpiece-based BERT, WordBERT makes significant improvements on cloze test and machine reading comprehension. Since the pipeline is language-independent, we train WordBERT for Chinese language and obtain significant gains on five natural language understanding datasets.
  arXiv  Detail & Related papers  (2022-02-24T15:15:48Z)
• Dict-BERT: Enhancing Language Model Pre-training with Dictionary [42.0998323292348]
  Pre-trained language models (PLMs) aim to learn universal language representations by conducting self-supervised training tasks on large-scale corpora. In this work, we focus on enhancing language model pre-training by leveraging definitions of rare words in dictionaries. We propose two novel self-supervised pre-training tasks on word and sentence-level alignment between input text sequence and rare word definitions.
  arXiv  Detail & Related papers  (2021-10-13T04:29:14Z)
• Deconstructing word embedding algorithms [17.797952730495453]
  We propose a retrospective on some of the most well-known word embedding algorithms. In this work, we deconstruct Word2vec, GloVe, and others, into a common form, unveiling some of the common conditions that seem to be required for making performant word embeddings.
  arXiv  Detail & Related papers  (2020-11-12T14:23:35Z)
• Look It Up: Bilingual Dictionaries Improve Neural Machine Translation [17.385945558427863]
  We describe a new method for "attaching" dictionary definitions to rare words so that the network can learn the best way to use them. We demonstrate improvements of up to 1.8 BLEU using bilingual dictionaries.
  arXiv  Detail & Related papers  (2020-10-12T19:53:08Z)
• BERT for Monolingual and Cross-Lingual Reverse Dictionary [56.8627517256663]
  We propose a simple but effective method to make BERT generate the target word for this specific task. By using the BERT (mBERT), we can efficiently conduct the cross-lingual reverse dictionary with one subword embedding.
  arXiv  Detail & Related papers  (2020-09-30T17:00:10Z)
• Taking Notes on the Fly Helps BERT Pre-training [94.43953312613577]
  Taking Notes on the Fly (TNF) takes notes for rare words on the fly during pre-training to help the model understand them when they occur next time. TNF provides better data utilization since cross-sentence information is employed to cover the inadequate semantics caused by rare words in the sentences.
  arXiv  Detail & Related papers  (2020-08-04T11:25:09Z)
• When Dictionary Learning Meets Deep Learning: Deep Dictionary Learning and Coding Network for Image Recognition with Limited Data [74.75557280245643]
  We present a new Deep Dictionary Learning and Coding Network (DDLCN) for image recognition tasks with limited data. We empirically compare DDLCN with several leading dictionary learning methods and deep learning models. Experimental results on five popular datasets show that DDLCN achieves competitive results compared with state-of-the-art methods when the training data is limited.
  arXiv  Detail & Related papers  (2020-05-21T23:12:10Z)
• Word Sense Disambiguation for 158 Languages using Word Embeddings Only [80.79437083582643]
  Disambiguation of word senses in context is easy for humans, but a major challenge for automatic approaches. We present a method that takes as input a standard pre-trained word embedding model and induces a fully-fledged word sense inventory. We use this method to induce a collection of sense inventories for 158 languages on the basis of the original pre-trained fastText word embeddings.
  arXiv  Detail & Related papers  (2020-03-14T14:50:04Z)
• Lexical Sememe Prediction using Dictionary Definitions by Capturing Local Semantic Correspondence [94.79912471702782]
  Sememes, defined as the minimum semantic units of human languages, have been proven useful in many NLP tasks. We propose a Sememe Correspondence Pooling (SCorP) model, which is able to capture this kind of matching to predict sememes. We evaluate our model and baseline methods on a famous sememe KB HowNet and find that our model achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2020-01-16T17:30:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.