A Comprehensive Survey on Word Representation Models: From Classical to State-Of-The-Art Word Representation Language Models

• URL: http://arxiv.org/abs/2010.15036v1
• Date: Wed, 28 Oct 2020 15:15:13 GMT
• Title: A Comprehensive Survey on Word Representation Models: From Classical to State-Of-The-Art Word Representation Language Models
• Authors: Usman Naseem, Imran Razzak, Shah Khalid Khan, Mukesh Prasad
• Abstract summary: Survey explores different word representation models and its power of expression. We describe a variety of text representation methods, and model designs have blossomed in the context of NLP. We briefly discuss the commonly used ML and DL based classifiers, evaluation metrics and the applications of these word embeddings in different NLP tasks.
• Score: 7.977161233209228
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Word representation has always been an important research area in the history of natural language processing (NLP). Understanding such complex text data is imperative, given that it is rich in information and can be used widely across various applications. In this survey, we explore different word representation models and its power of expression, from the classical to modern-day state-of-the-art word representation language models (LMS). We describe a variety of text representation methods, and model designs have blossomed in the context of NLP, including SOTA LMs. These models can transform large volumes of text into effective vector representations capturing the same semantic information. Further, such representations can be utilized by various machine learning (ML) algorithms for a variety of NLP related tasks. In the end, this survey briefly discusses the commonly used ML and DL based classifiers, evaluation metrics and the applications of these word embeddings in different NLP tasks.

Related papers

• On The Landscape of Spoken Language Models: A Comprehensive Survey [144.11278973534203]
  spoken language models (SLMs) act as universal speech processing systems. Work in this area is very diverse, with a range of terminology and evaluation settings.
  arXiv  Detail & Related papers  (2025-04-11T13:40:53Z)
• Natural Language Processing for Dialects of a Language: A Survey [56.93337350526933]
  State-of-the-art natural language processing (NLP) models are trained on massive training corpora, and report a superlative performance on evaluation datasets. This survey delves into an important attribute of these datasets: the dialect of a language. Motivated by the performance degradation of NLP models for dialectal datasets and its implications for the equity of language technologies, we survey past research in NLP for dialects in terms of datasets, and approaches.
  arXiv  Detail & Related papers  (2024-01-11T03:04:38Z)
• An Inclusive Notion of Text [69.36678873492373]
  We argue that clarity on the notion of text is crucial for reproducible and generalizable NLP. We introduce a two-tier taxonomy of linguistic and non-linguistic elements that are available in textual sources and can be used in NLP modeling.
  arXiv  Detail & Related papers  (2022-11-10T14:26:43Z)
• A Unified Understanding of Deep NLP Models for Text Classification [88.35418976241057]
  We have developed a visual analysis tool, DeepNLPVis, to enable a unified understanding of NLP models for text classification. The key idea is a mutual information-based measure, which provides quantitative explanations on how each layer of a model maintains the information of input words in a sample. A multi-level visualization, which consists of a corpus-level, a sample-level, and a word-level visualization, supports the analysis from the overall training set to individual samples.
  arXiv  Detail & Related papers  (2022-06-19T08:55:07Z)
• On the Universality of Deep COntextual Language Models [15.218264849664715]
  Deep Contextual Language Models (LMs) like ELMO, BERT, and their successors dominate the landscape of Natural Language Processing. Multilingual versions of such models like XLM-R and mBERT have given promising results in zero-shot cross-lingual transfer. Due to this initial success, pre-trained models are being used as Universal Language Models'
  arXiv  Detail & Related papers  (2021-09-15T08:00:33Z)
• High-dimensional distributed semantic spaces for utterances [0.2907403645801429]
  This paper describes a model for high-dimensional representation for utterance and text level data. It is based on a mathematically principled and behaviourally plausible approach to representing linguistic information. The paper shows how the implemented model is able to represent a broad range of linguistic features in a common integral framework of fixed dimensionality.
  arXiv  Detail & Related papers  (2021-04-01T12:09:47Z)
• Accurate Word Representations with Universal Visual Guidance [55.71425503859685]
  This paper proposes a visual representation method to explicitly enhance conventional word embedding with multiple-aspect senses from visual guidance. We build a small-scale word-image dictionary from a multimodal seed dataset where each word corresponds to diverse related images. Experiments on 12 natural language understanding and machine translation tasks further verify the effectiveness and the generalization capability of the proposed approach.
  arXiv  Detail & Related papers  (2020-12-30T09:11:50Z)
• Learning Universal Representations from Word to Sentence [89.82415322763475]
  This work introduces and explores the universal representation learning, i.e., embeddings of different levels of linguistic unit in a uniform vector space. We present our approach of constructing analogy datasets in terms of words, phrases and sentences. We empirically verify that well pre-trained Transformer models incorporated with appropriate training settings may effectively yield universal representation.
  arXiv  Detail & Related papers  (2020-09-10T03:53:18Z)
• Deep learning models for representing out-of-vocabulary words [1.4502611532302039]
  We present a performance evaluation of deep learning models for representing out-of-vocabulary (OOV) words. Although the best technique for handling OOV words is different for each task, Comick, a deep learning method that infers the embedding based on the context and the morphological structure of the OOV word, obtained promising results.
  arXiv  Detail & Related papers  (2020-07-14T19:31:25Z)
• A Comparative Study of Lexical Substitution Approaches based on Neural Language Models [117.96628873753123]
  We present a large-scale comparative study of popular neural language and masked language models. We show that already competitive results achieved by SOTA LMs/MLMs can be further improved if information about the target word is injected properly.
  arXiv  Detail & Related papers  (2020-05-29T18:43:22Z)
• Evaluating Transformer-Based Multilingual Text Classification [55.53547556060537]
  We argue that NLP tools perform unequally across languages with different syntactic and morphological structures. We calculate word order and morphological similarity indices to aid our empirical study.
  arXiv  Detail & Related papers  (2020-04-29T03:34:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.