Sentiment Analysis with Contextual Embeddings and Self-Attention

• URL: http://arxiv.org/abs/2003.05574v2
• Date: Mon, 5 Oct 2020 23:02:42 GMT
• Title: Sentiment Analysis with Contextual Embeddings and Self-Attention
• Authors: Katarzyna Biesialska, Magdalena Biesialska and Henryk Rybinski
• Abstract summary: In natural language the intended meaning of a word or phrase is often implicit and depends on the context. We propose a simple yet effective method for sentiment analysis using contextual embeddings and a self-attention mechanism. The experimental results for three languages, including morphologically rich Polish and German, show that our model is comparable to or even outperforms state-of-the-art models.
• Score: 3.0079490585515343
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In natural language the intended meaning of a word or phrase is often implicit and depends on the context. In this work, we propose a simple yet effective method for sentiment analysis using contextual embeddings and a self-attention mechanism. The experimental results for three languages, including morphologically rich Polish and German, show that our model is comparable to or even outperforms state-of-the-art models. In all cases the superiority of models leveraging contextual embeddings is demonstrated. Finally, this work is intended as a step towards introducing a universal, multilingual sentiment classifier.

Related papers

• Evaluating Contextualized Representations of (Spanish) Ambiguous Words: A New Lexical Resource and Empirical Analysis [2.2530496464901106]
  We evaluate semantic representations of Spanish ambiguous nouns in context in a suite of Spanish-language monolingual and multilingual BERT-based models. We find that various BERT-based LMs' contextualized semantic representations capture some variance in human judgments but fall short of the human benchmark.
  arXiv  Detail & Related papers  (2024-06-20T18:58:11Z)
• Natural Language Decompositions of Implicit Content Enable Better Text Representations [56.85319224208865]
  We introduce a method for the analysis of text that takes implicitly communicated content explicitly into account. We use a large language model to produce sets of propositions that are inferentially related to the text that has been observed. Our results suggest that modeling the meanings behind observed language, rather than the literal text alone, is a valuable direction for NLP.
  arXiv  Detail & Related papers  (2023-05-23T23:45:20Z)
• Transparency Helps Reveal When Language Models Learn Meaning [71.96920839263457]
  Our systematic experiments with synthetic data reveal that, with languages where all expressions have context-independent denotations, both autoregressive and masked language models learn to emulate semantic relations between expressions. Turning to natural language, our experiments with a specific phenomenon -- referential opacity -- add to the growing body of evidence that current language models do not well-represent natural language semantics.
  arXiv  Detail & Related papers  (2022-10-14T02:35:19Z)
• Testing the Ability of Language Models to Interpret Figurative Language [69.59943454934799]
  Figurative and metaphorical language are commonplace in discourse. It remains an open question to what extent modern language models can interpret nonliteral phrases. We introduce Fig-QA, a Winograd-style nonliteral language understanding task.
  arXiv  Detail & Related papers  (2022-04-26T23:42:22Z)
• A Latent-Variable Model for Intrinsic Probing [93.62808331764072]
  We propose a novel latent-variable formulation for constructing intrinsic probes. We find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of morphosyntax.
  arXiv  Detail & Related papers  (2022-01-20T15:01:12Z)
• Exploring the Representation of Word Meanings in Context: A Case Study on Homonymy and Synonymy [0.0]
  We assess the ability of both static and contextualized models to adequately represent different lexical-semantic relations. Experiments are performed in Galician, Portuguese, English, and Spanish.
  arXiv  Detail & Related papers  (2021-06-25T10:54:23Z)
• The Rediscovery Hypothesis: Language Models Need to Meet Linguistics [8.293055016429863]
  We study whether linguistic knowledge is a necessary condition for good performance of modern language models. We show that language models that are significantly compressed but perform well on their pretraining objectives retain good scores when probed for linguistic structures. This result supports the rediscovery hypothesis and leads to the second contribution of our paper: an information-theoretic framework that relates language modeling objective with linguistic information.
  arXiv  Detail & Related papers  (2021-03-02T15:57:39Z)
• Multi-sense embeddings through a word sense disambiguation process [2.2344764434954256]
  Most Suitable Sense. (MSSA) disambiguates and annotates each word by its specific sense, considering the semantic effects of its context. We test our approach on six different benchmarks for the word similarity task, showing that our approach can produce state-of-the-art results.
  arXiv  Detail & Related papers  (2021-01-21T16:22:34Z)
• Probing Contextual Language Models for Common Ground with Visual Representations [76.05769268286038]
  We design a probing model that evaluates how effective are text-only representations in distinguishing between matching and non-matching visual representations. Our findings show that language representations alone provide a strong signal for retrieving image patches from the correct object categories. Visually grounded language models slightly outperform text-only language models in instance retrieval, but greatly under-perform humans.
  arXiv  Detail & Related papers  (2020-05-01T21:28:28Z)
• How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context [59.13515950353125]
  We present a grammar-based decoding semantic parsing and adapt typical context modeling methods on top of it. We evaluate 13 context modeling methods on two large cross-domain datasets, and our best model achieves state-of-the-art performances.
  arXiv  Detail & Related papers  (2020-02-03T11:28:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.