Representing Affect Information in Word Embeddings

• URL: http://arxiv.org/abs/2209.10583v1
• Date: Wed, 21 Sep 2022 18:16:33 GMT
• Title: Representing Affect Information in Word Embeddings
• Authors: Yuhan Zhang, Wenqi Chen, Ruihan Zhang, Xiajie Zhang
• Abstract summary: We investigated whether and how the affect meaning of a word is encoded in word embeddings pre-trained in large neural networks. The embeddings varied in being static or contextualized, and how much affect specific information was prioritized during the pre-training and fine-tuning phase.
• Score: 5.378735006566249
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: A growing body of research in natural language processing (NLP) and natural language understanding (NLU) is investigating human-like knowledge learned or encoded in the word embeddings from large language models. This is a step towards understanding what knowledge language models capture that resembles human understanding of language and communication. Here, we investigated whether and how the affect meaning of a word (i.e., valence, arousal, dominance) is encoded in word embeddings pre-trained in large neural networks. We used the human-labeled dataset as the ground truth and performed various correlational and classification tests on four types of word embeddings. The embeddings varied in being static or contextualized, and how much affect specific information was prioritized during the pre-training and fine-tuning phase. Our analyses show that word embedding from the vanilla BERT model did not saliently encode the affect information of English words. Only when the BERT model was fine-tuned on emotion-related tasks or contained extra contextualized information from emotion-rich contexts could the corresponding embedding encode more relevant affect information.

Related papers

• Pixel Sentence Representation Learning [67.4775296225521]
  In this work, we conceptualize the learning of sentence-level textual semantics as a visual representation learning process. We employ visually-grounded text perturbation methods like typos and word order shuffling, resonating with human cognitive patterns, and enabling perturbation to be perceived as continuous. Our approach is further bolstered by large-scale unsupervised topical alignment training and natural language inference supervision.
  arXiv  Detail & Related papers  (2024-02-13T02:46:45Z)
• Agentivit\`a e telicit\`a in GilBERTo: implicazioni cognitive [77.71680953280436]
  The goal of this study is to investigate whether a Transformer-based neural language model infers lexical semantics. The semantic properties considered are telicity (also combined with definiteness) and agentivity.
  arXiv  Detail & Related papers  (2023-07-06T10:52:22Z)
• Joint processing of linguistic properties in brains and language models [14.997785690790032]
  We investigate the correspondence between the detailed processing of linguistic information by the human brain versus language models. We find that elimination of specific linguistic properties results in a significant decrease in brain alignment. These findings provide clear evidence for the role of specific linguistic information in the alignment between brain and language models.
  arXiv  Detail & Related papers  (2022-12-15T19:13:42Z)
• A Linguistic Investigation of Machine Learning based Contradiction Detection Models: An Empirical Analysis and Future Perspectives [0.34998703934432673]
  We analyze two Natural Language Inference data sets with respect to their linguistic features. The goal is to identify those syntactic and semantic properties that are particularly hard to comprehend for a machine learning model.
  arXiv  Detail & Related papers  (2022-10-19T10:06:03Z)
• Transparency Helps Reveal When Language Models Learn Meaning [71.96920839263457]
  Our systematic experiments with synthetic data reveal that, with languages where all expressions have context-independent denotations, both autoregressive and masked language models learn to emulate semantic relations between expressions. Turning to natural language, our experiments with a specific phenomenon -- referential opacity -- add to the growing body of evidence that current language models do not well-represent natural language semantics.
  arXiv  Detail & Related papers  (2022-10-14T02:35:19Z)
• Knowledge Graph Fusion for Language Model Fine-tuning [0.0]
  We investigate the benefits of knowledge incorporation into the fine-tuning stages of BERT. An existing K-BERT model, which enriches sentences with triplets from a Knowledge Graph, is adapted for the English language. Changes made to K-BERT for accommodating the English language also extend to other word-based languages.
  arXiv  Detail & Related papers  (2022-06-21T08:06:22Z)
• CogAlign: Learning to Align Textual Neural Representations to Cognitive Language Processing Signals [60.921888445317705]
  We propose a CogAlign approach to integrate cognitive language processing signals into natural language processing models. We show that CogAlign achieves significant improvements with multiple cognitive features over state-of-the-art models on public datasets.
  arXiv  Detail & Related papers  (2021-06-10T07:10:25Z)
• Probing Pretrained Language Models for Lexical Semantics [76.73599166020307]
  We present a systematic empirical analysis across six typologically diverse languages and five different lexical tasks. Our results indicate patterns and best practices that hold universally, but also point to prominent variations across languages and tasks.
  arXiv  Detail & Related papers  (2020-10-12T14:24:01Z)
• Do Language Embeddings Capture Scales? [54.1633257459927]
  We show that pretrained language models capture a significant amount of information about the scalar magnitudes of objects. We identify contextual information in pre-training and numeracy as two key factors affecting their performance.
  arXiv  Detail & Related papers  (2020-10-11T21:11:09Z)
• Linguistic Profiling of a Neural Language Model [1.0552465253379135]
  We investigate the linguistic knowledge learned by a Neural Language Model (NLM) before and after a fine-tuning process. We show that BERT is able to encode a wide range of linguistic characteristics, but it tends to lose this information when trained on specific downstream tasks.
  arXiv  Detail & Related papers  (2020-10-05T09:09:01Z)
• On the Effects of Knowledge-Augmented Data in Word Embeddings [0.6749750044497732]
  We propose a novel approach for linguistic knowledge injection through data augmentation to learn word embeddings. We show our knowledge augmentation approach improves the intrinsic characteristics of the learned embeddings while not significantly altering their results on a downstream text classification task.
  arXiv  Detail & Related papers  (2020-10-05T02:14:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.