Sense representations for Portuguese: experiments with sense embeddings and deep neural language models

• URL: http://arxiv.org/abs/2109.00025v1
• Date: Tue, 31 Aug 2021 18:07:01 GMT
• Title: Sense representations for Portuguese: experiments with sense embeddings and deep neural language models
• Authors: Jessica Rodrigues da Silva, Helena de Medeiros Caseli
• Abstract summary: Unsupervised sense representations can induce different senses of a word by analyzing its contextual semantics in a text. We present the first experiments carried out for generating sense embeddings for Portuguese.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sense representations have gone beyond word representations like Word2Vec, GloVe and FastText and achieved innovative performance on a wide range of natural language processing tasks. Although very useful in many applications, the traditional approaches for generating word embeddings have a strict drawback: they produce a single vector representation for a given word ignoring the fact that ambiguous words can assume different meanings. In this paper, we explore unsupervised sense representations which, different from traditional word embeddings, are able to induce different senses of a word by analyzing its contextual semantics in a text. The unsupervised sense representations investigated in this paper are: sense embeddings and deep neural language models. We present the first experiments carried out for generating sense embeddings for Portuguese. Our experiments show that the sense embedding model (Sense2vec) outperformed traditional word embeddings in syntactic and semantic analogies task, proving that the language resource generated here can improve the performance of NLP tasks in Portuguese. We also evaluated the performance of pre-trained deep neural language models (ELMo and BERT) in two transfer learning approaches: feature based and fine-tuning, in the semantic textual similarity task. Our experiments indicate that the fine tuned Multilingual and Portuguese BERT language models were able to achieve better accuracy than the ELMo model and baselines.

Related papers

• What do Language Models know about word senses? Zero-Shot WSD with Language Models and Domain Inventories [23.623074512572593]
  We aim to explore to what extent language models are capable of discerning among senses at inference time. We leverage the relation between word senses and domains, and cast Word Sense Disambiguation (WSD) as a textual entailment problem. Our results show that this approach is indeed effective, close to supervised systems.
  arXiv  Detail & Related papers  (2023-02-07T09:55:07Z)
• ABINet++: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Spotting [121.11880210592497]
  We argue that the limited capacity of language models comes from 1) implicit language modeling; 2) unidirectional feature representation; and 3) language model with noise input. We propose an autonomous, bidirectional and iterative ABINet++ for scene text spotting.
  arXiv  Detail & Related papers  (2022-11-19T03:50:33Z)
• Learning an Artificial Language for Knowledge-Sharing in Multilingual Translation [15.32063273544696]
  We discretize the latent space of multilingual models by assigning encoder states to entries in a codebook. We validate our approach on large-scale experiments with realistic data volumes and domains. We also use the learned artificial language to analyze model behavior, and discover that using a similar bridge language increases knowledge-sharing among the remaining languages.
  arXiv  Detail & Related papers  (2022-11-02T17:14:42Z)
• Transparency Helps Reveal When Language Models Learn Meaning [71.96920839263457]
  Our systematic experiments with synthetic data reveal that, with languages where all expressions have context-independent denotations, both autoregressive and masked language models learn to emulate semantic relations between expressions. Turning to natural language, our experiments with a specific phenomenon -- referential opacity -- add to the growing body of evidence that current language models do not well-represent natural language semantics.
  arXiv  Detail & Related papers  (2022-10-14T02:35:19Z)
• Multilingual Word Sense Disambiguation with Unified Sense Representation [55.3061179361177]
  We propose building knowledge and supervised-based Multilingual Word Sense Disambiguation (MWSD) systems. We build unified sense representations for multiple languages and address the annotation scarcity problem for MWSD by transferring annotations from rich-sourced languages to poorer ones. Evaluations of SemEval-13 and SemEval-15 datasets demonstrate the effectiveness of our methodology.
  arXiv  Detail & Related papers  (2022-10-14T01:24:03Z)
• A Latent-Variable Model for Intrinsic Probing [93.62808331764072]
  We propose a novel latent-variable formulation for constructing intrinsic probes. We find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of morphosyntax.
  arXiv  Detail & Related papers  (2022-01-20T15:01:12Z)
• LMMS Reloaded: Transformer-based Sense Embeddings for Disambiguation and Beyond [2.9005223064604078]
  Recent Transformer-based Language Models have proven capable of producing contextual word representations that reliably convey sense-specific information. We introduce a more principled approach to leverage information from all layers of NLMs, informed by a probing analysis on 14 NLM variants. We also emphasize the versatility of these sense embeddings in contrast to task-specific models, applying them on several sense-related tasks, besides WSD.
  arXiv  Detail & Related papers  (2021-05-26T10:14:22Z)
• AM2iCo: Evaluating Word Meaning in Context across Low-ResourceLanguages with Adversarial Examples [51.048234591165155]
  We present AM2iCo, Adversarial and Multilingual Meaning in Context. It aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts. Results reveal that current SotA pretrained encoders substantially lag behind human performance.
  arXiv  Detail & Related papers  (2021-04-17T20:23:45Z)
• Infusing Finetuning with Semantic Dependencies [62.37697048781823]
  We show that, unlike syntax, semantics is not brought to the surface by today's pretrained models. We then use convolutional graph encoders to explicitly incorporate semantic parses into task-specific finetuning.
  arXiv  Detail & Related papers  (2020-12-10T01:27:24Z)
• On the Effects of Using word2vec Representations in Neural Networks for Dialogue Act Recognition [0.6767885381740952]
  We propose a new deep neural network that explores recurrent models to capture word sequences within sentences. We validate this model on three languages: English, French and Czech.
  arXiv  Detail & Related papers  (2020-10-22T07:21:17Z)
• Generating Sense Embeddings for Syntactic and Semantic Analogy for Portuguese [0.0]
  We use techniques to generate sense embeddings and present the first experiments carried out for Portuguese. Our experiments show that sense vectors outperform traditional word vectors in syntactic and semantic analogy tasks.
  arXiv  Detail & Related papers  (2020-01-21T14:39:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.