Disentangling semantics in language through VAEs and a certain architectural choice

• URL: http://arxiv.org/abs/2012.13031v2
• Date: Mon, 28 Dec 2020 18:48:35 GMT
• Title: Disentangling semantics in language through VAEs and a certain architectural choice
• Authors: Ghazi Felhi, Joseph Le Roux, Djam\'e Seddah
• Abstract summary: We train a Variational Autoencoder to translate the sentence to a fixed number of hierarchically structured latent variables. We show that varying the corresponding latent variables results in varying these elements in sentences, and that swapping them between couples of sentences leads to the expected partial semantic swap.
• Score: 1.8907108368038217
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present an unsupervised method to obtain disentangled representations of sentences that single out semantic content. Using modified Transformers as building blocks, we train a Variational Autoencoder to translate the sentence to a fixed number of hierarchically structured latent variables. We study the influence of each latent variable in generation on the dependency structure of sentences, and on the predicate structure it yields when passed through an Open Information Extraction model. Our model could separate verbs, subjects, direct objects, and prepositional objects into latent variables we identified. We show that varying the corresponding latent variables results in varying these elements in sentences, and that swapping them between couples of sentences leads to the expected partial semantic swap.

Related papers

• Tracking linguistic information in transformer-based sentence embeddings through targeted sparsification [1.6021932740447968]
  Analyses of transformer-based models have shown that they encode a variety of linguistic information from their textual input. We test to what degree information about chunks (in particular noun, verb or prepositional phrases) can be localized in sentence embeddings. Our results show that such information is not distributed over the entire sentence embedding, but rather it is encoded in specific regions.
  arXiv  Detail & Related papers  (2024-07-25T15:27:08Z)
• Learning Syntax Without Planting Trees: Understanding When and Why Transformers Generalize Hierarchically [74.96551626420188]
  Transformers trained on natural language data have been shown to learn its hierarchical structure and generalize to sentences with unseen syntactic structures. We investigate sources of inductive bias in transformer models and their training that could cause such generalization behavior to emerge.
  arXiv  Detail & Related papers  (2024-04-25T07:10:29Z)
• Semantics of Multiword Expressions in Transformer-Based Models: A Survey [8.372465442144048]
  Multiword expressions (MWEs) are composed of multiple words and exhibit variable degrees of compositionality. We provide the first in-depth survey of MWE processing with transformer models. We find that they capture MWE semantics inconsistently, as shown by reliance on surface patterns and memorized information.
  arXiv  Detail & Related papers  (2024-01-27T11:51:11Z)
• Bridging Continuous and Discrete Spaces: Interpretable Sentence Representation Learning via Compositional Operations [80.45474362071236]
  It is unclear whether the compositional semantics of sentences can be directly reflected as compositional operations in the embedding space. We propose InterSent, an end-to-end framework for learning interpretable sentence embeddings.
  arXiv  Detail & Related papers  (2023-05-24T00:44:49Z)
• Syntactic Substitutability as Unsupervised Dependency Syntax [31.488677474152794]
  We model a more general property implicit in the definition of dependency relations, syntactic substitutability. This property captures the fact that words at either end of a dependency can be substituted with words from the same category. We show that increasing the number of substitutions used improves parsing accuracy on natural data.
  arXiv  Detail & Related papers  (2022-11-29T09:01:37Z)
• Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Roles [0.9582466286528458]
  We develop an Attention-Driven Variational Autoencoder (ADVAE) We show that it is possible to obtain representations of sentences where different syntactic roles correspond to clearly identified latent variables. Our work constitutes a first step towards unsupervised controllable content generation.
  arXiv  Detail & Related papers  (2022-06-22T15:50:01Z)
• Understanding Synonymous Referring Expressions via Contrastive Features [105.36814858748285]
  We develop an end-to-end trainable framework to learn contrastive features on the image and object instance levels. We conduct extensive experiments to evaluate the proposed algorithm on several benchmark datasets.
  arXiv  Detail & Related papers  (2021-04-20T17:56:24Z)
• Fake it Till You Make it: Self-Supervised Semantic Shifts for Monolingual Word Embedding Tasks [58.87961226278285]
  We propose a self-supervised approach to model lexical semantic change. We show that our method can be used for the detection of semantic change with any alignment method. We illustrate the utility of our techniques using experimental results on three different datasets.
  arXiv  Detail & Related papers  (2021-01-30T18:59:43Z)
• Unsupervised Distillation of Syntactic Information from Contextualized Word Representations [62.230491683411536]
  We tackle the task of unsupervised disentanglement between semantics and structure in neural language representations. To this end, we automatically generate groups of sentences which are structurally similar but semantically different. We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics.
  arXiv  Detail & Related papers  (2020-10-11T15:13:18Z)
• Composed Variational Natural Language Generation for Few-shot Intents [118.37774762596123]
  We generate training examples for few-shot intents in the realistic imbalanced scenario. To evaluate the quality of the generated utterances, experiments are conducted on the generalized few-shot intent detection task. Our proposed model achieves state-of-the-art performances on two real-world intent detection datasets.
  arXiv  Detail & Related papers  (2020-09-21T17:48:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.