Disentangling continuous and discrete linguistic signals in transformer-based sentence embeddings

• URL: http://arxiv.org/abs/2312.11272v1
• Date: Mon, 18 Dec 2023 15:16:54 GMT
• Title: Disentangling continuous and discrete linguistic signals in transformer-based sentence embeddings
• Authors: Vivi Nastase and Paola Merlo
• Abstract summary: We explore whether we can compress transformer-based sentence embeddings into a representation that separates different linguistic signals. We show that by compressing an input sequence that shares a targeted phenomenon into the latent layer of a variational autoencoder-like system, the targeted linguistic information becomes more explicit.
• Score: 1.8927791081850118
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Sentence and word embeddings encode structural and semantic information in a distributed manner. Part of the information encoded -- particularly lexical information -- can be seen as continuous, whereas other -- like structural information -- is most often discrete. We explore whether we can compress transformer-based sentence embeddings into a representation that separates different linguistic signals -- in particular, information relevant to subject-verb agreement and verb alternations. We show that by compressing an input sequence that shares a targeted phenomenon into the latent layer of a variational autoencoder-like system, the targeted linguistic information becomes more explicit. A latent layer with both discrete and continuous components captures better the targeted phenomena than a latent layer with only discrete or only continuous components. These experiments are a step towards separating linguistic signals from distributed text embeddings and linking them to more symbolic representations.

Related papers

• Tracking linguistic information in transformer-based sentence embeddings through targeted sparsification [1.6021932740447968]
  Analyses of transformer-based models have shown that they encode a variety of linguistic information from their textual input. We test to what degree information about chunks (in particular noun, verb or prepositional phrases) can be localized in sentence embeddings. Our results show that such information is not distributed over the entire sentence embedding, but rather it is encoded in specific regions.
  arXiv  Detail & Related papers  (2024-07-25T15:27:08Z)
• Non-verbal information in spontaneous speech -- towards a new framework of analysis [0.5559722082623594]
  This paper offers an analytical schema and a technological proof-of-concept for the categorization of prosodic signals. We present a classification process that disentangles prosodic phenomena of three orders. Disentangling prosodic patterns can direct a theory of communication and speech organization.
  arXiv  Detail & Related papers  (2024-03-06T08:03:05Z)
• Quantifying the redundancy between prosody and text [67.07817268372743]
  We use large language models to estimate how much information is redundant between prosody and the words themselves. We find a high degree of redundancy between the information carried by the words and prosodic information across several prosodic features. Still, we observe that prosodic features can not be fully predicted from text, suggesting that prosody carries information above and beyond the words.
  arXiv  Detail & Related papers  (2023-11-28T21:15:24Z)
• Bridging Continuous and Discrete Spaces: Interpretable Sentence Representation Learning via Compositional Operations [80.45474362071236]
  It is unclear whether the compositional semantics of sentences can be directly reflected as compositional operations in the embedding space. We propose InterSent, an end-to-end framework for learning interpretable sentence embeddings.
  arXiv  Detail & Related papers  (2023-05-24T00:44:49Z)
• Generating Coherent Narratives by Learning Dynamic and Discrete Entity States with a Contrastive Framework [68.1678127433077]
  We extend the Transformer model to dynamically conduct entity state updates and sentence realization for narrative generation. Experiments on two narrative datasets show that our model can generate more coherent and diverse narratives than strong baselines.
  arXiv  Detail & Related papers  (2022-08-08T09:02:19Z)
• Latent Topology Induction for Understanding Contextualized Representations [84.7918739062235]
  We study the representation space of contextualized embeddings and gain insight into the hidden topology of large language models. We show there exists a network of latent states that summarize linguistic properties of contextualized representations.
  arXiv  Detail & Related papers  (2022-06-03T11:22:48Z)
• Variable-rate discrete representation learning [20.81400194698063]
  We propose slow autoencoders for unsupervised learning of high-level variable-rate discrete representations of sequences. We show that the resulting event-based representations automatically grow or shrink depending on the density of salient information in the input signals. We develop run-length Transformers for event-based representation modelling and use them to construct language models in the speech domain.
  arXiv  Detail & Related papers  (2021-03-10T14:42:31Z)
• Unsupervised Distillation of Syntactic Information from Contextualized Word Representations [62.230491683411536]
  We tackle the task of unsupervised disentanglement between semantics and structure in neural language representations. To this end, we automatically generate groups of sentences which are structurally similar but semantically different. We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics.
  arXiv  Detail & Related papers  (2020-10-11T15:13:18Z)
• Multi-channel Transformers for Multi-articulatory Sign Language Translation [59.38247587308604]
  We tackle the multi-articulatory sign language translation task and propose a novel multi-channel transformer architecture. The proposed architecture allows both the inter and intra contextual relationships between different sign articulators to be modelled within the transformer network itself.
  arXiv  Detail & Related papers  (2020-09-01T09:10:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.