Formal Semantic Geometry over Transformer-based Variational AutoEncoder

• URL: http://arxiv.org/abs/2210.06230v2
• Date: Tue, 11 Jun 2024 17:15:02 GMT
• Title: Formal Semantic Geometry over Transformer-based Variational AutoEncoder
• Authors: Yingji Zhang, Danilo S. Carvalho, Ian Pratt-Hartmann, André Freitas,
• Abstract summary: We frame the sentence semantics as the composition of textitsemantic role - word content features and propose the formal semantic geometry. We propose a new probing algorithm to guide the movement of sentence vectors over such geometry. Experimental results reveal that the formal semantic geometry can potentially deliver better control and interpretation to sentence generation.
• Score: 10.209508287003155
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Formal/symbolic semantics can provide canonical, rigid controllability and interpretability to sentence representations due to their \textit{localisation} or \textit{composition} property. How can we deliver such property to the current distributional sentence representations to control and interpret the generation of language models (LMs)? In this work, we theoretically frame the sentence semantics as the composition of \textit{semantic role - word content} features and propose the formal semantic geometry. To inject such geometry into Transformer-based LMs (i.e. GPT2), we deploy Transformer-based Variational AutoEncoder with a supervision approach, where the sentence generation can be manipulated and explained over low-dimensional latent Gaussian space. In addition, we propose a new probing algorithm to guide the movement of sentence vectors over such geometry. Experimental results reveal that the formal semantic geometry can potentially deliver better control and interpretation to sentence generation.

Related papers

• Unveiling Transformer Perception by Exploring Input Manifolds [41.364418162255184]
  This paper introduces a general method for the exploration of equivalence classes in the input space of Transformer models. The proposed approach is based on sound mathematical theory which describes the internal layers of a Transformer architecture as sequential deformations of the input manifold.
  arXiv  Detail & Related papers  (2024-10-08T13:20:31Z)
• Transformers Can Represent $n$-gram Language Models [56.06361029539347]
  We focus on the relationship between transformer LMs and $n$-gram LMs, a simple and historically relevant class of language models. We show that transformer LMs using the hard or sparse attention mechanisms can exactly represent any $n$-gram LM.
  arXiv  Detail & Related papers  (2024-04-23T12:51:37Z)
• Principled Paraphrase Generation with Parallel Corpora [52.78059089341062]
  We formalize the implicit similarity function induced by round-trip Machine Translation. We show that it is susceptible to non-paraphrase pairs sharing a single ambiguous translation. We design an alternative similarity metric that mitigates this issue.
  arXiv  Detail & Related papers  (2022-05-24T17:22:42Z)
• Frame Averaging for Equivariant Shape Space Learning [85.42901997467754]
  A natural way to incorporate symmetries in shape space learning is to ask that the mapping to the shape space (encoder) and mapping from the shape space (decoder) are equivariant to the relevant symmetries. We present a framework for incorporating equivariance in encoders and decoders by introducing two contributions.
  arXiv  Detail & Related papers  (2021-12-03T06:41:19Z)
• Self-Interpretable Model with TransformationEquivariant Interpretation [11.561544418482585]
  We propose a self-interpretable model SITE with transformation-equivariant interpretations. We focus on the robustness and self-consistency of the interpretations of geometric transformations.
  arXiv  Detail & Related papers  (2021-11-09T03:21:25Z)
• GTAE: Graph-Transformer based Auto-Encoders for Linguistic-Constrained Text Style Transfer [119.70961704127157]
  Non-parallel text style transfer has attracted increasing research interests in recent years. Current approaches still lack the ability to preserve the content and even logic of original sentences. We propose a method called Graph Transformer based Auto-GTAE, which models a sentence as a linguistic graph and performs feature extraction and style transfer at the graph level.
  arXiv  Detail & Related papers  (2021-02-01T11:08:45Z)
• Unsupervised Distillation of Syntactic Information from Contextualized Word Representations [62.230491683411536]
  We tackle the task of unsupervised disentanglement between semantics and structure in neural language representations. To this end, we automatically generate groups of sentences which are structurally similar but semantically different. We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics.
  arXiv  Detail & Related papers  (2020-10-11T15:13:18Z)
• Neural Syntactic Preordering for Controlled Paraphrase Generation [57.5316011554622]
  Our work uses syntactic transformations to softly "reorder'' the source sentence and guide our neural paraphrasing model. First, given an input sentence, we derive a set of feasible syntactic rearrangements using an encoder-decoder model. Next, we use each proposed rearrangement to produce a sequence of position embeddings, which encourages our final encoder-decoder paraphrase model to attend to the source words in a particular order.
  arXiv  Detail & Related papers  (2020-05-05T09:02:25Z)
• Putting a Spin on Language: A Quantum Interpretation of Unary Connectives for Linguistic Applications [0.0]
  Lambek Calculus relies on unary modalities to allow controlled application of structural rules. Proposals for compositional interpretation of Lambek Calculus in the compact closed category of FVect and linear maps have been made. Our aim is to turn the modalities into first-class citizens of the vectorial interpretation.
  arXiv  Detail & Related papers  (2020-04-08T17:25:11Z)
• Syntax-driven Iterative Expansion Language Models for Controllable Text Generation [2.578242050187029]
  We propose a new paradigm for introducing a syntactic inductive bias into neural text generation. Our experiments show that this paradigm is effective at text generation, with quality between LSTMs and Transformers, and comparable diversity.
  arXiv  Detail & Related papers  (2020-04-05T14:29:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.