Interpretable Neural Embeddings with Sparse Self-Representation

• URL: http://arxiv.org/abs/2306.14135v1
• Date: Sun, 25 Jun 2023 05:57:01 GMT
• Title: Interpretable Neural Embeddings with Sparse Self-Representation
• Authors: Minxue Xia and Hao Zhu
• Abstract summary: Existing word embeddings are generally dense representations. Hence, the meaning of latent dimensions is difficult to interpret. This makes word embeddings like a black-box and prevents them from being human-readable and further manipulation. We propose a novel method to associate data self-representation with a shallow neural network to learn expressive, interpretable word embeddings.
• Score: 6.969983808566474
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Interpretability benefits the theoretical understanding of representations. Existing word embeddings are generally dense representations. Hence, the meaning of latent dimensions is difficult to interpret. This makes word embeddings like a black-box and prevents them from being human-readable and further manipulation. Many methods employ sparse representation to learn interpretable word embeddings for better interpretability. However, they also suffer from the unstable issue of grouped selection in $\ell1$ and online dictionary learning. Therefore, they tend to yield different results each time. To alleviate this challenge, we propose a novel method to associate data self-representation with a shallow neural network to learn expressive, interpretable word embeddings. In experiments, we report that the resulting word embeddings achieve comparable and even slightly better interpretability than baseline embeddings. Besides, we also evaluate that our approach performs competitively well on all downstream tasks and outperforms benchmark embeddings on a majority of them.

Related papers

• Pixel Sentence Representation Learning [67.4775296225521]
  In this work, we conceptualize the learning of sentence-level textual semantics as a visual representation learning process. We employ visually-grounded text perturbation methods like typos and word order shuffling, resonating with human cognitive patterns, and enabling perturbation to be perceived as continuous. Our approach is further bolstered by large-scale unsupervised topical alignment training and natural language inference supervision.
  arXiv  Detail & Related papers  (2024-02-13T02:46:45Z)
• SensePOLAR: Word sense aware interpretability for pre-trained contextual word embeddings [4.479834103607384]
  Adding interpretability to word embeddings represents an area of active research in text representation. We present SensePOLAR, an extension of the original POLAR framework that enables word-sense aware interpretability for pre-trained contextual word embeddings.
  arXiv  Detail & Related papers  (2023-01-11T20:25:53Z)
• Relational Sentence Embedding for Flexible Semantic Matching [86.21393054423355]
  We present Sentence Embedding (RSE), a new paradigm to discover further the potential of sentence embeddings. RSE is effective and flexible in modeling sentence relations and outperforms a series of state-of-the-art embedding methods.
  arXiv  Detail & Related papers  (2022-12-17T05:25:17Z)
• SUN: Exploring Intrinsic Uncertainties in Text-to-SQL Parsers [61.48159785138462]
  This paper aims to improve the performance of text-to-dependence by exploring the intrinsic uncertainties in the neural network based approaches (called SUN) Extensive experiments on five benchmark datasets demonstrate that our method significantly outperforms competitors and achieves new state-of-the-art results.
  arXiv  Detail & Related papers  (2022-09-14T06:27:51Z)
• Lost in Context? On the Sense-wise Variance of Contextualized Word Embeddings [11.475144702935568]
  We quantify how much the contextualized embeddings of each word sense vary across contexts in typical pre-trained models. We find that word representations are position-biased, where the first words in different contexts tend to be more similar.
  arXiv  Detail & Related papers  (2022-08-20T12:27:25Z)
• Fair Interpretable Learning via Correction Vectors [68.29997072804537]
  We propose a new framework for fair representation learning centered around the learning of "correction vectors" The corrections are then simply summed up to the original features, and can therefore be analyzed as an explicit penalty or bonus to each feature. We show experimentally that a fair representation learning problem constrained in such a way does not impact performance.
  arXiv  Detail & Related papers  (2022-01-17T10:59:33Z)
• Contextualized Semantic Distance between Highly Overlapped Texts [85.1541170468617]
  Overlapping frequently occurs in paired texts in natural language processing tasks like text editing and semantic similarity evaluation. This paper aims to address the issue with a mask-and-predict strategy. We take the words in the longest common sequence as neighboring words and use masked language modeling (MLM) to predict the distributions on their positions. Experiments on Semantic Textual Similarity show NDD to be more sensitive to various semantic differences, especially on highly overlapped paired texts.
  arXiv  Detail & Related papers  (2021-10-04T03:59:15Z)
• Perturbing Inputs for Fragile Interpretations in Deep Natural Language Processing [18.91129968022831]
  Interpretability methods need to be robust for trustworthy NLP applications in high-stake areas like medicine or finance. Our paper demonstrates how interpretations can be manipulated by making simple word perturbations on an input text.
  arXiv  Detail & Related papers  (2021-08-11T02:07:21Z)
• Interpretable Deep Learning: Interpretations, Interpretability, Trustworthiness, and Beyond [49.93153180169685]
  We introduce and clarify two basic concepts-interpretations and interpretability-that people usually get confused. We elaborate the design of several recent interpretation algorithms, from different perspectives, through proposing a new taxonomy. We summarize the existing work in evaluating models' interpretability using "trustworthy" interpretation algorithms.
  arXiv  Detail & Related papers  (2021-03-19T08:40:30Z)
• Can a Fruit Fly Learn Word Embeddings? [16.280120177501733]
  The fruit fly brain is one of the best studied systems in neuroscience. We show that a network motif can learn semantic representations of words and can generate both static and context-dependent word embeddings. It is shown that not only can the fruit fly network motif achieve performance comparable to existing methods in NLP, but, additionally, it uses only a fraction of the computational resources.
  arXiv  Detail & Related papers  (2021-01-18T05:41:50Z)
• Word Equations: Inherently Interpretable Sparse Word Embeddingsthrough Sparse Coding [0.0]
  We create a system where each dimension is associated with some human understandable hint that can describe the meaning of that dimension. We construct these embeddings through sparse coding, where each vector in the basis set is itself a word embedding. We show that models trained using these sparse embeddings can achieve good performance and are more interpretable in practice, including through human evaluations.
  arXiv  Detail & Related papers  (2020-04-08T19:49:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.