Adjusting Interpretable Dimensions in Embedding Space with Human Judgments

• URL: http://arxiv.org/abs/2404.02619v1
• Date: Wed, 3 Apr 2024 10:13:18 GMT
• Title: Adjusting Interpretable Dimensions in Embedding Space with Human Judgments
• Authors: Katrin Erk, Marianna Apidianaki,
• Abstract summary: Embedding spaces contain interpretable dimensions indicating gender, formality in style, or even object properties. The standard way to compute these dimensions uses contrasting seed words and computes difference vectors over them. We combine seed-based vectors with guidance from human ratings of where words fall along a specific dimension, and evaluate on predicting both object properties like size and danger.
• Score: 15.311454588182707
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Embedding spaces contain interpretable dimensions indicating gender, formality in style, or even object properties. This has been observed multiple times. Such interpretable dimensions are becoming valuable tools in different areas of study, from social science to neuroscience. The standard way to compute these dimensions uses contrasting seed words and computes difference vectors over them. This is simple but does not always work well. We combine seed-based vectors with guidance from human ratings of where words fall along a specific dimension, and evaluate on predicting both object properties like size and danger, and the stylistic properties of formality and complexity. We obtain interpretable dimensions with markedly better performance especially in cases where seed-based dimensions do not work well.

Related papers

• Relative intrinsic dimensionality is intrinsic to learning [49.5738281105287]
  We introduce a new notion of the intrinsic dimension of a data distribution, which precisely captures the separability properties of the data. For this intrinsic dimension, the rule of thumb above becomes a law: high intrinsic dimension guarantees highly separable data. We show thisRelative intrinsic dimension provides both upper and lower bounds on the probability of successfully learning and generalising in a binary classification problem.
  arXiv  Detail & Related papers  (2023-10-10T10:41:45Z)
• Representation Of Lexical Stylistic Features In Language Models' Embedding Space [28.60690854046176]
  We show that it is possible to derive a vector representation for each of these stylistic notions from only a small number of seed pairs. We conduct experiments on five datasets and find that static embeddings encode these features more accurately at the level of words and phrases. The lower performance of contextualized representations at the word level is partially attributable to the anisotropy of their vector space.
  arXiv  Detail & Related papers  (2023-05-29T23:44:26Z)
• Simple, Interpretable and Stable Method for Detecting Words with Usage Change across Corpora [54.757845511368814]
  The problem of comparing two bodies of text and searching for words that differ in their usage arises often in digital humanities and computational social science. This is commonly approached by training word embeddings on each corpus, aligning the vector spaces, and looking for words whose cosine distance in the aligned space is large. We propose an alternative approach that does not use vector space alignment, and instead considers the neighbors of each word.
  arXiv  Detail & Related papers  (2021-12-28T23:46:00Z)
• All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality [5.203329540700176]
  We call into question the informativity of such measures for contextualized language models. We find that a small number of rogue dimensions, often just 1-3, dominate similarity measures.
  arXiv  Detail & Related papers  (2021-09-09T16:45:15Z)
• HumanGPS: Geodesic PreServing Feature for Dense Human Correspondences [60.89437526374286]
  Prior art either assumes small motion between frames or relies on local descriptors, which cannot handle large motion or visually ambiguous body parts. We propose a deep learning framework that maps each pixel to a feature space, where the feature distances reflect the geodesic distances among pixels. Without any semantic annotation, the proposed embeddings automatically learn to differentiate visually similar parts and align different subjects into an unified feature space.
  arXiv  Detail & Related papers  (2021-03-29T12:43:44Z)
• Quadric hypersurface intersection for manifold learning in feature space [52.83976795260532]
  manifold learning technique suitable for moderately high dimension and large datasets. The technique is learned from the training data in the form of an intersection of quadric hypersurfaces. At test time, this manifold can be used to introduce an outlier score for arbitrary new points.
  arXiv  Detail & Related papers  (2021-02-11T18:52:08Z)
• ABID: Angle Based Intrinsic Dimensionality [0.0]
  The intrinsic dimensionality refers to the true'' dimensionality of the data, as opposed to the dimensionality of the data representation. Most popular methods for estimating the local intrinsic dimensionality are based on distances. We derive the theoretical distribution of angles and use this to construct an estimator for intrinsic dimensionality.
  arXiv  Detail & Related papers  (2020-06-23T10:19:34Z)
• Mental representations of objects reflect the ways in which we interact with them [1.0207955314209531]
  We introduce a method to represent objects in a space where each dimension corresponds to a broad mode of interaction. We show that the dimensions in this space can be used to predict categorical and functional dimensions in a state-of-the-art mental representation of objects.
  arXiv  Detail & Related papers  (2020-06-22T22:05:56Z)
• Geometry of Similarity Comparisons [51.552779977889045]
  We show that the ordinal capacity of a space form is related to its dimension and the sign of its curvature. More importantly, we show that the statistical behavior of the ordinal spread random variables defined on a similarity graph can be used to identify its underlying space form.
  arXiv  Detail & Related papers  (2020-06-17T13:37:42Z)
• Word Equations: Inherently Interpretable Sparse Word Embeddingsthrough Sparse Coding [0.0]
  We create a system where each dimension is associated with some human understandable hint that can describe the meaning of that dimension. We construct these embeddings through sparse coding, where each vector in the basis set is itself a word embedding. We show that models trained using these sparse embeddings can achieve good performance and are more interpretable in practice, including through human evaluations.
  arXiv  Detail & Related papers  (2020-04-08T19:49:49Z)
• Anchor & Transform: Learning Sparse Embeddings for Large Vocabularies [60.285091454321055]
  We design a simple and efficient embedding algorithm that learns a small set of anchor embeddings and a sparse transformation matrix. On text classification, language modeling, and movie recommendation benchmarks, we show that ANT is particularly suitable for large vocabulary sizes.
  arXiv  Detail & Related papers  (2020-03-18T13:07:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.