Disentangling Linguistic Features with Dimension-Wise Analysis of Vector Embeddings

• URL: http://arxiv.org/abs/2504.14766v1
• Date: Sun, 20 Apr 2025 23:38:16 GMT
• Title: Disentangling Linguistic Features with Dimension-Wise Analysis of Vector Embeddings
• Authors: Saniya Karwa, Navpreet Singh,
• Abstract summary: This paper proposes a framework for uncovering the specific dimensions of vector embeddings that encode distinct linguistic properties (LPs)<n>We introduce the Linguistically Distinct Sentence Pairs dataset, which isolates ten key linguistic features such as synonymy, negation, tense, and quantity.<n>Using this dataset, we analyze BERT embeddings with various methods to identify the most influential dimensions for each LP.<n>Our findings show that certain properties, such as negation and polarity, are robustly encoded in specific dimensions, while others, like synonymy, exhibit more complex patterns.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding the inner workings of neural embeddings, particularly in models such as BERT, remains a challenge because of their high-dimensional and opaque nature. This paper proposes a framework for uncovering the specific dimensions of vector embeddings that encode distinct linguistic properties (LPs). We introduce the Linguistically Distinct Sentence Pairs (LDSP-10) dataset, which isolates ten key linguistic features such as synonymy, negation, tense, and quantity. Using this dataset, we analyze BERT embeddings with various methods, including the Wilcoxon signed-rank test, mutual information, and recursive feature elimination, to identify the most influential dimensions for each LP. We introduce a new metric, the Embedding Dimension Impact (EDI) score, which quantifies the relevance of each embedding dimension to a LP. Our findings show that certain properties, such as negation and polarity, are robustly encoded in specific dimensions, while others, like synonymy, exhibit more complex patterns. This study provides insights into the interpretability of embeddings, which can guide the development of more transparent and optimized language models, with implications for model bias mitigation and the responsible deployment of AI systems.

Related papers

• LayerFlow: Layer-wise Exploration of LLM Embeddings using Uncertainty-aware Interlinked Projections [11.252261879736102]
  LayerFlow is a visual analytics workspace that displays embeddings in an interlinked projection design.<n>It communicates the transformation, representation, and interpretation uncertainty.<n>We show the usability of the presented workspace through replication and expert case studies.
  arXiv  Detail & Related papers  (2025-04-09T12:24:58Z)
• The Complexity of Learning Sparse Superposed Features with Feedback [0.9838799448847586]
  We investigate whether the underlying learned features of a model can be efficiently retrieved through feedback from an agent.<n>We analyze the feedback complexity associated with learning a feature matrix in sparse settings.<n>Our results establish tight bounds when the agent is permitted to construct activations and demonstrate strong upper bounds in sparse scenarios.
  arXiv  Detail & Related papers  (2025-02-08T01:54:23Z)
• Making Pre-trained Language Models Great on Tabular Prediction [50.70574370855663]
  The transferability of deep neural networks (DNNs) has made significant progress in image and language processing. We present TP-BERTa, a specifically pre-trained LM for tabular data prediction. A novel relative magnitude tokenization converts scalar numerical feature values to finely discrete, high-dimensional tokens, and an intra-feature attention approach integrates feature values with the corresponding feature names.
  arXiv  Detail & Related papers  (2024-03-04T08:38:56Z)
• Learning Intrinsic Dimension via Information Bottleneck for Explainable Aspect-based Sentiment Analysis [30.16902652669842]
  We propose an Information Bottleneck-based Gradient (texttIBG) explanation framework for Aspect-based Sentiment Analysis (ABSA) Our framework refines word embeddings into a concise intrinsic dimension, maintaining essential features and omitting unrelated information. It considerably improves both the models' performance and interpretability by identifying sentiment-aware features.
  arXiv  Detail & Related papers  (2024-02-28T08:11:05Z)
• Investigating semantic subspaces of Transformer sentence embeddings through linear structural probing [2.5002227227256864]
  We present experiments with semantic structural probing, a method for studying sentence-level representations. We apply our method to language models from different families (encoder-only, decoder-only, encoder-decoder) and of different sizes in the context of two tasks. We find that model families differ substantially in their performance and layer dynamics, but that the results are largely model-size invariant.
  arXiv  Detail & Related papers  (2023-10-18T12:32:07Z)
• An Empirical Investigation of Commonsense Self-Supervision with Knowledge Graphs [67.23285413610243]
  Self-supervision based on the information extracted from large knowledge graphs has been shown to improve the generalization of language models. We study the effect of knowledge sampling strategies and sizes that can be used to generate synthetic data for adapting language models.
  arXiv  Detail & Related papers  (2022-05-21T19:49:04Z)
• Exploring Dimensionality Reduction Techniques in Multilingual Transformers [64.78260098263489]
  This paper gives a comprehensive account of the impact of dimensional reduction techniques on the performance of state-of-the-art multilingual Siamese Transformers. It shows that it is possible to achieve an average reduction in the number of dimensions of $91.58% pm 2.59%$ and $54.65% pm 32.20%$, respectively.
  arXiv  Detail & Related papers  (2022-04-18T17:20:55Z)
• A Latent-Variable Model for Intrinsic Probing [93.62808331764072]
  We propose a novel latent-variable formulation for constructing intrinsic probes. We find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of morphosyntax.
  arXiv  Detail & Related papers  (2022-01-20T15:01:12Z)
• A Comparative Study on Structural and Semantic Properties of Sentence Embeddings [77.34726150561087]
  We propose a set of experiments using a widely-used large-scale data set for relation extraction. We show that different embedding spaces have different degrees of strength for the structural and semantic properties. These results provide useful information for developing embedding-based relation extraction methods.
  arXiv  Detail & Related papers  (2020-09-23T15:45:32Z)
• Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck [52.08901549360262]
  Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. VAEs tend to ignore latent variables with a strong auto-regressive decoder. We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
  arXiv  Detail & Related papers  (2020-04-22T14:41:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.