Uncovering Meanings of Embeddings via Partial Orthogonality

• URL: http://arxiv.org/abs/2310.17611v1
• Date: Thu, 26 Oct 2023 17:34:32 GMT
• Title: Uncovering Meanings of Embeddings via Partial Orthogonality
• Authors: Yibo Jiang, Bryon Aragam, Victor Veitch
• Abstract summary: Machine learning tools often rely on embedding text as vectors of real numbers. We study how the semantic structure of language is encoded in the algebraic structure of such embeddings.
• Score: 29.190972879474526
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Machine learning tools often rely on embedding text as vectors of real numbers. In this paper, we study how the semantic structure of language is encoded in the algebraic structure of such embeddings. Specifically, we look at a notion of ``semantic independence'' capturing the idea that, e.g., ``eggplant'' and ``tomato'' are independent given ``vegetable''. Although such examples are intuitive, it is difficult to formalize such a notion of semantic independence. The key observation here is that any sensible formalization should obey a set of so-called independence axioms, and thus any algebraic encoding of this structure should also obey these axioms. This leads us naturally to use partial orthogonality as the relevant algebraic structure. We develop theory and methods that allow us to demonstrate that partial orthogonality does indeed capture semantic independence. Complementary to this, we also introduce the concept of independence preserving embeddings where embeddings preserve the conditional independence structures of a distribution, and we prove the existence of such embeddings and approximations to them.

Related papers

• A Complexity-Based Theory of Compositionality [53.025566128892066]
  In AI, compositional representations can enable a powerful form of out-of-distribution generalization. Here, we propose a formal definition of compositionality that accounts for and extends our intuitions about compositionality. The definition is conceptually simple, quantitative, grounded in algorithmic information theory, and applicable to any representation.
  arXiv  Detail & Related papers  (2024-10-18T18:37:27Z)
• Transport of Algebraic Structure to Latent Embeddings [8.693845596949892]
  Machine learning often aims to produce latent embeddings of inputs which lie in a larger, abstract mathematical space. How can we learn to "union" two sets using only their latent embeddings while respecting associativity? We propose a general procedure for parameterizing latent space operations that are provably consistent with the laws on the input space.
  arXiv  Detail & Related papers  (2024-05-27T02:24:57Z)
• On Separation Logic, Computational Independence, and Pseudorandomness (Extended Version) [1.024113475677323]
  This work explores the nature of computational independence in a cryptographic scenario. We show that the semantics of separation logic can be adapted so as to account for complexity bounded adversaries. Remarkably, this allows for a fruitful interplay between independence and pseudorandomness.
  arXiv  Detail & Related papers  (2024-05-20T12:39:28Z)
• Evaluating the Robustness of Interpretability Methods through Explanation Invariance and Equivariance [72.50214227616728]
  Interpretability methods are valuable only if their explanations faithfully describe the explained model. We consider neural networks whose predictions are invariant under a specific symmetry group.
  arXiv  Detail & Related papers  (2023-04-13T17:59:03Z)
• On the Complexity of Representation Learning in Contextual Linear Bandits [110.84649234726442]
  We show that representation learning is fundamentally more complex than linear bandits. In particular, learning with a given set of representations is never simpler than learning with the worst realizable representation in the set.
  arXiv  Detail & Related papers  (2022-12-19T13:08:58Z)
• A substructural logic for quantum measurements [1.8782750537161614]
  This paper presents a substructural logic of sequents with very restricted exchange and weakening rules. It is sound with respect to sequences of measurements of a quantic system.
  arXiv  Detail & Related papers  (2022-12-06T09:11:42Z)
• Formalizing the presumption of independence [2.658812114255374]
  A key ingredient in such reasoning is the use of a "default" estimate of $mathbbE[XY] = mathbbE[X] mathbbE[Y]$. Reasoning based on this is commonplace, intuitively compelling, and often quite successful -- but completely informal. We present our main open problem: is there an estimator that formalizes intuitively valid applications of the presumption of independence without also accepting spurious arguments?
  arXiv  Detail & Related papers  (2022-11-12T20:28:19Z)
• Reply to Comment on "Strong Quantum Darwinism and Strong Independence are Equivalent to Spectrum Broadcast Structure" [77.34726150561087]
  In a recent comment [Feller et. al. arXiv:2101.09186] on our Letter [Phys. Lett. 122, 010403], Feller et. al identified a mistake in our mathematical expression of "strong independence" for states that satisfy Spectrum Broadcast Structure. We concede that we wrote a mathematical condition that is necessary but not sufficient. However, we used the original and correct qualitative definition for "strong independence" throughout the paper and in our proofs, therefore the proofs and statements, aside from the aforementioned mathematical expression, remain correct.
  arXiv  Detail & Related papers  (2021-01-26T12:55:49Z)
• Comment on "Strong Quantum Darwinism and Strong Independence are Equivalent to Spectrum Broadcast Structure" [62.997667081978825]
  We show that the mathematical formulation of condition (b) is necessary but not sufficient to ensure the equivalence. We propose a simple counter-example, together with a strengthened formulation of condition (b)
  arXiv  Detail & Related papers  (2021-01-21T16:06:25Z)
• Unsupervised Distillation of Syntactic Information from Contextualized Word Representations [62.230491683411536]
  We tackle the task of unsupervised disentanglement between semantics and structure in neural language representations. To this end, we automatically generate groups of sentences which are structurally similar but semantically different. We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics.
  arXiv  Detail & Related papers  (2020-10-11T15:13:18Z)
• Context-theoretic Semantics for Natural Language: an Algebraic Framework [0.0]
  We present a framework for natural language semantics in which words, phrases and sentences are all represented as vectors. We show that the vector representations of words can be considered as elements of an algebra over a field.
  arXiv  Detail & Related papers  (2020-09-22T13:31:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.