Related papers: Compositional Structures in Neural Embedding and Interaction Decompositions

Compositional Structures in Neural Embedding and Interaction Decompositions

URL: http://arxiv.org/abs/2407.08934v1
Date: Fri, 12 Jul 2024 02:39:50 GMT
Title: Compositional Structures in Neural Embedding and Interaction Decompositions
Authors: Matthew Trager, Alessandro Achille, Pramuditha Perera, Luca Zancato, Stefano Soatto,
Abstract summary: We describe a basic correspondence between linear algebraic structures within vector embeddings in artificial neural networks. We introduce a characterization of compositional structures in terms of "interaction decompositions" We establish necessary and sufficient conditions for the presence of such structures within the representations of a model.
Score: 101.40245125955306
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: We describe a basic correspondence between linear algebraic structures within vector embeddings in artificial neural networks and conditional independence constraints on the probability distributions modeled by these networks. Our framework aims to shed light on the emergence of structural patterns in data representations, a phenomenon widely acknowledged but arguably still lacking a solid formal grounding. Specifically, we introduce a characterization of compositional structures in terms of "interaction decompositions," and we establish necessary and sufficient conditions for the presence of such structures within the representations of a model.

Related papers

Scaling Laws with Hidden Structure [2.474908349649168]
Recent advances suggest that text and image data contain such hidden structures, which help mitigate the curse of dimensionality. In this paper, we present a controlled experimental framework to test whether neural networks can indeed exploit such hidden factorial structures'' We find that they do leverage these latent patterns to learn discrete distributions more efficiently, and derive scaling laws linking model sizes, hidden factorizations, and accuracy.
arXiv Detail & Related papers (2024-11-02T22:32:53Z)
Geometric Relational Embeddings [19.383110247906256]
We propose relational embeddings, a paradigm of embeddings that respect the underlying symbolic structures. Results obtained from benchmark real-world datasets demonstrate the efficacy of geometric relational embeddings.
arXiv Detail & Related papers (2024-09-18T22:02:24Z)
Form-Finding and Physical Property Predictions of Tensegrity Structures Using Deep Neural Networks [39.19016806159609]
We develop a deep neural network (DNN) approach to predict the geometric configurations and physical properties of tensegrity structures. For validation, we analyze three tensegrity structures, including a tensegrity D-bar, prism, and lander.
arXiv Detail & Related papers (2024-06-15T16:39:53Z)
Learning Discrete Concepts in Latent Hierarchical Models [73.01229236386148]
Learning concepts from natural high-dimensional data holds potential in building human-aligned and interpretable machine learning models. We formalize concepts as discrete latent causal variables that are related via a hierarchical causal model. We substantiate our theoretical claims with synthetic data experiments.
arXiv Detail & Related papers (2024-06-01T18:01:03Z)
What makes Models Compositional? A Theoretical View: With Supplement [60.284698521569936]
We propose a general neuro-symbolic definition of compositional functions and their compositional complexity. We show how various existing general and special purpose sequence processing models fit this definition and use it to analyze their compositional complexity.
arXiv Detail & Related papers (2024-05-02T20:10:27Z)
StructRe: Rewriting for Structured Shape Modeling [63.792684115318906]
We present StructRe, a structure rewriting system, as a novel approach to structured shape modeling. Given a 3D object represented by points and components, StructRe can rewrite it upward into more concise structures, or downward into more detailed structures.
arXiv Detail & Related papers (2023-11-29T10:35:00Z)
Geometric Constraints in Probabilistic Manifolds: A Bridge from Molecular Dynamics to Structured Diffusion Processes [0.0]
We propose a method that enables sampling from distributions that rigorously adhere to arbitrary sets of geometric constraints in Euclidean spaces. This is achieved by integrating a constraint projection operator within the well-regarded architecture of Denoising Diffusion Probabilistic Models.
arXiv Detail & Related papers (2023-07-10T11:31:15Z)
Linear Spaces of Meanings: Compositional Structures in Vision-Language Models [110.00434385712786]
We investigate compositional structures in data embeddings from pre-trained vision-language models (VLMs) We first present a framework for understanding compositional structures from a geometric perspective. We then explain what these structures entail probabilistically in the case of VLM embeddings, providing intuitions for why they arise in practice.
arXiv Detail & Related papers (2023-02-28T08:11:56Z)
Non-standard entanglement structure of local unitary self-dual models as a saturated situation of repeatability in general probabilistic theories [61.12008553173672]
We show the existence of infinite structures of quantum composite system such that it is self-dual with local unitary symmetry. We also show the existence of a structure of quantum composite system such that non-orthogonal states in the structure are perfectly distinguishable.
arXiv Detail & Related papers (2021-11-29T23:37:58Z)
Emergent entanglement structures and self-similarity in quantum spin chains [0.0]
We introduce an experimentally accessible network representation for many-body quantum states based on entanglement between all pairs of its constituents. We illustrate the power of this representation by applying it to a paradigmatic spin chain model, the XX model, and showing that it brings to light new phenomena.
arXiv Detail & Related papers (2020-07-14T12:13:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.