How do Transformer Embeddings Represent Compositions? A Functional Analysis

• URL: http://arxiv.org/abs/2506.00914v1
• Date: Sun, 01 Jun 2025 09:02:56 GMT
• Title: How do Transformer Embeddings Represent Compositions? A Functional Analysis
• Abstract summary: We test compositionality in Mistral, OpenAI Large, and Google embedding models, and compare them with BERT.<n>We find that ridge regression, albeit linear, best accounts for compositionality.<n>Surprisingly, we find that the classic vector addition model performs almost as well as any other model.
• Score: 5.0310636905746975
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Compositionality is a key aspect of human intelligence, essential for reasoning and generalization. While transformer-based models have become the de facto standard for many language modeling tasks, little is known about how they represent compound words, and whether these representations are compositional. In this study, we test compositionality in Mistral, OpenAI Large, and Google embedding models, and compare them with BERT. First, we evaluate compositionality in the representations by examining six diverse models of compositionality (addition, multiplication, dilation, regression, etc.). We find that ridge regression, albeit linear, best accounts for compositionality. Surprisingly, we find that the classic vector addition model performs almost as well as any other model. Next, we verify that most embedding models are highly compositional, while BERT shows much poorer compositionality. We verify and visualize our findings with a synthetic dataset consisting of fully transparent adjective-noun compositions. Overall, we present a thorough investigation of compositionality.

Related papers

• Quantifying Compositionality of Classic and State-of-the-Art Embeddings [9.862058044710704]
  We formalize a two-step evaluation that measures the linearity between known entity attributes and their embeddings.<n>We also check reconstruction metrics such as L2 loss, cosine similarity, and retrieval accuracy.<n> Stronger compositional signals are observed in later training stages across data modalities.
  arXiv  Detail & Related papers  (2025-09-14T18:28:21Z)
• Does Data Scaling Lead to Visual Compositional Generalization? [21.242714408660508]
  We find that compositional generalization is driven by data diversity, not mere data scale.<n>We prove this structure is key to efficiency, enabling perfect generalization from few observed combinations.
  arXiv  Detail & Related papers  (2025-07-09T17:59:03Z)
• A Systematic Study of Compositional Syntactic Transformer Language Models [37.38087762297668]
  This paper focuses on compositional SLMs that are based on constituency parse trees and contain explicit bottom-up composition of constituent representations.<n>We identify key aspects of design choices in existing compositional SLMs and propose a unified framework encompassing both existing models and novel variants.
  arXiv  Detail & Related papers  (2025-06-28T18:32:23Z)
• Towards a Comparative Framework for Compositional AI Models [0.0]
  We show how models can learn to compositionally generalise using the DisCoCirc framework for natural language processing.<n>We compare both quantum circuit based models, as well as classical neural networks, on a dataset derived from one of the bAbI tasks.<n>Both architectures score within 5% of one another on the productivity and substitutivity tasks, but differ by at least 10% for the systematicity task.
  arXiv  Detail & Related papers  (2025-06-27T15:59:14Z)
• How compositional generalization and creativity improve as diffusion models are trained [82.08869888944324]
  How many samples do generative models need in order to learn composition rules?<n>What signal in the data is exploited to learn those rules?<n>We discuss connections between the hierarchical clustering mechanism we introduce here and the renormalization group in physics.
  arXiv  Detail & Related papers  (2025-02-17T18:06:33Z)
• A Complexity-Based Theory of Compositionality [53.025566128892066]
  In AI, compositional representations can enable a powerful form of out-of-distribution generalization.<n>Here, we propose a definition, which we call representational compositionality, that accounts for and extends our intuitions about compositionality.<n>We show how it unifies disparate intuitions from across the literature in both AI and cognitive science.
  arXiv  Detail & Related papers  (2024-10-18T18:37:27Z)
• MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Models [85.10375181040436]
  We propose MMCOMPOSITION, a novel human-annotated benchmark for comprehensively and accurately evaluating Vision-Language Models. We find GPT-4o's compositionality inferior to the best open-source model, and we analyze the underlying reasons.
  arXiv  Detail & Related papers  (2024-10-13T05:35:09Z)
• What makes Models Compositional? A Theoretical View: With Supplement [60.284698521569936]
  We propose a general neuro-symbolic definition of compositional functions and their compositional complexity. We show how various existing general and special purpose sequence processing models fit this definition and use it to analyze their compositional complexity.
  arXiv  Detail & Related papers  (2024-05-02T20:10:27Z)
• Syntax-guided Neural Module Distillation to Probe Compositionality in Sentence Embeddings [0.0]
  We construct a neural module net based on its syntax parse and train it end-to-end to approximate the sentence's embedding. We find differences in the distillability of various sentence embedding models that broadly correlate with their performance. Preliminary evidence that much syntax-guided composition in sentence embedding models is linear.
  arXiv  Detail & Related papers  (2023-01-21T19:42:02Z)
• When and why vision-language models behave like bags-of-words, and what to do about it? [39.90099818890488]
  We create the Attribution, Relation, and Order benchmark to evaluate the ability of VLMs to understand different types of relationships, attributes, and order. ARO is orders of magnitude larger than previous benchmarks of compositionality, with more than 50,000 test cases. We show where state-of-the-art VLMs have poor relational understanding, can blunder when linking objects to their attributes, and demonstrate a severe lack of order sensitivity.
  arXiv  Detail & Related papers  (2022-10-04T22:13:25Z)
• What do Toothbrushes do in the Kitchen? How Transformers Think our World is Structured [137.83584233680116]
  We investigate what extent transformer-based language models allow for extracting knowledge about object relations. We show that the models combined with the different similarity measures differ greatly in terms of the amount of knowledge they allow for extracting. Surprisingly, static models perform almost as well as contextualized models -- in some cases even better.
  arXiv  Detail & Related papers  (2022-04-12T10:00:20Z)
• Compositionality as Lexical Symmetry [42.37422271002712]
  In tasks like semantic parsing, instruction following, and question answering, standard deep networks fail to generalize compositionally from small datasets. We present a domain-general and model-agnostic formulation of compositionality as a constraint on symmetries of data distributions rather than models. We describe a procedure called LEXSYM that discovers these transformations automatically, then applies them to training data for ordinary neural sequence models.
  arXiv  Detail & Related papers  (2022-01-30T21:44:46Z)
• Learning Graph Embeddings for Compositional Zero-shot Learning [73.80007492964951]
  In compositional zero-shot learning, the goal is to recognize unseen compositions of observed visual primitives states. We propose a novel graph formulation called Compositional Graph Embedding (CGE) that learns image features and latent representations of visual primitives in an end-to-end manner. By learning a joint compatibility that encodes semantics between concepts, our model allows for generalization to unseen compositions without relying on an external knowledge base like WordNet.
  arXiv  Detail & Related papers  (2021-02-03T10:11:03Z)
• Assessing Phrasal Representation and Composition in Transformers [13.460125148455143]
  Deep transformer models have pushed performance on NLP tasks to new limits. We present systematic analysis of phrasal representations in state-of-the-art pre-trained transformers. We find that phrase representation in these models relies heavily on word content, with little evidence of nuanced composition.
  arXiv  Detail & Related papers  (2020-10-08T04:59:39Z)
• A Study of Compositional Generalization in Neural Models [22.66002315559978]
  We introduce ConceptWorld, which enables the generation of images from compositional and relational concepts. We perform experiments to test the ability of standard neural networks to generalize on relations with compositional arguments. For simple problems, all models generalize well to close concepts but struggle with longer compositional chains.
  arXiv  Detail & Related papers  (2020-06-16T18:29:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.