TPR-Attention for Combinatorial Generalization

• URL: http://arxiv.org/abs/2608.30124v1
• Date: Mon, 31 Aug 2026 01:18:44 GMT
• Title: TPR-Attention for Combinatorial Generalization
• Abstract summary: We introduce a new architectural component that embeds structured inductive bias into deep learning.<n>We show that this TPR-attention mechanism outperforms existing architectural components in generalization.
• Score: 1.1688030627514532
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Systematic generalization remains a significant challenge in deep learning. In particular, combinatorial generalization - generalizing to new configurations of known factors of variation - is effortless for humans but difficult for standard neural architectures that rely on statistical correlations rather than explicit structural representations. We introduce a new architectural component that embeds structured inductive bias into deep learning: an attention mechanism operating over tensor-product representations (TPRs). Through controlled experiments on compositional tasks, we show that this TPR-attention mechanism outperforms existing architectural components in combinatorial generalization. These results highlight the value of integrating explicit compositional structure into neural attention and point toward a promising path for models capable of systematic generalization.

Related papers

• Rethinking Generalization in Graph Neural Networks: A Structural Complexity Perspective [41.727617218821386]
  Graph neural networks (GNNs) have emerged as a fundamental tool for learning from graph-structured data.<n>We ask: does the graph structure actually influence generalization, and if so, by how much?<n>To answer the first question and validate our intuition, we theoretically prove that incorporating more edges into the prediction process transforms the input representations to be overly accommodating to the output model, thereby inducing overfitting.<n>To address the second question, we formulate a structural complexity measure based on the number of effective edges and derive a Rademacher complexity-based generalization bound.
  arXiv  Detail & Related papers  (2026-05-13T14:32:46Z)
• Reinforcement Learning for Compositional Generalization with Outcome-Level Optimization [11.743149066177642]
  Existing approaches often rely on supervised fine-tuning, which encourages models to imitate target outputs.<n>In this work, we investigate whether compositional generalization can instead be improved through outcome-level reinforcement learning.
  arXiv  Detail & Related papers  (2026-05-06T13:47:31Z)
• Layer Specialization Underlying Compositional Reasoning in Transformers [6.581088182267414]
  Transformers exhibit compositional reasoning on sequences not observed during training.<n>We investigate this phenomenon using the Random Hierarchy Model (RHM), a probabilistic context-free grammar.<n>Mechanistically, we identify a progressive emergence of layer specialization during training that correlates with generalization performance.
  arXiv  Detail & Related papers  (2025-10-20T12:08:22Z)
• Unlocking Out-of-Distribution Generalization in Transformers via Recursive Latent Space Reasoning [50.99796659680724]
  This work investigates out-of-distribution (OOD) generalization in Transformer networks using a GSM8K-style modular arithmetic on computational graphs task as a testbed.<n>We introduce and explore a set of four architectural mechanisms aimed at enhancing OOD generalization.<n>We complement these empirical results with a detailed mechanistic interpretability analysis that reveals how these mechanisms give rise to robust OOD generalization abilities.
  arXiv  Detail & Related papers  (2025-10-15T21:03:59Z)
• What makes Models Compositional? A Theoretical View: With Supplement [60.284698521569936]
  We propose a general neuro-symbolic definition of compositional functions and their compositional complexity. We show how various existing general and special purpose sequence processing models fit this definition and use it to analyze their compositional complexity.
  arXiv  Detail & Related papers  (2024-05-02T20:10:27Z)
• On the Role of Information Structure in Reinforcement Learning for Partially-Observable Sequential Teams and Games [55.2480439325792]
  In a sequential decision-making problem, the information structure is the description of how events in the system occurring at different points in time affect each other. By contrast, real-world sequential decision-making problems typically involve a complex and time-varying interdependence of system variables. We formalize a novel reinforcement learning model which explicitly represents the information structure.
  arXiv  Detail & Related papers  (2024-03-01T21:28:19Z)
• Inducing Systematicity in Transformers by Attending to Structurally Quantized Embeddings [60.698130703909804]
  Transformers generalize to novel compositions of structures and entities after being trained on a complex dataset. We propose SQ-Transformer that explicitly encourages systematicity in the embeddings and attention layers. We show that SQ-Transformer achieves stronger compositional generalization than the vanilla Transformer on multiple low-complexity semantic parsing and machine translation datasets.
  arXiv  Detail & Related papers  (2024-02-09T15:53:15Z)
• On the generalization capacity of neural networks during generic multimodal reasoning [20.1430673356983]
  We evaluate and compare large language models' capacity for multimodal generalization. For multimodal distractor and systematic generalization, either cross-modal attention or models with deeper attention layers are key architectural features required to integrate multimodal inputs.
  arXiv  Detail & Related papers  (2024-01-26T17:42:59Z)
• Neural-Symbolic Recursive Machine for Systematic Generalization [113.22455566135757]
  We introduce the Neural-Symbolic Recursive Machine (NSR), whose core is a Grounded Symbol System (GSS) NSR integrates neural perception, syntactic parsing, and semantic reasoning. We evaluate NSR's efficacy across four challenging benchmarks designed to probe systematic generalization capabilities.
  arXiv  Detail & Related papers  (2022-10-04T13:27:38Z)
• On Neural Architecture Inductive Biases for Relational Tasks [76.18938462270503]
  We introduce a simple architecture based on similarity-distribution scores which we name Compositional Network generalization (CoRelNet) We find that simple architectural choices can outperform existing models in out-of-distribution generalizations.
  arXiv  Detail & Related papers  (2022-06-09T16:24:01Z)
• Iterated learning for emergent systematicity in VQA [3.977144385787228]
  neural module networks have an architectural bias towards compositionality. When learning layouts and modules jointly, compositionality does not arise automatically and an explicit pressure is necessary for the emergence of layouts exhibiting the right structure. We propose to address this problem using iterated learning, a cognitive science theory of the emergence of compositional languages in nature.
  arXiv  Detail & Related papers  (2021-05-03T18:44:06Z)
• Improving Compositional Generalization in Semantic Parsing [54.4720965813889]
  Generalization of models to out-of-distribution (OOD) data has captured tremendous attention recently. We investigate compositional generalization in semantic parsing, a natural test-bed for compositional generalization.
  arXiv  Detail & Related papers  (2020-10-12T12:34:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.