Partial Symmetry Enforced Attention Decomposition (PSEAD): A Group-Theoretic Framework for Equivariant Transformers in Biological Systems

• URL: http://arxiv.org/abs/2507.14908v1
• Date: Sun, 20 Jul 2025 10:44:31 GMT
• Title: Partial Symmetry Enforced Attention Decomposition (PSEAD): A Group-Theoretic Framework for Equivariant Transformers in Biological Systems
• Authors: Daniel Ayomide Olanrewaju,
• Abstract summary: This research introduces the Theory of Partial Symmetry Enforced Attention Decomposition (PSEAD)<n>We formalize the concept of local permutation subgroup actions on windows of biological data, proving that under such actions, the attention mechanism naturally decomposes into a direct sum of irreducible components.<n>This work lays the groundwork for a new generation of biologically informed, symmetry-aware artificial intelligence models.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This research introduces the Theory of Partial Symmetry Enforced Attention Decomposition (PSEAD), a new and rigorous group-theoretic framework designed to seamlessly integrate local symmetry awareness into the core architecture of self-attention mechanisms within Transformer models. We formalize the concept of local permutation subgroup actions on windows of biological data, proving that under such actions, the attention mechanism naturally decomposes into a direct sum of orthogonal irreducible components. Critically, these components are intrinsically aligned with the irreducible representations of the acting permutation subgroup, thereby providing a powerful mathematical basis for disentangling symmetric and asymmetric features. We show that PSEAD offers substantial advantages. These include enhanced generalization capabilities to novel biological motifs exhibiting similar partial symmetries, unprecedented interpretability by allowing direct visualization and analysis of attention contributions from different symmetry channels, and significant computational efficiency gains by focusing representational capacity on relevant symmetric subspaces. Beyond static data analysis, we extend PSEAD's applicability to dynamic biological processes within reinforcement learning paradigms, showcasing its potential to accelerate the discovery and optimization of biologically meaningful policies in complex environments like protein folding and drug discovery. This work lays the groundwork for a new generation of biologically informed, symmetry-aware artificial intelligence models.

Related papers

• Generalized Linear Mode Connectivity for Transformers [87.32299363530996]
  A striking phenomenon is linear mode connectivity (LMC), where independently trained models can be connected by low- or zero-loss paths.<n>Prior work has predominantly focused on neuron re-ordering through permutations, but such approaches are limited in scope.<n>We introduce a unified framework that captures four symmetry classes: permutations, semi-permutations, transformations, and general invertible maps.<n>This generalization enables, for the first time, the discovery of low- and zero-barrier linear paths between independently trained Vision Transformers and GPT-2 models.
  arXiv  Detail & Related papers  (2025-06-28T01:46:36Z)
• Lie symmetries and ghost-free representations of the Pais-Uhlenbeck model [44.99833362998488]
  We investigate the Pais-Uhlenbeck (PU) model, a paradigmatic example of a higher time-derivative theory.<n>Exploiting Lie symmetries in conjunction with the model's Bi-Hamiltonian structure, we construct distinct Poisson bracket formulations.<n>Our approach yields a unified framework for interpreting and stabilising higher time-derivative dynamics.
  arXiv  Detail & Related papers  (2025-05-09T15:16:40Z)
• Learning Symmetries via Weight-Sharing with Doubly Stochastic Tensors [46.59269589647962]
  Group equivariance has emerged as a valuable inductive bias in deep learning.<n>Group equivariant methods require the groups of interest to be known beforehand.<n>We show that when the dataset exhibits strong symmetries, the permutation matrices will converge to regular group representations.
  arXiv  Detail & Related papers  (2024-12-05T20:15:34Z)
• Deep Signature: Characterization of Large-Scale Molecular Dynamics [29.67824486345836]
  Deep Signature is a novel computationally tractable framework that characterizes complex dynamics and interatomic interactions.<n>Our approach incorporates soft spectral clustering that locally aggregates cooperative dynamics to reduce the size of the system, as well as signature transform to provide a global characterization of the non-smooth interactive dynamics.
  arXiv  Detail & Related papers  (2024-10-03T16:37:48Z)
• Relative Representations: Topological and Geometric Perspectives [53.88896255693922]
  Relative representations are an established approach to zero-shot model stitching.<n>We introduce a normalization procedure in the relative transformation, resulting in invariance to non-isotropic rescalings and permutations.<n>Second, we propose to deploy topological densification when fine-tuning relative representations, a topological regularization loss encouraging clustering within classes.
  arXiv  Detail & Related papers  (2024-09-17T08:09:22Z)
• Discovering Symmetry Breaking in Physical Systems with Relaxed Group Convolution [21.034937143252314]
  We focus on learning asymmetries of data using relaxed group convolutions.<n>We uncover various symmetry-breaking factors that are interpretable and physically meaningful in different physical systems.
  arXiv  Detail & Related papers  (2023-10-03T14:03:21Z)
• On discrete symmetries of robotics systems: A group-theoretic and data-driven analysis [38.92081817503126]
  We study discrete morphological symmetries of dynamical systems. These symmetries arise from the presence of one or more planes/axis of symmetry in the system's morphology. We exploit these symmetries using data augmentation and $G$-equivariant neural networks.
  arXiv  Detail & Related papers  (2023-02-21T04:10:16Z)
• The Surprising Effectiveness of Equivariant Models in Domains with Latent Symmetry [6.716931832076628]
  We show that imposing symmetry constraints that do not exactly match the domain symmetry is very helpful in learning the true symmetry in the environment. We demonstrate that an equivariant model can significantly outperform non-equivariant methods on domains with latent symmetries both in supervised learning and in reinforcement learning for robotic manipulation and control problems.
  arXiv  Detail & Related papers  (2022-11-16T21:51:55Z)
• Complexity from Adaptive-Symmetries Breaking: Global Minima in the Statistical Mechanics of Deep Neural Networks [0.0]
  An antithetical concept, adaptive symmetry, to conservative symmetry in physics is proposed to understand the deep neural networks (DNNs) We characterize the optimization process of a DNN system as an extended adaptive-symmetry-breaking process. More specifically, this process is characterized by a statistical-mechanical model that could be appreciated as a generalization of statistics physics.
  arXiv  Detail & Related papers  (2022-01-03T09:06:44Z)
• Discovering Latent Causal Variables via Mechanism Sparsity: A New Principle for Nonlinear ICA [81.4991350761909]
  Independent component analysis (ICA) refers to an ensemble of methods which formalize this goal and provide estimation procedure for practical application. We show that the latent variables can be recovered up to a permutation if one regularizes the latent mechanisms to be sparse.
  arXiv  Detail & Related papers  (2021-07-21T14:22:14Z)
• Harnessing Heterogeneity: Learning from Decomposed Feedback in Bayesian Modeling [68.69431580852535]
  We introduce a novel GP regression to incorporate the subgroup feedback. Our modified regression has provably lower variance -- and thus a more accurate posterior -- compared to previous approaches. We execute our algorithm on two disparate social problems.
  arXiv  Detail & Related papers  (2021-07-07T03:57:22Z)
• Inverse Learning of Symmetries [71.62109774068064]
  We learn the symmetry transformation with a model consisting of two latent subspaces. Our approach is based on the deep information bottleneck in combination with a continuous mutual information regulariser. Our model outperforms state-of-the-art methods on artificial and molecular datasets.
  arXiv  Detail & Related papers  (2020-02-07T13:48:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.