Multiplication Beyond Groups: Stratified Fourier Mechanisms in Transformer Circuits

• URL: http://arxiv.org/abs/2607.07066v1
• Date: Wed, 08 Jul 2026 06:49:48 GMT
• Title: Multiplication Beyond Groups: Stratified Fourier Mechanisms in Transformer Circuits
• Abstract summary: We investigate how small transformers learn modular integer multiplication over composite moduli.<n>We propose a localized generalization of Group Composition via Representation (GCR) that suggests the learned computation does not rely on a single global representation space.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformers have demonstrated a remarkable ability to learn algorithmic reasoning, yet mechanistic analyses have mostly focused on globally invertible operations such as cyclic addition and group composition. In this work, we investigate how small transformers learn modular integer multiplication over composite moduli, a fundamentally non-invertible operation due to the presence of zero-divisors. We propose the monoid extension: a localized generalization of Group Composition via Representation (GCR) that suggests the learned computation does not rely on a single global representation space. Instead, the model partitions the input space into local hierarchical algebraic regions, where group-like structure survives and Fourier mechanisms can be applied. In transformers trained on square-free modular multiplication, we find that embeddings organize around these regions, attention exhibits class-sensitive routing and low-rank write directions, and local character features explain a large fraction of the model's output logits. Our results suggest that representation-theoretic mechanisms previously identified for group operations can extend beyond groups to more general structures.

Related papers

• Neural Networks Provably Learn Spectral Representations for Group Composition [51.637084292404616]
  We study how structured internal structure emerges during neural network training.<n>We prove that the training dynamics are governed by a gradient ascent on a representation-theoretic energy functional.<n>This framework provides a representation-theoretic account of feature learning and characterizes a novel low-rank compression phenomenon.
  arXiv  Detail & Related papers  (2026-06-02T01:04:21Z)
• Assign and Add: A Mechanistic Study of Compositional Arithmetic [17.098170202572348]
  We study a mechanism for compositional generalization in transformers.<n>We observe that small transformers are able to generalize to previously unseen combinations of variables and numbers.<n>These results suggest that compositional generalization can be a natural consequence of the compositionality of internal mechanisms intransformers.
  arXiv  Detail & Related papers  (2026-05-29T16:19:20Z)
• Beyond Components: Singular Vector-Based Interpretability of Transformer Circuits [22.333229451408414]
  Transformer-based language models exhibit complex and distributed behavior, yet their internal computations remain poorly understood.<n>Existing interpretability methods treat attention heads and multilayer perceptron layers (MLPs) as indivisible units, overlooking possibilities of functional substructure learned within them.<n>We introduce a more fine-grained perspective that decomposes these components into singular directions, revealing superposed and independent computations within a single head or mechanistic.
  arXiv  Detail & Related papers  (2025-11-25T12:59:15Z)
• Unlocking Out-of-Distribution Generalization in Transformers via Recursive Latent Space Reasoning [50.99796659680724]
  This work investigates out-of-distribution (OOD) generalization in Transformer networks using a GSM8K-style modular arithmetic on computational graphs task as a testbed.<n>We introduce and explore a set of four architectural mechanisms aimed at enhancing OOD generalization.<n>We complement these empirical results with a detailed mechanistic interpretability analysis that reveals how these mechanisms give rise to robust OOD generalization abilities.
  arXiv  Detail & Related papers  (2025-10-15T21:03:59Z)
• Discovering Abstract Symbolic Relations by Learning Unitary Group Representations [7.303827428956944]
  We investigate a principled approach for symbolic operation completion (SOC) SOC poses a unique challenge in modeling abstract relationships between discrete symbols. We demonstrate that SOC can be efficiently solved by a minimal model - a bilinear map - with a novel factorized architecture.
  arXiv  Detail & Related papers  (2024-02-26T20:18:43Z)
• Towards Empirical Interpretation of Internal Circuits and Properties in Grokked Transformers on Modular Polynomials [29.09237503747052]
  Grokking on modular addition has been known to implement Fourier representation and its calculation circuits with trigonometric identities in Transformers.<n>We show that the transferability among the models grokked with each operation can be only limited to specific combinations.<n>Some multi-task mixtures may lead to co-grokking, where grokking simultaneously happens for all the tasks.
  arXiv  Detail & Related papers  (2024-02-26T16:48:12Z)
• How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations [98.7450564309923]
  This paper takes initial steps on understanding in-context learning (ICL) in more complex scenarios, by studying learning with representations. We construct synthetic in-context learning problems with a compositional structure, where the label depends on the input through a possibly complex but fixed representation function. We show theoretically the existence of transformers that approximately implement such algorithms with mild depth and size.
  arXiv  Detail & Related papers  (2023-10-16T17:40:49Z)
• A General Framework for Equivariant Neural Networks on Reductive Lie Groups [2.0769531810371307]
  Reductive Lie Groups play essential roles across scientific fields as diverse as high energy physics, quantum mechanics, quantum chromodynamics, molecular dynamics, computer vision, and imaging. We present a general Equivariant Neural Network architecture capable of respecting the finite-dimensional representations of any reductive Lie Group G.
  arXiv  Detail & Related papers  (2023-05-31T18:09:37Z)
• Homomorphism Autoencoder -- Learning Group Structured Representations from Observed Transitions [51.71245032890532]
  We propose methods enabling an agent acting upon the world to learn internal representations of sensory information consistent with actions that modify it. In contrast to existing work, our approach does not require prior knowledge of the group and does not restrict the set of actions the agent can perform.
  arXiv  Detail & Related papers  (2022-07-25T11:22:48Z)
• Capacity of Group-invariant Linear Readouts from Equivariant Representations: How Many Objects can be Linearly Classified Under All Possible Views? [21.06669693699965]
  We find that the fraction of separable dichotomies is determined by the dimension of the space that is fixed by the group action. We show how this relation extends to operations such as convolutions, element-wise nonlinearities, and global and local pooling.
  arXiv  Detail & Related papers  (2021-10-14T15:46:53Z)
• LieTransformer: Equivariant self-attention for Lie Groups [49.9625160479096]
  Group equivariant neural networks are used as building blocks of group invariant neural networks. We extend the scope of the literature to self-attention, that is emerging as a prominent building block of deep learning models. We propose the LieTransformer, an architecture composed of LieSelfAttention layers that are equivariant to arbitrary Lie groups and their discrete subgroups.
  arXiv  Detail & Related papers  (2020-12-20T11:02:49Z)
• Invariant Feature Coding using Tensor Product Representation [75.62232699377877]
  We prove that the group-invariant feature vector contains sufficient discriminative information when learning a linear classifier. A novel feature model that explicitly consider group action is proposed for principal component analysis and k-means clustering.
  arXiv  Detail & Related papers  (2019-06-05T07:15:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.