Equivariant Music Transformer

• URL: http://arxiv.org/abs/2608.03920v1
• Date: Tue, 04 Aug 2026 16:51:58 GMT
• Title: Equivariant Music Transformer
• Abstract summary: We show that standard music transformers map such time-shifted or pitch-transposed inputs onto uncorrelated representations.<n>This suggests that additional model capacity is allocated to memorizing absolute patterns rather than capturing shared musical structures.<n>We propose the Equivariant Music Transformer (EMT), which enforces equivariance through self-distillation.
• Score: 12.784257850736372
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Humans recognize a musical passage even when it is shifted in time or transposed in pitch, indicating a notion of equivariance in the representation space. Our analysis, however, shows that standard music transformers map such time-shifted or pitch-transposed inputs onto uncorrelated representations: these models become progressively less equivariant as they scale in size or train longer. This suggests that in standard music transformers, additional model capacity is allocated to memorizing absolute patterns rather than capturing shared musical structures. In this paper, we propose the Equivariant Music Transformer (EMT), which enforces equivariance through self-distillation by jointly optimizing a next-token-prediction and an auxiliary equivariance regularization loss. We find that the additional equivariance loss acts as a beneficial regularizer, simultaneously improving next-token prediction and producing equivariant latent representations. Through both objective and subjective evaluations, EMT demonstrates superior equivariance and generative capability compared to data augmentation, feature engineering, and state-of-the-art (SOTA) baselines. More broadly, our findings reveal that standard language modeling methods alone do not capture music's translational symmetries, and dedicated inductive biases are required to produce better music representations. The code, weights and demos are available online.

Related papers

• Emergent musical properties of a transformer under contrastive self-supervised learning [8.007443439979944]
  We train a vision transformer with one-dimensional patches in the time--frequency domain.<n>We observe that, potentially thanks to weight sharing, informative musical properties emerge in ViT-1D's sequence tokens.<n>Our paper does not focus on improving performance but advances the musical interpretation of transformers.
  arXiv  Detail & Related papers  (2025-06-30T14:04:59Z)
• Music102: An $D_{12}$-equivariant transformer for chord progression accompaniment [0.0]
  Music102 is an advanced model aimed at enhancing chord progression accompaniment through a $D_12$-equivariant transformer.<n>Inspired by group theory and symbolic music structures, Music102 leverages musical symmetry--such as transposition and reflection operations--integrating these properties into the transformer architecture.
  arXiv  Detail & Related papers  (2024-10-23T03:11:01Z)
• Differential Transformer [99.5117269150629]
  Transformer tends to overallocate attention to irrelevant context.<n>We introduce Diff Transformer, which amplifies attention to relevant context while canceling noise.<n>It offers notable advantages in practical applications, such as long-context modeling, key information retrieval, hallucination mitigation, in-context learning, and reduction of activation outliers.
  arXiv  Detail & Related papers  (2024-10-07T17:57:38Z)
• Masked Mixers for Language Generation and Retrieval [0.0]
  We observe poor input representation accuracy in transformers and more accurate representation in what we term masked mixers.<n>A small masked mixer is shown to outperform a large and near state-of-the-art transformer-based retrieval model.
  arXiv  Detail & Related papers  (2024-09-02T22:17:18Z)
• iTransformer: Inverted Transformers Are Effective for Time Series Forecasting [62.40166958002558]
  We propose iTransformer, which simply applies the attention and feed-forward network on the inverted dimensions. The iTransformer model achieves state-of-the-art on challenging real-world datasets.
  arXiv  Detail & Related papers  (2023-10-10T13:44:09Z)
• Self-Supervised Learning for Group Equivariant Neural Networks [75.62232699377877]
  Group equivariant neural networks are the models whose structure is restricted to commute with the transformations on the input. We propose two concepts for self-supervised tasks: equivariant pretext labels and invariant contrastive loss. Experiments on standard image recognition benchmarks demonstrate that the equivariant neural networks exploit the proposed self-supervised tasks.
  arXiv  Detail & Related papers  (2023-03-08T08:11:26Z)
• The Lie Derivative for Measuring Learned Equivariance [84.29366874540217]
  We study the equivariance properties of hundreds of pretrained models, spanning CNNs, transformers, and Mixer architectures. We find that many violations of equivariance can be linked to spatial aliasing in ubiquitous network layers, such as pointwise non-linearities. For example, transformers can be more equivariant than convolutional neural networks after training.
  arXiv  Detail & Related papers  (2022-10-06T15:20:55Z)
• Topographic VAEs learn Equivariant Capsules [84.33745072274942]
  We introduce the Topographic VAE: a novel method for efficiently training deep generative models with topographically organized latent variables. We show that such a model indeed learns to organize its activations according to salient characteristics such as digit class, width, and style on MNIST. We demonstrate approximate equivariance to complex transformations, expanding upon the capabilities of existing group equivariant neural networks.
  arXiv  Detail & Related papers  (2021-09-03T09:25:57Z)
• Grounding inductive biases in natural images:invariance stems from variations in data [20.432568247732206]
  We study the factors of variation in a real dataset, ImageNet. We show standard augmentation relies on a precise combination of translation and scale. We find that the main factors of variation in ImageNet mostly relate to appearance.
  arXiv  Detail & Related papers  (2021-06-09T14:58:57Z)
• Relative Positional Encoding for Speech Recognition and Direct Translation [72.64499573561922]
  We adapt the relative position encoding scheme to the Speech Transformer. As a result, the network can better adapt to the variable distributions present in speech data.
  arXiv  Detail & Related papers  (2020-05-20T09:53:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.