What is my math transformer doing? -- Three results on interpretability and generalization

• URL: http://arxiv.org/abs/2211.00170v1
• Date: Mon, 31 Oct 2022 22:31:13 GMT
• Title: What is my math transformer doing? -- Three results on interpretability and generalization
• Authors: Fran\c{c}ois Charton
• Abstract summary: I show that incorrect model predictions still retain deep mathematical properties of the solution. I also show that the careful choice of a training dataset can accelerate training, while allowing the model to generalize out of its training distribution.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper investigates the failure cases and out-of-distribution behavior of transformers trained on matrix inversion and eigenvalue decomposition. I show that incorrect model predictions still retain deep mathematical properties of the solution (e.g. correct eigenvalues, unit norm of eigenvectors), and that almost all model failures can be attributed to, and predicted from, properties of the problem or solution. This demonstrates that, when in doubt, math transformers do not hallucinate absurd solutions (as was sometimes proposed) but remain ``roughly right''. I also show that the careful choice of a training dataset can accelerate training, while allowing the model to generalize out of its training distribution, invalidating the idea that transformers ``merely interpolate'' from memorized examples.

Related papers

• Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls [54.57326125204404]
  Language models are increasingly capable, yet still fail at a seemingly simple task of multi-digit multiplication.<n>We study why, by reverse-engineering a model that successfully learns multiplication via emphimplicit chain-of-thought'
  arXiv  Detail & Related papers  (2025-09-30T19:03:26Z)
• Born a Transformer -- Always a Transformer? [57.37263095476691]
  We study a family of $textitretrieval$ and $textitcopying$ tasks inspired by Liu et al.<n>We observe an $textitinduction-versus-anti-induction$ asymmetry, where pretrained models are better at retrieving tokens to the right (induction) than the left (anti-induction) of a query token.<n>Mechanistic analysis reveals that this asymmetry is connected to the differences in the strength of induction versus anti-induction circuits within pretrained transformers.
  arXiv  Detail & Related papers  (2025-05-27T21:36:50Z)
• How Transformers Learn In-Context Recall Tasks? Optimality, Training Dynamics and Generalization [23.759737527800585]
  We study the approximation capabilities, convergence speeds and on-convergence behaviors of transformers trained on in-context recall tasks.<n>We show that the trained transformers exhibit out-of-distribution generalization, i.e., generalizing to samples outside of the population distribution.
  arXiv  Detail & Related papers  (2025-05-21T01:26:44Z)
• On Vanishing Variance in Transformer Length Generalization [23.706900145711913]
  We show that even for today's frontier models, a longer sequence length results in a decrease in variance in the output of the multi-head attention modules. Our analyses attribute this improvement to a reduction-though not a complete elimination-of the distribution shift caused by vanishing variance.
  arXiv  Detail & Related papers  (2025-04-03T17:59:56Z)
• Symmetry and Generalisation in Machine Learning [0.0]
  We show that for any predictor that is not equivariant, there is an equivariant predictor with strictly lower test risk on all regression problems. We adopt an alternative perspective and formalise the common intuition that learning with invariant models reduces to a problem in terms of orbit representatives.
  arXiv  Detail & Related papers  (2025-01-07T15:14:58Z)
• Graph Transformers Dream of Electric Flow [72.06286909236827]
  We show that the linear Transformer, when applied to graph data, can implement algorithms that solve canonical problems. We present explicit weight configurations for implementing each such graph algorithm, and we bound the errors of the constructed Transformers by the errors of the underlying algorithms.
  arXiv  Detail & Related papers  (2024-10-22T05:11:45Z)
• Unsupervised Representation Learning from Sparse Transformation Analysis [79.94858534887801]
  We propose to learn representations from sequence data by factorizing the transformations of the latent variables into sparse components. Input data are first encoded as distributions of latent activations and subsequently transformed using a probability flow model.
  arXiv  Detail & Related papers  (2024-10-07T23:53:25Z)
• Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalization [88.5582111768376]
  We study the optimization of a Transformer composed of a self-attention layer with softmax followed by a fully connected layer under gradient descent on a certain data distribution model. Our results establish a sharp condition that can distinguish between the small test error phase and the large test error regime, based on the signal-to-noise ratio in the data model.
  arXiv  Detail & Related papers  (2024-09-28T13:24:11Z)
• Scaling and renormalization in high-dimensional regression [72.59731158970894]
  This paper presents a succinct derivation of the training and generalization performance of a variety of high-dimensional ridge regression models. We provide an introduction and review of recent results on these topics, aimed at readers with backgrounds in physics and deep learning.
  arXiv  Detail & Related papers  (2024-05-01T15:59:00Z)
• Setting the Record Straight on Transformer Oversmoothing [35.125957267464756]
  As model depth increases, Transformers oversmooth, i.e., inputs become more and more similar. We show that smoothing behavior depends on the eigenspectrum of the value and projection weights. Our analysis reveals a simple way to parameterize the weights of the Transformer update equations to influence smoothing behavior.
  arXiv  Detail & Related papers  (2024-01-09T01:19:03Z)
• Transformers can optimally learn regression mixture models [22.85684729248361]
  We show that transformers can learn an optimal predictor for mixtures of regressions. Experiments also demonstrate that transformers can learn mixtures of regressions in a sample-efficient fashion. We prove constructively that the decision-theoretic optimal procedure is indeed implementable by a transformer.
  arXiv  Detail & Related papers  (2023-11-14T18:09:15Z)
• Trained Transformers Learn Linear Models In-Context [39.56636898650966]
  Attention-based neural networks as transformers have demonstrated a remarkable ability to exhibit inattention learning (ICL) We show that when transformer training over random instances of linear regression problems, these models' predictions mimic nonlinear of ordinary squares.
  arXiv  Detail & Related papers  (2023-06-16T15:50:03Z)
• Transformers learn in-context by gradient descent [58.24152335931036]
  Training Transformers on auto-regressive objectives is closely related to gradient-based meta-learning formulations. We show how trained Transformers become mesa-optimizers i.e. learn models by gradient descent in their forward pass.
  arXiv  Detail & Related papers  (2022-12-15T09:21:21Z)
• The Lie Derivative for Measuring Learned Equivariance [84.29366874540217]
  We study the equivariance properties of hundreds of pretrained models, spanning CNNs, transformers, and Mixer architectures. We find that many violations of equivariance can be linked to spatial aliasing in ubiquitous network layers, such as pointwise non-linearities. For example, transformers can be more equivariant than convolutional neural networks after training.
  arXiv  Detail & Related papers  (2022-10-06T15:20:55Z)
• Linear algebra with transformers [0.0]
  We show that transformers can be trained to perform numerical calculations with high accuracy. We consider problems of linear algebra: matrix transposition, addition, multiplication, eigenvalues and vectors, singular value decomposition, and inversion.
  arXiv  Detail & Related papers  (2021-12-03T13:21:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.