Why shared attention vectors fail: a case for outcome-indexed tuning

• URL: http://arxiv.org/abs/2609.08615v1
• Date: Tue, 08 Sep 2026 11:52:08 GMT
• Title: Why shared attention vectors fail: a case for outcome-indexed tuning
• Abstract summary: We show that under multi-outcome learning, where models predict more than one outcome, a globally shared attentional vector becomes unstable.<n>We introduce an outcome-indexed attentional matrix that converts globally shared attentional tuning into an outcome-indexed representation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dimensional attention in learning is often implemented as a globally shared attention vector, where each stimulus dimension corresponds to a single scalar. These scalars are learned by models through gradient-descent on error, where predictive features acquire more salience. We show that under multi-outcome learning, where models predict more than one outcome, this shared vector becomes unstable; it collapses to its bounds and prevents the models from learning meaningful attentional tunings for learning and generalization. We address this by introducing an outcome-indexed attentional matrix that converts globally shared attentional tuning into an outcome-indexed representation. We present an analysis of the unstable shared vectors and derive the conditions under which it holds. Empirically, three synthetic experiments benchmark the proposed attention matrices and show that they converge to meaningful representations, something shared attention vectors fail to do. These results suggest that outcome-indexed attentional matrices are a general fix for gradient-based attentional processes, which improves models of learning under multi-outcome conditions.

Related papers

• Deconfounding Scores and Representation Learning for Causal Effect Estimation with Weak Overlap [75.41420283374737]
  Overlap is a key condition for causal treatment effect estimation.<n>We propose a class of feature representations called deconfounding scores.<n>We show that prognostic scores are overlap-optimal within this class.
  arXiv  Detail & Related papers  (2026-04-01T12:19:42Z)
• Early Quantization Shrinks Codebook: A Simple Fix for Diversity-Preserving Tokenization [15.662726701547896]
  Vector quantization is a technique in machine learning that discretizes continuous representations into a set of discrete vectors.<n>Despite its prevalence, the characteristics and behaviors of vector quantization in generative models remain largely underexplored.<n>This study systematically investigates the issue of collapses in vector quantization, where collapsed representations are observed across discrete codebook tokens and continuous latent embeddings.
  arXiv  Detail & Related papers  (2026-03-17T18:32:21Z)
• InfoNCE Induces Gaussian Distribution [7.8922077372145685]
  A loss in contrastive training is InfoNCE and its variants.<n>We show that the InfoNCE objective induces Gaussian structure in representations that emerge from contrastive training.<n>The resulting Gaussian model enables principled analytical treatment of learned representations and is expected to support a wide range of applications in contrastive learning.
  arXiv  Detail & Related papers  (2026-02-27T13:35:58Z)
• Generalization for Least Squares Regression With Simple Spiked Covariances [3.9134031118910264]
  The generalization properties of even two-layer neural networks trained by gradient descent remain poorly understood. Recent work has made progress by describing the spectrum of the feature matrix at the hidden layer. Yet, the generalization error for linear models with spiked covariances has not been previously determined.
  arXiv  Detail & Related papers  (2024-10-17T19:46:51Z)
• CARLA: Self-supervised Contrastive Representation Learning for Time Series Anomaly Detection [53.83593870825628]
  One main challenge in time series anomaly detection (TSAD) is the lack of labelled data in many real-life scenarios. Most of the existing anomaly detection methods focus on learning the normal behaviour of unlabelled time series in an unsupervised manner. We introduce a novel end-to-end self-supervised ContrAstive Representation Learning approach for time series anomaly detection.
  arXiv  Detail & Related papers  (2023-08-18T04:45:56Z)
• Learning Graphical Factor Models with Riemannian Optimization [70.13748170371889]
  This paper proposes a flexible algorithmic framework for graph learning under low-rank structural constraints. The problem is expressed as penalized maximum likelihood estimation of an elliptical distribution. We leverage geometries of positive definite matrices and positive semi-definite matrices of fixed rank that are well suited to elliptical models.
  arXiv  Detail & Related papers  (2022-10-21T13:19:45Z)
• On the Benefits of Large Learning Rates for Kernel Methods [110.03020563291788]
  We show that a phenomenon can be precisely characterized in the context of kernel methods. We consider the minimization of a quadratic objective in a separable Hilbert space, and show that with early stopping, the choice of learning rate influences the spectral decomposition of the obtained solution.
  arXiv  Detail & Related papers  (2022-02-28T13:01:04Z)
• Information-Theoretic Generalization Bounds for Iterative Semi-Supervised Learning [81.1071978288003]
  In particular, we seek to understand the behaviour of the em generalization error of iterative SSL algorithms using information-theoretic principles. Our theoretical results suggest that when the class conditional variances are not too large, the upper bound on the generalization error decreases monotonically with the number of iterations, but quickly saturates.
  arXiv  Detail & Related papers  (2021-10-03T05:38:49Z)
• Eigen Analysis of Self-Attention and its Reconstruction from Partial Computation [58.80806716024701]
  We study the global structure of attention scores computed using dot-product based self-attention. We find that most of the variation among attention scores lie in a low-dimensional eigenspace. We propose to compute scores only for a partial subset of token pairs, and use them to estimate scores for the remaining pairs.
  arXiv  Detail & Related papers  (2021-06-16T14:38:42Z)
• Benign overfitting in ridge regression [0.0]
  We provide non-asymptotic generalization bounds for overparametrized ridge regression. We identify when small or negative regularization is sufficient for obtaining small generalization error.
  arXiv  Detail & Related papers  (2020-09-29T20:00:31Z)
• Extreme Memorization via Scale of Initialization [72.78162454173803]
  We construct an experimental setup in which changing the scale of initialization strongly impacts the implicit regularization induced by SGD. We find that the extent and manner in which generalization ability is affected depends on the activation and loss function used. In the case of the homogeneous ReLU activation, we show that this behavior can be attributed to the loss function.
  arXiv  Detail & Related papers  (2020-08-31T04:53:11Z)
• The Slow Deterioration of the Generalization Error of the Random Feature Model [12.865834066050427]
  We show, theoretically and experimentally, that there is a dynamic self-correction mechanism at work. This gives us ample time to stop the training process and obtain solutions with good generalization property.
  arXiv  Detail & Related papers  (2020-08-13T00:35:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.