A Coulomb Particle Model for Learning Kernel Attention in Transformers

• URL: http://arxiv.org/abs/2607.23869v1
• Date: Sun, 26 Jul 2026 22:23:35 GMT
• Title: A Coulomb Particle Model for Learning Kernel Attention in Transformers
• Abstract summary: Randomized features provide a scalable approximation to kernel machines.<n>We propose a particle-based method that learns this distribution by optimizing kernel-target alignment.<n>We instantiate the method in linearized Transformer attention by learning positive random-feature maps in a first alignment phase.
• Score: 1.7685947618629572
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Randomized features provide a scalable approximation to kernel machines, but their performance depends strongly on the choice of feature distribution. We propose a particle-based method that learns this distribution by optimizing kernel-target alignment while regularizing particles with a Riesz/Coulomb repulsive potential. The resulting Hamiltonian yields diverse, task-adaptive random features and admits a mean-field description through a McKean--Vlasov equation. We instantiate the method in linearized Transformer attention by learning positive random-feature maps in a first alignment phase, then freezing the kernel and training the remaining network parameters with cross-entropy. Experiments on synthetic classification and sentence-level benchmarks show that learned kernelized attention can improve accuracy, calibration, and robustness for several feature maps while preserving linear-attention inference complexity.

Related papers

• On Interpolation Formulas Describing Neural Network Generalization [8.154667543705687]
  We introduce a gradient kernel that extends the deterministic version via a continuous-time diffusion.<n>It reveals that training samples contribute through loss-dependent weights and gradient alignment along the trajectory.<n>We visualize then link the generalization error to the null space of the integral operator induced by the gradient kernel.
  arXiv  Detail & Related papers  (2026-03-14T10:03:29Z)
• Data-Aware Random Feature Kernel for Transformers [7.455696124254561]
  In pretrained models, queries and keys are typically anisotropic. This induces high Monte Carlo variance in isotropic sampling schemes.<n>We introduce DARKFormer, a Data-Aware Random-feature Kernel transformer that features a data-aligned kernel geometry.<n>By combining random-feature efficiency with data-aware kernels, DARKFormer advances kernel-based attention in resource-constrained settings.
  arXiv  Detail & Related papers  (2026-03-04T14:43:11Z)
• Feature maps for the Laplacian kernel and its generalizations [3.671202973761375]
  Unlike the Gaussian kernel, the Laplacian kernel is not separable.<n>We provide random features for the Laplacian kernel and its two generalizations.<n>We demonstrate the efficacy of these random feature maps on real datasets.
  arXiv  Detail & Related papers  (2025-02-21T16:36:20Z)
• Variance-Reducing Couplings for Random Features [57.73648780299374]
  Random features (RFs) are a popular technique to scale up kernel methods in machine learning. We find couplings to improve RFs defined on both Euclidean and discrete input spaces. We reach surprising conclusions about the benefits and limitations of variance reduction as a paradigm.
  arXiv  Detail & Related papers  (2024-05-26T12:25:09Z)
• Promises and Pitfalls of the Linearized Laplace in Bayesian Optimization [73.80101701431103]
  The linearized-Laplace approximation (LLA) has been shown to be effective and efficient in constructing Bayesian neural networks. We study the usefulness of the LLA in Bayesian optimization and highlight its strong performance and flexibility.
  arXiv  Detail & Related papers  (2023-04-17T14:23:43Z)
• Variational Autoencoder Kernel Interpretation and Selection for Classification [59.30734371401315]
  This work proposed kernel selection approaches for probabilistic classifiers based on features produced by the convolutional encoder of a variational autoencoder. In the proposed implementation, each latent variable was sampled from the distribution associated with a single kernel of the last encoder's convolution layer, as an individual distribution was created for each kernel. choosing relevant features on the sampled latent variables makes it possible to perform kernel selection, filtering the uninformative features and kernels.
  arXiv  Detail & Related papers  (2022-09-10T17:22:53Z)
• Hybrid Random Features [60.116392415715275]
  We propose a new class of random feature methods for linearizing softmax and Gaussian kernels called hybrid random features (HRFs) HRFs automatically adapt the quality of kernel estimation to provide most accurate approximation in the defined regions of interest.
  arXiv  Detail & Related papers  (2021-10-08T20:22:59Z)
• Kernel Identification Through Transformers [54.3795894579111]
  Kernel selection plays a central role in determining the performance of Gaussian Process (GP) models. This work addresses the challenge of constructing custom kernel functions for high-dimensional GP regression models. We introduce a novel approach named KITT: Kernel Identification Through Transformers.
  arXiv  Detail & Related papers  (2021-06-15T14:32:38Z)
• Scalable Variational Gaussian Processes via Harmonic Kernel Decomposition [54.07797071198249]
  We introduce a new scalable variational Gaussian process approximation which provides a high fidelity approximation while retaining general applicability. We demonstrate that, on a range of regression and classification problems, our approach can exploit input space symmetries such as translations and reflections. Notably, our approach achieves state-of-the-art results on CIFAR-10 among pure GP models.
  arXiv  Detail & Related papers  (2021-06-10T18:17:57Z)
• Symplectic Gaussian Process Regression of Hamiltonian Flow Maps [0.8029049649310213]
  We present an approach to construct appropriate and efficient emulators for Hamiltonian flow maps. Intended future applications are long-term tracing of fast charged particles in accelerators and magnetic plasma confinement.
  arXiv  Detail & Related papers  (2020-09-11T17:56:35Z)
• Scaling up Kernel Ridge Regression via Locality Sensitive Hashing [6.704115928005158]
  We introduce a weighted version of random binning features and show that the corresponding kernel function generates smooth Gaussian processes. We show that our weighted random binning features provide a spectral approximation to the corresponding kernel matrix, leading to efficient algorithms for kernel ridge regression.
  arXiv  Detail & Related papers  (2020-03-21T21:41:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.