Eigen-Spike Emergence and Quadratic Equivalents for Conjugate Kernels on Nonlinearly Separable Data

• URL: http://arxiv.org/abs/2605.29669v1
• Date: Thu, 28 May 2026 09:32:19 GMT
• Title: Eigen-Spike Emergence and Quadratic Equivalents for Conjugate Kernels on Nonlinearly Separable Data
• Abstract summary: We consider the conjugate kernel (CK), which is the nonlinear feature map of a feedforward NN, under a canonical nonlinearly separable dataset.<n>We develop a robust quadratic equivalent to the spiked CK matrix that enables a precise analysis of emergent informative spikes.<n>We derive a precise BBP-type phase transition in which linear classification via the CK eigenvectors becomes possible.
• Score: 41.325451060228126
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent work in random matrix theory (RMT) has developed the notion of deterministic equivalents: typically linear surrogate models that approximate the spectral behavior of large nonlinear random matrices, such as nonlinear feature maps in neural networks (NNs). On the one hand, these deterministic equivalents make theoretical predictions tractable by reducing a complex model to a simpler model with properties that fall under the umbrella of classical RMT tools. However, this leaves open the question of whether this idealized linear equivalence remains meaningful when dealing with high-dimensional nonlinearly separable data, such as performing clssification on nonlinearly separable data. Motivated by this, we consider the conjugate kernel (CK), which is the nonlinear feature map of a feedforward NN, under a canonical nonlinearly separable dataset, the XOR problem; and we use the study of informative outlier eigenvalues in the CK and whether their corresponding eigenvectors asymptotically align with XOR labels as a proxy for nonlinear learnability. We develop a robust quadratic equivalent to the spiked CK matrix that enables a precise analysis of emergent informative spikes, as one modifies various knobs common in ML practice: sample complexity, signal-to-noise ratio (SNR), nonlinear activation choice, and pretrained features. In each of these scenarios, we derive a precise BBP-type phase transition in which linear classification via the CK eigenvectors becomes possible. Our analysis helps translate the power of deterministic equivalence tools in RMT to study problems of practical relevance in ML.

Related papers

• A Geometric Measure of Linear Separability for Neural Representations [12.476368894025233]
  We introduce the directional linear separability measure (LSM), a finite-sample diagnostic for one-sided affine separability.<n>For a target class A and a competing set B, LSM measures the smallest competing-sample intrusion that must remain on the target side, normalized by |A|.
  arXiv  Detail & Related papers  (2026-06-07T16:31:54Z)
• Disordered Dynamics in High Dimensions: Connections to Random Matrices and Machine Learning [52.26396748560348]
  We provide an overview of high dimensional dynamical systems driven by random matrices.<n>We focus on applications to simple models of learning and generalization in machine learning theory.
  arXiv  Detail & Related papers  (2026-01-03T00:12:32Z)
• Explicit Discovery of Nonlinear Symmetries from Dynamic Data [50.20526548924647]
  LieNLSD is the first method capable of determining the number of infinitesimal generators with nonlinear terms and their explicit expressions.<n>LieNLSD shows qualitative advantages over existing methods and improves the long rollout accuracy of neural PDE solvers by over 20%.
  arXiv  Detail & Related papers  (2025-10-02T09:54:08Z)
• A Random Matrix Analysis of In-context Memorization for Nonlinear Attention [18.90197287760915]
  We show that nonlinear Attention incurs higher memorization error than linear ridge regression on random inputs.<n>Our results reveal how nonlinearity and input structure interact with each other to govern the memorization performance of nonlinear Attention.
  arXiv  Detail & Related papers  (2025-06-23T13:56:43Z)
• Importance Sampling for Nonlinear Models [5.421981644827842]
  We introduce the concept of the adjoint operator of a nonlinear map.<n>We demonstrate that sampling based on these notions of norm and leverage scores provides approximation guarantees for the underlying nonlinear mapping.
  arXiv  Detail & Related papers  (2025-05-18T10:34:39Z)
• Gram-Schmidt Methods for Unsupervised Feature Extraction and Selection [7.373617024876725]
  We propose a Gram-Schmidt process over function spaces to detect and map out nonlinear dependencies.<n>We provide experimental results for synthetic and real-world benchmark datasets.<n>Surprisingly, our linear feature extraction algorithms are comparable and often outperform several important nonlinear feature extraction methods.
  arXiv  Detail & Related papers  (2023-11-15T21:29:57Z)
• Function Classes for Identifiable Nonlinear Independent Component Analysis [10.828616610785524]
  Unsupervised learning of latent variable models (LVMs) is widely used to represent data in machine learning. Recent work suggests that constraining the function class of such models may promote identifiability. We prove that a subclass of these transformations, conformal maps, is identifiable and provide novel theoretical results.
  arXiv  Detail & Related papers  (2022-08-12T17:58:31Z)
• Hessian Eigenspectra of More Realistic Nonlinear Models [73.31363313577941]
  We make a emphprecise characterization of the Hessian eigenspectra for a broad family of nonlinear models. Our analysis takes a step forward to identify the origin of many striking features observed in more complex machine learning models.
  arXiv  Detail & Related papers  (2021-03-02T06:59:52Z)
• Sparse Quantized Spectral Clustering [85.77233010209368]
  We exploit tools from random matrix theory to make precise statements about how the eigenspectrum of a matrix changes under such nonlinear transformations. We show that very little change occurs in the informative eigenstructure even under drastic sparsification/quantization.
  arXiv  Detail & Related papers  (2020-10-03T15:58:07Z)
• Understanding Implicit Regularization in Over-Parameterized Single Index Model [55.41685740015095]
  We design regularization-free algorithms for the high-dimensional single index model. We provide theoretical guarantees for the induced implicit regularization phenomenon.
  arXiv  Detail & Related papers  (2020-07-16T13:27:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.