Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticity

• URL: http://arxiv.org/abs/2010.11775v2
• Date: Thu, 29 Oct 2020 17:23:44 GMT
• Title: Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticity
• Abstract summary: We introduce two label-aware kernels that are each a superimposition of a label-agnostic part and a hierarchy of label-aware parts. We show that the models trained with the proposed kernels better simulate NNs in terms of generalization ability and local elasticity.
• Score: 39.401063331251066
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As a popular approach to modeling the dynamics of training overparametrized neural networks (NNs), the neural tangent kernels (NTK) are known to fall behind real-world NNs in generalization ability. This performance gap is in part due to the \textit{label agnostic} nature of the NTK, which renders the resulting kernel not as \textit{locally elastic} as NNs~\citep{he2019local}. In this paper, we introduce a novel approach from the perspective of \emph{label-awareness} to reduce this gap for the NTK. Specifically, we propose two label-aware kernels that are each a superimposition of a label-agnostic part and a hierarchy of label-aware parts with increasing complexity of label dependence, using the Hoeffding decomposition. Through both theoretical and empirical evidence, we show that the models trained with the proposed kernels better simulate NNs in terms of generalization ability and local elasticity.

Related papers

• Label-NTK Alignments and A Tighter Convergence Bound in the NTK Regime [1.8466658684464594]
  We develop sharper convergence guarantees by characterizing the interaction between data labels and the NTK eigenspectrum.<n>We derive a refined convergence bound that depends on the full spectrum and closely matches practical training dynamics.
  arXiv  Detail & Related papers  (2026-05-24T22:04:02Z)
• Kernel vs. Kernel: Exploring How the Data Structure Affects Neural Collapse [9.975341265604577]
  "Neural Collapse" is the decrease in the within class variability of the network's deepest features, dubbed as NC1. We provide a kernel-based analysis that does not suffer from this limitation. We show that the NTK does not represent more collapsed features than the NNGP for prototypical data models.
  arXiv  Detail & Related papers  (2024-06-04T08:33:56Z)
• A Unified Kernel for Neural Network Learning [4.0759204898334715]
  We present the Unified Neural Kernel (UNK), which characterizes the learning dynamics of neural networks with gradient descents. UNK maintains the limiting properties of both NNGP and NTK, exhibiting behaviors akin to NTK with a finite learning step. We also theoretically characterize the uniform tightness and learning convergence of the UNK kernel.
  arXiv  Detail & Related papers  (2024-03-26T07:55:45Z)
• Benign Overfitting in Deep Neural Networks under Lazy Training [72.28294823115502]
  We show that when the data distribution is well-separated, DNNs can achieve Bayes-optimal test error for classification. Our results indicate that interpolating with smoother functions leads to better generalization.
  arXiv  Detail & Related papers  (2023-05-30T19:37:44Z)
• Gradient Descent in Neural Networks as Sequential Learning in RKBS [63.011641517977644]
  We construct an exact power-series representation of the neural network in a finite neighborhood of the initial weights. We prove that, regardless of width, the training sequence produced by gradient descent can be exactly replicated by regularized sequential learning.
  arXiv  Detail & Related papers  (2023-02-01T03:18:07Z)
• On Feature Learning in Neural Networks with Global Convergence Guarantees [49.870593940818715]
  We study the optimization of wide neural networks (NNs) via gradient flow (GF) We show that when the input dimension is no less than the size of the training set, the training loss converges to zero at a linear rate under GF. We also show empirically that, unlike in the Neural Tangent Kernel (NTK) regime, our multi-layer model exhibits feature learning and can achieve better generalization performance than its NTK counterpart.
  arXiv  Detail & Related papers  (2022-04-22T15:56:43Z)
• Neural Structured Prediction for Inductive Node Classification [29.908759584092167]
  This paper studies node classification in the inductive setting, aiming to learn a model on labeled training graphs and generalize it to infer node labels on unlabeled test graphs. We present a new approach called the Structured Proxy Network (SPN), which combines the advantages of both worlds.
  arXiv  Detail & Related papers  (2022-04-15T15:50:27Z)
• On the Equivalence between Neural Network and Support Vector Machine [23.174679357972984]
  The dynamics of an infinitely wide neural network (NN) trained by gradient descent can be characterized by Tangent Neural Kernel (NTK) We establish the equivalence between NN and support vector machine (SVM) Our main theoretical results include establishing the equivalence between NN and a broad family of $ell$ regularized KMs with finite-width bounds.
  arXiv  Detail & Related papers  (2021-11-11T06:05:00Z)
• Uniform Generalization Bounds for Overparameterized Neural Networks [5.945320097465419]
  We prove uniform generalization bounds for overparameterized neural networks in kernel regimes. Our bounds capture the exact error rates depending on the differentiability of the activation functions. We show the equivalence between the RKHS corresponding to the NT kernel and its counterpart corresponding to the Mat'ern family of kernels.
  arXiv  Detail & Related papers  (2021-09-13T16:20:13Z)
• On the Equivalence of Decoupled Graph Convolution Network and Label Propagation [60.34028546202372]
  Some work shows that coupling is inferior to decoupling, which supports deep graph propagation better. Despite effectiveness, the working mechanisms of the decoupled GCN are not well understood. We propose a new label propagation method named propagation then training Adaptively (PTA), which overcomes the flaws of the decoupled GCN.
  arXiv  Detail & Related papers  (2020-10-23T13:57:39Z)
• Infinite attention: NNGP and NTK for deep attention networks [38.55012122588628]
  We identify an equivalence between wide neural networks (NNs) and Gaussian processes (GPs) We show that unlike single-head attention, which induces non-Gaussian behaviour, multi-head attention architectures behave as GPs as the number of heads tends to infinity. We introduce new features to the Neural Tangents library allowing applications of NNGP/NTK models, with and without attention, to variable-length sequences.
  arXiv  Detail & Related papers  (2020-06-18T13:57:01Z)
• A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks [87.23360438947114]
  We show that noisy gradient descent with weight decay can still exhibit a " Kernel-like" behavior. This implies that the training loss converges linearly up to a certain accuracy. We also establish a novel generalization error bound for two-layer neural networks trained by noisy gradient descent with weight decay.
  arXiv  Detail & Related papers  (2020-02-10T18:56:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.