How Deep Are Deep GPs, Really? A Sharp Threshold and a Non-Gaussian Limit for Compositional GPs

• URL: http://arxiv.org/abs/2606.08218v1
• Date: Sat, 06 Jun 2026 15:12:43 GMT
• Title: How Deep Are Deep GPs, Really? A Sharp Threshold and a Non-Gaussian Limit for Compositional GPs
• Abstract summary: Previous work has established that for the RBF kernel and a certain range of bandwidths $r$, the prior degenerates in the limit.<n>We prove that for $r$ below the threshold $r_c(d)$ the prior converges to a limit distribution $_barZ$.
• Score: 48.096969031315744
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Compositional priors describe the generic properties of layered functions in deep Bayesian models, where deep neural networks with random weights are a canonical example.In the wide-network limit, the prior is a Gaussian process with a depth-dependent kernel, and its behaviour as depth grows has been extensively studied through this kernel. Here, we study another case, where each layer itself is a vector valued Gaussian process, and our aim is similarly to understand the limiting behaviour of the prior as depth grows. Previous GP work has established that for the RBF kernel and a certain range of bandwidths $r$, the prior degenerates in the limit, converging to the set of constant functions -- which is not useful as a probabilistic model. In this paper we establish several new results. First, we identify a sharp bandwidth threshold $r_c(d) = Θ(\sqrt{d})$ above which the limit is degenerate, strengthening the earlier bounds. Second, and more importantly, we show that for $r$ below the threshold $r_c(d)$ the prior converges to a limit distribution $π_{\bar{Z}}$. We also prove that these distributions are non-degenerate and non-Gaussian, with non-vanishing dependence between coordinates. In contrast to the previously known degenerate regime, deep Gaussian process priors can therefore admit non-trivial limits. Empirically, we verify the threshold across a range of dimensions $d$, and demonstrate a complex multimodal behaviour of the limit distributions $π_{\bar{Z}}$ -- a regime that becomes increasingly narrow with $d$ and would be hard to identify without knowing the threshold.

Related papers

• Pointwise Complexity for Gaussian Fields: Upper Envelopes, Algorithmic Lower Bounds, and Separation [5.082462420126421]
  We prove a variance-aware pointwise majorizing-measure theorem for centered Gaussian processes.<n>The theorem provides a reusable field-level refinement of classical generic chaining.<n>We show that algorithmic lower bounds provide local-geometric certificates of pointwise complexity for fixed estimators.
  arXiv  Detail & Related papers  (2026-06-06T01:50:06Z)
• Provably Adaptive Linear Approximation for the Shapley Value and Beyond [73.0940890296463]
  A fundamental and long-standing challenge is their efficient approximation.<n>We develop a linear-space algorithm that requires $P(|hatboldsymbol-boldsymbol|_2geq)leq $ for all commonly used semi-values.<n>Our algorithm allows explicit minimization of the mean square error for each specific utility function.
  arXiv  Detail & Related papers  (2026-04-09T16:38:14Z)
• Hardness of High-Dimensional Linear Classification [58.29089693778071]
  We establish new exponential in dimension lower bounds for the Maximum Halfspace Discrepancy problem.<n>Both are fundamental problems in computational geometry and machine learning in their exact and approximate forms.
  arXiv  Detail & Related papers  (2026-03-19T15:53:41Z)
• Finite-Dimensional Gaussian Approximation for Deep Neural Networks: Universality in Random Weights [15.424946932398713]
  We study the Finite-Dimensional Distributions (FDDs) of deep neural networks with randomly weights that have finite-order moments.<n>We establish Gaussian approximation bounds in the Wasserstein-$1$ norm between the FDDs and their Gaussian limit.<n>In the special case where all widths are proportional to a common scale parameter $n$ and there are $L-1$ hidden layers, we obtain convergence rates of order $n-(1/6)L-1 + epsilon$, for any $epsilon > 0$.
  arXiv  Detail & Related papers  (2025-07-16T23:41:09Z)
• Infinite-Width Limit of a Single Attention Layer: Analysis via Tensor Programs [15.559242245579382]
  In this paper, we identify the infinite-width limit distribution of variables within a single attention layer under realistic architectural dimensionality.<n>We derive the exact form of this limit law without resorting to infinite-head approximations or tailored scalings.<n>Our findings lay the groundwork for developing a unified theory of deep Transformer architectures in the infinite-width regime.
  arXiv  Detail & Related papers  (2025-06-01T05:53:47Z)
• Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks [54.177130905659155]
  Recent studies show that a reproducing kernel Hilbert space (RKHS) is not a suitable space to model functions by neural networks. In this paper, we study a suitable function space for over- parameterized two-layer neural networks with bounded norms.
  arXiv  Detail & Related papers  (2024-04-29T15:04:07Z)
• Neural signature kernels as infinite-width-depth-limits of controlled ResNets [5.306881553301636]
  We consider randomly controlled ResNets defined as Euler-discretizations of neural controlled differential equations (Neural CDEs) We show that in the infinite-width-depth limit and under proper scaling, these architectures converge weakly to Gaussian processes indexed on some spaces of continuous paths. We show that in the infinite-depth regime, finite-width controlled ResNets converge in distribution to Neural CDEs with random vector fields.
  arXiv  Detail & Related papers  (2023-03-30T19:20:16Z)
• A theory of representation learning gives a deep generalisation of kernel methods [22.260038428890383]
  We develop a new infinite width limit, the Bayesian representation learning limit. We show that it exhibits representation learning mirroring that in finite-width models. Next, we introduce the possibility of using this limit and objective as a flexible, deep generalisation of kernel methods.
  arXiv  Detail & Related papers  (2021-08-30T10:07:37Z)
• Large-width functional asymptotics for deep Gaussian neural networks [2.7561479348365734]
  We consider fully connected feed-forward deep neural networks where weights and biases are independent and identically distributed according to Gaussian distributions. Our results contribute to recent theoretical studies on the interplay between infinitely wide deep neural networks and processes.
  arXiv  Detail & Related papers  (2021-02-20T10:14:37Z)
• Bayesian Deep Ensembles via the Neural Tangent Kernel [49.569912265882124]
  We explore the link between deep ensembles and Gaussian processes (GPs) through the lens of the Neural Tangent Kernel (NTK) We introduce a simple modification to standard deep ensembles training, through addition of a computationally-tractable, randomised and untrainable function to each ensemble member. We prove that our Bayesian deep ensembles make more conservative predictions than standard deep ensembles in the infinite width limit.
  arXiv  Detail & Related papers  (2020-07-11T22:10:52Z)
• Differentially Quantized Gradient Methods [53.3186247068836]
  We show that Differentially Quantized Gradient Descent (DQ-GD) attains a linear contraction factor of $maxsigma_mathrmGD, rhon 2-R$. No algorithm within a certain class can converge faster than $maxsigma_mathrmGD, 2-R$.
  arXiv  Detail & Related papers  (2020-02-06T20:40:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.