Spectrally Deconfounded Gradient Boosting

• URL: http://arxiv.org/abs/2607.09371v1
• Date: Fri, 10 Jul 2026 12:51:42 GMT
• Title: Spectrally Deconfounded Gradient Boosting
• Abstract summary: We develop a nonlinear spectral deconfounding framework for gradient boosting.<n>We show that deconfounding is not achieved by the spectral loss alone.<n>We also extend the method to general likelihoods and nonlinear confounding.
• Score: 10.803532361460741
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Flexible machine-learning methods can be sensitive to hidden confounding: they may learn associations induced by unobserved confounders rather than stable signals. Spectral deconfounding mitigates this problem by shrinking high-variance directions of the covariate matrix that, under dense confounding, carry latent confounder information. Existing work has largely focused on linear models. We develop a nonlinear spectral deconfounding framework for gradient boosting. Our approach replaces the ordinary squared-error loss by a spectral loss, which alters the boosting dynamics by slowing down learning in confounding-aligned directions. We show that deconfounding is not achieved by the spectral loss alone, but by the interaction between spectral shrinkage and regularization, especially in terms of early stopping. Moreover, we provide a mixed-model interpretation that connects LAVA-type shrinkage to random-effects adjustment and yields an empirical-Bayes procedure for tuning the spectral loss. We also extend the method to general likelihoods and nonlinear confounding using Laplace approximations and kernel random effects. Across synthetic and real-world experiments, spectrally deconfounded boosting improves estimation of the target function under hidden confounding and is substantially more scalable than existing nonlinear spectral deconfounding baselines.

Related papers

• Revisiting Spectral Representations in Generative Diffusion Models [98.43552218868837]
  Diffusion models have shown remarkable performance on diverse generation tasks.<n>Recent work finds that imposing representation alignment on the hidden states of diffusion networks can both facilitate training convergence and enhance sampling quality.<n>We investigate the connection between self-supervised spectral representation learning and diffusion generative models through a shared perspective on perturbation kernels.
  arXiv  Detail & Related papers  (2026-09-08T04:54:02Z)
• Cone Extended Rayleigh Quotients for Directed Graph Learning: Minimax Spectral Certificates, Sensitivity, and Adaptive Control [0.0]
  We develop a learning-oriented methodology for spectral certification, sensitivity analysis, and control without requiring symmetry, nonnegativity, or cone preservation.<n>For a simple interior level, the right and left modes satisfy [ D_C(B)[H]=v_CT H u_C, ] yielding first-order optimal graph-supported interventions.<n> Numerical experiments demonstrate the applicability of the approach beyond cone-preserving operators and in directed learning settings.
  arXiv  Detail & Related papers  (2026-08-27T13:37:02Z)
• Robust Graph Representation Learning via Adaptive Spectral Contrast [0.9558392439655014]
  High-frequency signals are indispensable for encoding heterophily but exhibit significantly higher variance under spectrally concentrated perturbations.<n>We propose ASPECT, a framework that resolves this dilemma through a reliability-aware spectral gating mechanism.<n>ASPECT achieves new state-of-the-art performance on 8 out of 9 benchmarks, effectively decoupling meaningful structural heterophily from incidental noise.
  arXiv  Detail & Related papers  (2026-04-02T10:38:00Z)
• MirrorLA: Reflecting Feature Map for Vision Linear Attention [49.41670925034762]
  Linear attention significantly reduces the computational complexity of Transformers from quadratic to linear, yet it consistently lags behind softmax-based attention in performance.<n>We propose MirrorLA, a geometric framework that substitutes passive truncation with active reorientation.<n>MirrorLA achieves state-of-the-art performance across standard benchmarks, demonstrating that strictly linear efficiency can be achieved without compromising representational fidelity.
  arXiv  Detail & Related papers  (2026-02-04T09:14:09Z)
• On the Spectral Flattening of Quantized Embeddings [25.64641307046705]
  Training Large Language Models at ultra-low precision is critically impeded by instability rooted in the conflict between discrete quantization constraints and the intrinsic heavy-tailed spectral nature of linguistic data.<n>This work not only quantifies the spectral sensitivity of LLMs but also establishes spectral fidelity as a necessary condition for stable low-bit optimization.
  arXiv  Detail & Related papers  (2026-02-01T02:21:53Z)
• Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrieval [13.218607858857295]
  Spectral gradient methods modify gradient updates by preserving directional information while discarding scale.<n>We investigate the mechanisms underlying these gains through a dynamical analysis of a nonlinear phase retrieval model.
  arXiv  Detail & Related papers  (2026-01-30T07:12:58Z)
• Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbations [57.179679246370114]
  We identify the distribution of random perturbations that minimizes the estimator's variance as the perturbation stepsize tends to zero.<n>Our findings reveal that such desired perturbations can align directionally with the true gradient, instead of maintaining a fixed length.
  arXiv  Detail & Related papers  (2025-10-22T19:06:39Z)
• Holistic Physics Solver: Learning PDEs in a Unified Spectral-Physical Space [54.13671100638092]
  Holistic Physics Mixer (HPM) is a framework for integrating spectral and physical information in a unified space.<n>We show that HPM consistently outperforms state-of-the-art methods in both accuracy and computational efficiency.
  arXiv  Detail & Related papers  (2024-10-15T08:19:39Z)
• Experimental emulator of pulse dynamics in fractional nonlinear Schrödinger equation [6.084996921308185]
  We present a nonlinear optical platform that supports the pulse propagation governed by a generalized fractional nonlinear Schr"odinger equation (FNLSE)<n>Our approach distinguishes between intra-cavity and extra-cavity regimes, exploring the interplay between the effective fractional group-velocity dispersion (FGVD) and Kerr nonlinearity.<n>These findings open new avenues for experimental studies of spectral-temporal dynamics in fractional nonlinear systems.
  arXiv  Detail & Related papers  (2023-11-26T00:34:15Z)
• Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion [83.90492831583997]
  We show that a batch-normalized network can keep the optimal signal propagation properties, but avoid exploding gradients in depth. We use a Multi-Layer Perceptron (MLP) with linear activations and batch-normalization that provably has bounded depth. We also design an activation shaping scheme that empirically achieves the same properties for certain non-linear activations.
  arXiv  Detail & Related papers  (2023-10-03T12:35:02Z)
• Spectral Feature Augmentation for Graph Contrastive Learning and Beyond [64.78221638149276]
  We present a novel spectral feature argumentation for contrastive learning on graphs (and images) For each data view, we estimate a low-rank approximation per feature map and subtract that approximation from the map to obtain its complement. This is achieved by the proposed herein incomplete power iteration, a non-standard power regime which enjoys two valuable byproducts (under mere one or two iterations) Experiments on graph/image datasets show that our spectral feature augmentation outperforms baselines.
  arXiv  Detail & Related papers  (2022-12-02T08:48:11Z)
• Robust, Nonparametric, Efficient Decomposition of Spectral Peaks under Distortion and Interference [0.0]
  We propose a decomposition method for the spectral peaks in an observed frequency spectrum, which is efficiently acquired by utilizing the Fast Fourier Transform. We model the peaks in spectrum as pseudo-symmetric functions, where the only constraint is a nonincreasing behavior around a central frequency when the distance increases. Our approach is more robust against arbitrary distortion, interference and noise on the spectrum that may be caused by an observation system.
  arXiv  Detail & Related papers  (2022-04-18T17:08:37Z)
• Sparse Quantized Spectral Clustering [85.77233010209368]
  We exploit tools from random matrix theory to make precise statements about how the eigenspectrum of a matrix changes under such nonlinear transformations. We show that very little change occurs in the informative eigenstructure even under drastic sparsification/quantization.
  arXiv  Detail & Related papers  (2020-10-03T15:58:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.