Spectral Algorithms under Covariate Shift

• URL: http://arxiv.org/abs/2504.12625v1
• Date: Thu, 17 Apr 2025 04:02:06 GMT
• Title: Spectral Algorithms under Covariate Shift
• Authors: Jun Fan, Zheng-Chu Guo, Lei Shi,
• Abstract summary: Spectral algorithms leverage spectral regularization techniques to analyze and process data.<n>We investigate the convergence behavior of spectral algorithms under distribution shifts.<n>We propose a weighted spectral algorithm that incorporates density ratio information into the learning process.
• Score: 4.349399061959293
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spectral algorithms leverage spectral regularization techniques to analyze and process data, providing a flexible framework for addressing supervised learning problems. To deepen our understanding of their performance in real-world scenarios where the distributions of training and test data may differ, we conduct a rigorous investigation into the convergence behavior of spectral algorithms under distribution shifts, specifically within the framework of reproducing kernel Hilbert spaces. Our study focuses on the case of covariate shift. In this scenario, the marginal distributions of the input data differ between the training and test datasets, while the conditional distribution of the output given the input remains unchanged. Under this setting, we analyze the generalization error of spectral algorithms and show that they achieve minimax optimality when the density ratios between the training and test distributions are uniformly bounded. However, we also identify a critical limitation: when the density ratios are unbounded, the spectral algorithms may become suboptimal. To address this limitation, we propose a weighted spectral algorithm that incorporates density ratio information into the learning process. Our theoretical analysis shows that this weighted approach achieves optimal capacity-independent convergence rates. Furthermore, by introducing a weight clipping technique, we demonstrate that the convergence rates of the weighted spectral algorithm can approach the optimal capacity-dependent convergence rates arbitrarily closely. This improvement resolves the suboptimality issue in unbounded density ratio scenarios and advances the state-of-the-art by refining existing theoretical results.

Related papers

• A Bias-Correction Decentralized Stochastic Gradient Algorithm with Momentum Acceleration [19.83835152405735]
  We propose a momentum-celerated distributed gradient, termed Exact-Diffusion with Momentum (EDM)<n>EDM mitigates the bias from data heterogeneity and incorporates momentum techniques commonly used in deep learning.<n>Our theoretical analysis demonstrates that the EDM algorithm converges sublinearly to the neighborhood optimal solution.
  arXiv  Detail & Related papers  (2025-01-31T12:15:58Z)
• Rethinking Clustered Federated Learning in NOMA Enhanced Wireless Networks [60.09912912343705]
  This study explores the benefits of integrating the novel clustered federated learning (CFL) approach with non-independent and identically distributed (non-IID) datasets. A detailed theoretical analysis of the generalization gap that measures the degree of non-IID in the data distribution is presented. Solutions to address the challenges posed by non-IID conditions are proposed with the analysis of the properties.
  arXiv  Detail & Related papers  (2024-03-05T17:49:09Z)
• Distributed Markov Chain Monte Carlo Sampling based on the Alternating Direction Method of Multipliers [143.6249073384419]
  In this paper, we propose a distributed sampling scheme based on the alternating direction method of multipliers. We provide both theoretical guarantees of our algorithm's convergence and experimental evidence of its superiority to the state-of-the-art. In simulation, we deploy our algorithm on linear and logistic regression tasks and illustrate its fast convergence compared to existing gradient-based methods.
  arXiv  Detail & Related papers  (2024-01-29T02:08:40Z)
• Integral Operator Approaches for Scattered Data Fitting on Spheres [16.389581549801253]
  We study the approximation performance of a class of weighted spectral filter algorithms. We derive optimal Sobolev-type error estimates of weighted spectral filter algorithms.
  arXiv  Detail & Related papers  (2024-01-27T04:42:50Z)
• Spectral Decomposition Representation for Reinforcement Learning [100.0424588013549]
  We propose an alternative spectral method, Spectral Decomposition Representation (SPEDER), that extracts a state-action abstraction from the dynamics without inducing spurious dependence on the data collection policy. A theoretical analysis establishes the sample efficiency of the proposed algorithm in both the online and offline settings. An experimental investigation demonstrates superior performance over current state-of-the-art algorithms across several benchmarks.
  arXiv  Detail & Related papers  (2022-08-19T19:01:30Z)
• Understanding the Generalization Performance of Spectral Clustering Algorithms [11.025579607812167]
  We study the excess risk bounds of the popular spectral clustering algorithms: emphrelaxed RatioCut and emphrelaxed NCut. We propose two novel algorithms that can not only penalize this quantity, but also cluster the out-of-sample data without re-eigendecomposition on the overall sample.
  arXiv  Detail & Related papers  (2022-04-30T14:21:56Z)
• Amortized Implicit Differentiation for Stochastic Bilevel Optimization [53.12363770169761]
  We study a class of algorithms for solving bilevel optimization problems in both deterministic and deterministic settings. We exploit a warm-start strategy to amortize the estimation of the exact gradient. By using this framework, our analysis shows these algorithms to match the computational complexity of methods that have access to an unbiased estimate of the gradient.
  arXiv  Detail & Related papers  (2021-11-29T15:10:09Z)
• Optimizing Information-theoretical Generalization Bounds via Anisotropic Noise in SGLD [73.55632827932101]
  We optimize the information-theoretical generalization bound by manipulating the noise structure in SGLD. We prove that with constraint to guarantee low empirical risk, the optimal noise covariance is the square root of the expected gradient covariance.
  arXiv  Detail & Related papers  (2021-10-26T15:02:27Z)
• Consistency of Anchor-based Spectral Clustering [0.0]
  Anchor-based techniques reduce the computational complexity of spectral clustering algorithms. We show that it is amenable to rigorous analysis, as well as being effective in practice. We find that it is competitive with the state-of-the-art LSC method of Chen and Cai.
  arXiv  Detail & Related papers  (2020-06-24T18:34:41Z)
• A Distributional Analysis of Sampling-Based Reinforcement Learning Algorithms [67.67377846416106]
  We present a distributional approach to theoretical analyses of reinforcement learning algorithms for constant step-sizes. We show that value-based methods such as TD($lambda$) and $Q$-Learning have update rules which are contractive in the space of distributions of functions.
  arXiv  Detail & Related papers  (2020-03-27T05:13:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.