Information Geometry and Beta Link for Optimizing Sparse Variational Student-t Processes

• URL: http://arxiv.org/abs/2408.06699v1
• Date: Tue, 13 Aug 2024 07:53:39 GMT
• Title: Information Geometry and Beta Link for Optimizing Sparse Variational Student-t Processes
• Authors: Jian Xu, Delu Zeng, John Paisley,
• Abstract summary: Student-t Processes has been proposed to enhance computational efficiency and flexibility for real-world datasets using gradient descent. Traditional gradient descent methods like Adam may not fully exploit the parameter space geometry, potentially leading to slower convergence and suboptimal performance. We adopt natural gradient methods from information geometry for variational parameter optimization of Student-t Processes.
• Score: 6.37512592611305
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently, a sparse version of Student-t Processes, termed sparse variational Student-t Processes, has been proposed to enhance computational efficiency and flexibility for real-world datasets using stochastic gradient descent. However, traditional gradient descent methods like Adam may not fully exploit the parameter space geometry, potentially leading to slower convergence and suboptimal performance. To mitigate these issues, we adopt natural gradient methods from information geometry for variational parameter optimization of Student-t Processes. This approach leverages the curvature and structure of the parameter space, utilizing tools such as the Fisher information matrix which is linked to the Beta function in our model. This method provides robust mathematical support for the natural gradient algorithm when using Student's t-distribution as the variational distribution. Additionally, we present a mini-batch algorithm for efficiently computing natural gradients. Experimental results across four benchmark datasets demonstrate that our method consistently accelerates convergence speed.

Related papers

• Gradient-free neural topology optimization [0.0]
  gradient-free algorithms require many more iterations to converge when compared to gradient-based algorithms. This has made them unviable for topology optimization due to the high computational cost per iteration and high dimensionality of these problems. We propose a pre-trained neural reparameterization strategy that leads to at least one order of magnitude decrease in iteration count when optimizing the designs in latent space.
  arXiv  Detail & Related papers  (2024-03-07T23:00:49Z)
• ELRA: Exponential learning rate adaption gradient descent optimization method [83.88591755871734]
  We present a novel, fast (exponential rate), ab initio (hyper-free) gradient based adaption. The main idea of the method is to adapt the $alpha by situational awareness. It can be applied to problems of any dimensions n and scales only linearly.
  arXiv  Detail & Related papers  (2023-09-12T14:36:13Z)
• Stochastic Marginal Likelihood Gradients using Neural Tangent Kernels [78.6096486885658]
  We introduce lower bounds to the linearized Laplace approximation of the marginal likelihood. These bounds are amenable togradient-based optimization and allow to trade off estimation accuracy against computational complexity.
  arXiv  Detail & Related papers  (2023-06-06T19:02:57Z)
• Improving Gradient Methods via Coordinate Transformations: Applications to Quantum Machine Learning [0.0]
  Machine learning algorithms heavily rely on optimization algorithms based on gradients, such as gradient descent and alike. The overall performance is dependent on the appearance of local minima and barren plateaus, which slow-down calculations and lead to non-optimal solutions. In this paper we introduce a generic strategy to accelerate and improve the overall performance of such methods, allowing to alleviate the effect of barren plateaus and local minima.
  arXiv  Detail & Related papers  (2023-04-13T18:26:05Z)
• Natural Gradient Methods: Perspectives, Efficient-Scalable Approximations, and Analysis [0.0]
  Natural Gradient Descent is a second-degree optimization method motivated by the information geometry. It makes use of the Fisher Information Matrix instead of the Hessian which is typically used. Being a second-order method makes it infeasible to be used directly in problems with a huge number of parameters and data.
  arXiv  Detail & Related papers  (2023-03-06T04:03:56Z)
• Optimization using Parallel Gradient Evaluations on Multiple Parameters [51.64614793990665]
  We propose a first-order method for convex optimization, where gradients from multiple parameters can be used during each step of gradient descent. Our method uses gradients from multiple parameters in synergy to update these parameters together towards the optima.
  arXiv  Detail & Related papers  (2023-02-06T23:39:13Z)
• A Continuous-time Stochastic Gradient Descent Method for Continuous Data [0.0]
  We study a continuous-time variant of the gradient descent algorithm for optimization problems with continuous data. We study multiple sampling patterns for the continuous data space and allow for data simulated or streamed at runtime. We end with illustrating the applicability of the gradient process in a regression problem with noisy functional data, as well as in a physics-informed neural network.
  arXiv  Detail & Related papers  (2021-12-07T15:09:24Z)
• Learning Linearized Assignment Flows for Image Labeling [70.540936204654]
  We introduce a novel algorithm for estimating optimal parameters of linearized assignment flows for image labeling. We show how to efficiently evaluate this formula using a Krylov subspace and a low-rank approximation.
  arXiv  Detail & Related papers  (2021-08-02T13:38:09Z)
• AI-SARAH: Adaptive and Implicit Stochastic Recursive Gradient Methods [7.486132958737807]
  We present an adaptive variance reduced method with an implicit approach for adaptivity. We provide convergence guarantees for finite-sum minimization problems and show a faster convergence than SARAH can be achieved if local geometry permits. This algorithm implicitly computes step-size and efficiently estimates local Lipschitz smoothness of functions.
  arXiv  Detail & Related papers  (2021-02-19T01:17:15Z)
• Self-Tuning Stochastic Optimization with Curvature-Aware Gradient Filtering [53.523517926927894]
  We explore the use of exact per-sample Hessian-vector products and gradients to construct self-tuning quadratics. We prove that our model-based procedure converges in noisy gradient setting. This is an interesting step for constructing self-tuning quadratics.
  arXiv  Detail & Related papers  (2020-11-09T22:07:30Z)
• Variance Reduction with Sparse Gradients [82.41780420431205]
  Variance reduction methods such as SVRG and SpiderBoost use a mixture of large and small batch gradients. We introduce a new sparsity operator: The random-top-k operator. Our algorithm consistently outperforms SpiderBoost on various tasks including image classification, natural language processing, and sparse matrix factorization.
  arXiv  Detail & Related papers  (2020-01-27T08:23:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.