Fast Latent Factor Analysis via a Fuzzy PID-Incorporated Stochastic Gradient Descent Algorithm

• URL: http://arxiv.org/abs/2303.03941v1
• Date: Tue, 7 Mar 2023 14:51:09 GMT
• Title: Fast Latent Factor Analysis via a Fuzzy PID-Incorporated Stochastic Gradient Descent Algorithm
• Authors: Li Jinli and Yuan Ye
• Abstract summary: A gradient descent (SGD)-based latent factor analysis model is remarkably effective in extracting valuable information from an HDI matrix. A standard SGD algorithm learns a latent factor relying on the gradient of current instance error only without considering past update information. This paper proposes a Fuzzy PID-incorporated SGD algorithm with two-fold ideas: 1) rebuilding the instance error by considering the past update information in an efficient way following the principle of PID, and 2) implementing hyper-learnings and gain adaptation following the fuzzy rules.
• Score: 1.984879854062214
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A high-dimensional and incomplete (HDI) matrix can describe the complex interactions among numerous nodes in various big data-related applications. A stochastic gradient descent (SGD)-based latent factor analysis (LFA) model is remarkably effective in extracting valuable information from an HDI matrix. However, such a model commonly encounters the problem of slow convergence because a standard SGD algorithm learns a latent factor relying on the stochastic gradient of current instance error only without considering past update information. To address this critical issue, this paper innovatively proposes a Fuzzy PID-incorporated SGD (FPS) algorithm with two-fold ideas: 1) rebuilding the instance learning error by considering the past update information in an efficient way following the principle of PID, and 2) implementing hyper-parameters and gain parameters adaptation following the fuzzy rules. With it, an FPS-incorporated LFA model is further achieved for fast processing an HDI matrix. Empirical studies on six HDI datasets demonstrate that the proposed FPS-incorporated LFA model significantly outperforms the state-of-the-art LFA models in terms of computational efficiency for predicting the missing data of an HDI matrix with competitive accuracy.

Related papers

• PSLF: A PID Controller-incorporated Second-order Latent Factor Analysis Model for Recommender System [11.650076383080526]
  A second-order-based HDI model (SLF) analysis demonstrates superior performance in graph learning, particularly for high- and incomplete factor data rates.
  arXiv  Detail & Related papers  (2024-08-31T13:01:58Z)
• Mini-Hes: A Parallelizable Second-order Latent Factor Analysis Model [8.06111903129142]
  This paper proposes a miniblock diagonal hessian-free (Mini-Hes) optimization for building an LFA model. Experiment results indicate that, with Mini-Hes, the LFA model outperforms several state-of-the-art models in addressing missing data estimation task.
  arXiv  Detail & Related papers  (2024-02-19T08:43:00Z)
• Fast Dual-Regularized Autoencoder for Sparse Biological Data [65.268245109828]
  We develop a shallow autoencoder for the dual neighborhood-regularized matrix completion problem. We demonstrate the speed and accuracy advantage of our approach over the existing state-of-the-art in predicting drug-target interactions and drug-disease associations.
  arXiv  Detail & Related papers  (2024-01-30T01:28:48Z)
• An ADRC-Incorporated Stochastic Gradient Descent Algorithm for Latent Factor Analysis [6.843073158719234]
  A gradient descent (SGD)-based latent factor analysis (LFA) model is remarkably effective in extracting valuable information from an HDI matrix. A standard SGD algorithm only considers the current learning error to compute the gradient without considering the historical and future state of the learning error. This paper innovatively proposes an ADRC-incorporated SGD (ADS) algorithm by refining the instance learning error by considering the historical and future state.
  arXiv  Detail & Related papers  (2024-01-13T08:38:54Z)
• Robust Learning with Progressive Data Expansion Against Spurious Correlation [65.83104529677234]
  We study the learning process of a two-layer nonlinear convolutional neural network in the presence of spurious features. Our analysis suggests that imbalanced data groups and easily learnable spurious features can lead to the dominance of spurious features during the learning process. We propose a new training algorithm called PDE that efficiently enhances the model's robustness for a better worst-group performance.
  arXiv  Detail & Related papers  (2023-06-08T05:44:06Z)
• To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis [50.31589712761807]
  Large language models (LLMs) are notoriously token-hungry during pre-training, and high-quality text data on the web is approaching its scaling limit for LLMs. We investigate the consequences of repeating pre-training data, revealing that the model is susceptible to overfitting. Second, we examine the key factors contributing to multi-epoch degradation, finding that significant factors include dataset size, model parameters, and training objectives.
  arXiv  Detail & Related papers  (2023-05-22T17:02:15Z)
• Faster Adaptive Federated Learning [84.38913517122619]
  Federated learning has attracted increasing attention with the emergence of distributed data. In this paper, we propose an efficient adaptive algorithm (i.e., FAFED) based on momentum-based variance reduced technique in cross-silo FL.
  arXiv  Detail & Related papers  (2022-12-02T05:07:50Z)
• An Accelerated Doubly Stochastic Gradient Method with Faster Explicit Model Identification [97.28167655721766]
  We propose a novel doubly accelerated gradient descent (ADSGD) method for sparsity regularized loss minimization problems. We first prove that ADSGD can achieve a linear convergence rate and lower overall computational complexity.
  arXiv  Detail & Related papers  (2022-08-11T22:27:22Z)
• A Nonlinear PID-Enhanced Adaptive Latent Factor Analysis Model [6.2303427193075755]
  High-dimensional and incomplete (HDI) data holds tremendous interactive information in various industrial applications. A latent factor (LF) model is remarkably effective in extracting valuable information from HDI data with decent gradient (SGD) algorithm. An SGD-based LFA model suffers from slow convergence since it only considers the current learning error.
  arXiv  Detail & Related papers  (2022-08-04T07:48:19Z)
• PI-NLF: A Proportional-Integral Approach for Non-negative Latent Factor Analysis [9.087387628717952]
  A non-negative latent factor (NLF) model performs efficient representation learning to an HDI matrix. A PI-NLF model outperforms the state-of-the-art models in both computational efficiency and estimation accuracy for missing data of an HDI matrix.
  arXiv  Detail & Related papers  (2022-05-05T12:04:52Z)
• Distributed Dynamic Safe Screening Algorithms for Sparse Regularization [73.85961005970222]
  We propose a new distributed dynamic safe screening (DDSS) method for sparsity regularized models and apply it on shared-memory and distributed-memory architecture respectively. We prove that the proposed method achieves the linear convergence rate with lower overall complexity and can eliminate almost all the inactive features in a finite number of iterations almost surely.
  arXiv  Detail & Related papers  (2022-04-23T02:45:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.