SSGD: A safe and efficient method of gradient descent

• URL: http://arxiv.org/abs/2012.02076v2
• Date: Mon, 26 Apr 2021 04:33:08 GMT
• Title: SSGD: A safe and efficient method of gradient descent
• Authors: Jinhuan Duan, Xianxian Li, Shiqi Gao, Jinyan Wang and Zili Zhong
• Abstract summary: gradient descent method plays an important role in solving various optimization problems. Super gradient descent approach to update parameters by concealing the length of gradient. Our algorithm can defend against attacks on the gradient.
• Score: 0.5099811144731619
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With the vigorous development of artificial intelligence technology, various engineering technology applications have been implemented one after another. The gradient descent method plays an important role in solving various optimization problems, due to its simple structure, good stability and easy implementation. In multi-node machine learning system, the gradients usually need to be shared. Shared gradients are generally unsafe. Attackers can obtain training data simply by knowing the gradient information. In this paper, to prevent gradient leakage while keeping the accuracy of model, we propose the super stochastic gradient descent approach to update parameters by concealing the modulus length of gradient vectors and converting it or them into a unit vector. Furthermore, we analyze the security of super stochastic gradient descent approach. Our algorithm can defend against attacks on the gradient. Experiment results show that our approach is obviously superior to prevalent gradient descent approaches in terms of accuracy, robustness, and adaptability to large-scale batches.

Related papers

• Expected Grad-CAM: Towards gradient faithfulness [7.2203673761998495]
  gradient-weighted CAM approaches still rely on vanilla gradients. Our work proposes a gradient-weighted CAM augmentation that tackles the saturation and sensitivity problem.
  arXiv  Detail & Related papers  (2024-06-03T12:40:30Z)
• How to guess a gradient [68.98681202222664]
  We show that gradients are more structured than previously thought. Exploiting this structure can significantly improve gradient-free optimization schemes. We highlight new challenges in overcoming the large gap between optimizing with exact gradients and guessing the gradients.
  arXiv  Detail & Related papers  (2023-12-07T21:40:44Z)
• Class Gradient Projection For Continual Learning [99.105266615448]
  Catastrophic forgetting is one of the most critical challenges in Continual Learning (CL) We propose Class Gradient Projection (CGP), which calculates the gradient subspace from individual classes rather than tasks.
  arXiv  Detail & Related papers  (2023-11-25T02:45:56Z)
• Neural Gradient Learning and Optimization for Oriented Point Normal Estimation [53.611206368815125]
  We propose a deep learning approach to learn gradient vectors with consistent orientation from 3D point clouds for normal estimation. We learn an angular distance field based on local plane geometry to refine the coarse gradient vectors. Our method efficiently conducts global gradient approximation while achieving better accuracy and ability generalization of local feature description.
  arXiv  Detail & Related papers  (2023-09-17T08:35:11Z)
• Sampling-based Fast Gradient Rescaling Method for Highly Transferable Adversarial Attacks [18.05924632169541]
  We propose a Sampling-based Fast Gradient Rescaling Method (S-FGRM) Specifically, we use data rescaling to substitute the sign function without extra computational cost. Our method could significantly boost the transferability of gradient-based attacks and outperform the state-of-the-art baselines.
  arXiv  Detail & Related papers  (2023-07-06T07:52:42Z)
• Gradient Correction beyond Gradient Descent [63.33439072360198]
  gradient correction is apparently the most crucial aspect for the training of a neural network. We introduce a framework (textbfGCGD) to perform gradient correction. Experiment results show that our gradient correction framework can effectively improve the gradient quality to reduce training epochs by $sim$ 20% and also improve the network performance.
  arXiv  Detail & Related papers  (2022-03-16T01:42:25Z)
• Adapting Stepsizes by Momentumized Gradients Improves Optimization and Generalization [89.66571637204012]
  textscAdaMomentum on vision, and achieves state-the-art results consistently on other tasks including language processing. textscAdaMomentum on vision, and achieves state-the-art results consistently on other tasks including language processing. textscAdaMomentum on vision, and achieves state-the-art results consistently on other tasks including language processing.
  arXiv  Detail & Related papers  (2021-06-22T03:13:23Z)
• Decreasing scaling transition from adaptive gradient descent to stochastic gradient descent [1.7874193862154875]
  We propose a decreasing scaling transition from adaptive gradient descent to gradient descent method DSTAda. Our experimental results show that DSTAda has a faster speed, higher accuracy, and better stability and robustness.
  arXiv  Detail & Related papers  (2021-06-12T11:28:58Z)
• Staircase Sign Method for Boosting Adversarial Attacks [123.19227129979943]
  Crafting adversarial examples for the transfer-based attack is challenging and remains a research hot spot. We propose a novel Staircase Sign Method (S$2$M) to alleviate this issue, thus boosting transfer-based attacks. Our method can be generally integrated into any transfer-based attacks, and the computational overhead is negligible.
  arXiv  Detail & Related papers  (2021-04-20T02:31:55Z)
• Better scalability under potentially heavy-tailed gradients [9.36599317326032]
  We study a scalable alternative to robust gradient descent (RGD) techniques that can be used when the gradients can be heavy-tailed. The core technique is simple: instead of trying to robustly aggregate gradients at each step, we choose a candidate which does not diverge too far from the majority of cheap sub-processes run for a single pass over partitioned data.
  arXiv  Detail & Related papers  (2020-06-01T08:16:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.