Enhancing SignSGD: Small-Batch Convergence Analysis and a Hybrid Switching Strategy

• URL: http://arxiv.org/abs/2604.25550v1
• Date: Tue, 28 Apr 2026 12:15:55 GMT
• Title: Enhancing SignSGD: Small-Batch Convergence Analysis and a Hybrid Switching Strategy
• Abstract summary: SignSGD compresses each gradient coordinate to a single bit, offering substantial memory and communication savings.<n>We revisit SignSGD from a 1-bit quantization and dithering perspective and contribute three improvements.
• Score: 11.467993356247248
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: SignSGD compresses each stochastic gradient coordinate to a single bit, offering substantial memory and communication savings, but its 1-bit quantization removes magnitude information and is known to leave a generalization gap relative to well-tuned SGD. We revisit SignSGD from a 1-bit quantization and dithering perspective and contribute three improvements. First, we derive a small-batch convergence rate for SignSGD under unimodal symmetric gradient noise using a signal-to-noise weighted stationarity measure, removing the large-batch assumption of prior analyses. Second, we inject annealed Gaussian noise before the sign operator, which acts as a classical dithering mechanism and probabilistically restores magnitude information lost to hard thresholding. Third, we adapt the SWATS strategy to sign-based updates with a projection-based learning-rate calibration that smoothly transitions from SignSGD to SGD. Single-worker experiments on ResNet-18 isolate optimizer effects from communication aspects: pre-sign dithering surpasses Adam on CIFAR-100, and the calibrated switch reaches 92.18% test accuracy on CIFAR-10, outperforming both pure SGD 91.38% and pure SignSGD with momentum 90.82%.

Related papers

• Spectral Gradient Orthogonalization Improves Differentially Private Training at Scale [17.610398782149293]
  In vision models, where spatial correlation concentrates gradient energy into a low-rank subspace, most of this noise falls in directions that carry little signal.<n> Spectral gradient orthogonalization via polar decomposition is introduced as a post-processing step.<n>It improves accuracy only when the per-direction spectral signal-to-noise ratio (SNR) suffices for singular vector recovery.
  arXiv  Detail & Related papers  (2026-08-18T06:35:26Z)
• Prototype-Rectified Iterative Self-supervised Manifold Denoising under Severe Acoustic Shift [3.704942101286518]
  We introduce PRISM, a training-free, source-free Test-Time Adaptation (TTA) framework grounded in the Affine Noise Hypothesis.<n>At inference, adaptation reduces to one matrix-vector multiplication in 0.0009 ms, making it substantially faster than gradient-based TTA.<n>On UrbanSound8K, PRISM improves over the zero-shot baseline by 12.94 percentage points and surpasses an oracle-assisted TTA baseline by 9.41 percentage points.
  arXiv  Detail & Related papers  (2026-08-15T04:42:14Z)
• BGA: A noise-immune neural distillation framework for malicious signature extraction in high-entropy encrypted flows [12.604432348059133]
  We propose BGA, a noise-immune neural distillation framework for encrypted threat intelligence.<n>The architecture integrates Bidirectional Long-Term Memory (BiLSTM) for temporal dependency extraction and an Adaptive Gated Multi-Head Attention mechanism.<n>Extensive evaluations on CIC-IDS-2018 and Edge-IIoT benchmarks demonstrate a performance ceiling exceeding 9 latency across all key metrics.
  arXiv  Detail & Related papers  (2026-08-14T09:33:16Z)
• StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models [16.690425653502256]
  Sign-based optimization algorithms, such as SignSGD, have garnered significant attention for their remarkable performance in distributed learning and training large foundation models.<n>Despite their empirical superiority, SignSGD is known to diverge on non-text-smooth objectives.<n>We propose bfStoSignSGD, an algorithm that injects structurality into the sign operator while maintaining an unbiased update step.
  arXiv  Detail & Related papers  (2026-04-16T17:55:36Z)
• Scaling Laws of SignSGD in Linear Regression: When Does It Outperform SGD? [35.79321975718977]
  We study scaling laws of signSGD under a power-law random features (PLRF) model.<n>We analyze the population risk of a linear model trained with one-pass signSGD on Gaussian-sketched features.
  arXiv  Detail & Related papers  (2026-03-02T16:58:02Z)
• Denoising-Enhanced YOLO for Robust SAR Ship Detection [9.818917054838964]
  CPN-YOLO is a high-precision ship detection framework built upon YOLOv8.<n>We introduce a learnable large- kernel denoising module for input pre-processing.<n>Second, we design a feature extraction enhancement strategy based on the attention mechanism to strengthen multi-scale modeling.
  arXiv  Detail & Related papers  (2026-02-27T09:00:19Z)
• Multi-Granular Discretization for Interpretable Generalization in Precise Cyberattack Identification [0.0]
  Interpretable Generalization (IG) mechanism is used to learn coherent patterns.<n>IG-MD represents every continuous feature at several Gaussian-based resolutions.<n>On UKM-IDS20, IG-MD lifts precision by greater than or equal to 4 percentage points across all nine train-test splits.
  arXiv  Detail & Related papers  (2025-07-16T12:57:38Z)
• Pushing the Limits of Low-Bit Optimizers: A Focus on EMA Dynamics [64.62231094774211]
  Statefuls (e.g., Adam) maintain auxiliary information even 2x the model size in order to achieve optimal convergence.<n>SOLO enables Adam-styles to maintain quantized states with precision as low as 3 bits, or even 2 bits.<n>SOLO can thus be seamlessly applied to Adam-styles, leading to substantial memory savings with minimal accuracy loss.
  arXiv  Detail & Related papers  (2025-05-01T06:47:45Z)
• Bias-Aware Minimisation: Understanding and Mitigating Estimator Bias in Private SGD [56.01810892677744]
  We show a connection between per-sample gradient norms and the estimation bias of the private gradient oracle used in DP-SGD. We propose Bias-Aware Minimisation (BAM) that allows for the provable reduction of private gradient estimator bias.
  arXiv  Detail & Related papers  (2023-08-23T09:20:41Z)
• Patch-Level Contrasting without Patch Correspondence for Accurate and Dense Contrastive Representation Learning [79.43940012723539]
  ADCLR is a self-supervised learning framework for learning accurate and dense vision representation. Our approach achieves new state-of-the-art performance for contrastive methods.
  arXiv  Detail & Related papers  (2023-06-23T07:38:09Z)
• Decentralized SGD and Average-direction SAM are Asymptotically Equivalent [101.37242096601315]
  Decentralized gradient descent (D-SGD) allows collaborative learning on massive devices simultaneously without the control of a central server. Existing theories claim that decentralization invariably generalization.
  arXiv  Detail & Related papers  (2023-06-05T14:19:52Z)
• Smoothed Contrastive Learning for Unsupervised Sentence Embedding [41.09180639504244]
  We introduce a smoothing strategy upon the InfoNCE loss function, termedGaussian Smoothing InfoNCE (GS-InfoNCE) GS-InfoNCE outperforms the state-of-the-art unsup-SimCSE by an average Spear-man correlation of 1.38%, 0.72%, 1.17% and 0.28% on the base of BERT-base, BERT-large,RoBERTa-base and RoBERTa-large, respectively.
  arXiv  Detail & Related papers  (2021-09-09T14:54:24Z)
• Breaking (Global) Barriers in Parallel Stochastic Optimization with Wait-Avoiding Group Averaging [48.99717153937717]
  We present WAGMA-SGD, a wait-avoiding subgroup that reduces global communication via weight exchange.<n>We train ResNet-50 on ImageNet; Transformer for machine translation; and deep reinforcement learning for navigation at scale.<n>Compared with state-of-the-art decentralized SGD variants, WAGMA-SGD significantly improves training throughput.
  arXiv  Detail & Related papers  (2020-04-30T22:11:53Z)
• Detached Error Feedback for Distributed SGD with Random Sparsification [98.98236187442258]
  Communication bottleneck has been a critical problem in large-scale deep learning. We propose a new distributed error feedback (DEF) algorithm, which shows better convergence than error feedback for non-efficient distributed problems. We also propose DEFA to accelerate the generalization of DEF, which shows better bounds than DEF.
  arXiv  Detail & Related papers  (2020-04-11T03:50:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.