Stable Anderson Acceleration for Deep Learning

• URL: http://arxiv.org/abs/2110.14813v1
• Date: Tue, 26 Oct 2021 14:52:16 GMT
• Title: Stable Anderson Acceleration for Deep Learning
• Abstract summary: Anderson acceleration (AA) is designed to speed-up fixed-point iterations like those arising from the iterative training of DL models. We combine AA with an adaptive moving average procedure that smoothes the oscillations and results in a more regular sequence of descent updates.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Anderson acceleration (AA) is an extrapolation technique designed to speed-up fixed-point iterations like those arising from the iterative training of DL models. Training DL models requires large datasets processed in randomly sampled batches that tend to introduce in the fixed-point iteration stochastic oscillations of amplitude roughly inversely proportional to the size of the batch. These oscillations reduce and occasionally eliminate the positive effect of AA. To restore AA's advantage, we combine it with an adaptive moving average procedure that smoothes the oscillations and results in a more regular sequence of gradient descent updates. By monitoring the relative standard deviation between consecutive iterations, we also introduce a criterion to automatically assess whether the moving average is needed. We applied the method to the following DL instantiations: (i) multi-layer perceptrons (MLPs) trained on the open-source graduate admissions dataset for regression, (ii) physics informed neural networks (PINNs) trained on source data to solve 2d and 100d Burgers' partial differential equations (PDEs), and (iii) ResNet50 trained on the open-source ImageNet1k dataset for image classification. Numerical results obtained using up to 1,536 NVIDIA V100 GPUs on the OLCF supercomputer Summit showed the stabilizing effect of the moving average on AA for all the problems above.

Related papers

• Accelerating Diffusion Transformers with Gaussian Process Rectified Feature Cache [17.39547918764814]
  Diffusion Transformers have become the dominant paradigm in generative AI, but their high computational costs hinder real-time applications.<n>Prediction-based feature caching is widely used to accelerate diffusion transformers; however, as the number of steps increases, the deviation between its predictions and the reference full-compute trajectory gradually grows.<n>An intuitive idea is to use an online regression model to dynamically correct this deviation, but it faces the issue of label data being unavailable during the acceleration process.<n>This paper presents a statistical observation that the residuals between the features of full computation steps using caching methods and reference full-compute trajectory locally exhibit a
  arXiv  Detail & Related papers  (2026-09-05T08:54:54Z)
• Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Models [37.194177406897964]
  We propose a closed-form formulation that computes the linear probe induced by the synthetic set with a sample-space kernel ridge solver.<n>On ImageNet-100 with four pre-trained backbones, CLP-DD substantially improves over LGM without DSA and approaches LGM with DSA at a fraction of the computational cost.<n>On ImageNet-1K, CLP-DD matches or surpasses LGM with DSA on three of four backbones while running roughly $14times$ faster and using less than one-eighth of the GPU memory.
  arXiv  Detail & Related papers  (2026-05-08T03:41:51Z)
• iGaussian: Real-Time Camera Pose Estimation via Feed-Forward 3D Gaussian Splatting Inversion [62.09575122593993]
  iGaussian is a two-stage feed-forward framework that achieves real-time camera pose estimation through direct 3D Gaussian inversion.<n> Experimental results on the NeRF Synthetic, Mip-NeRF 360, and T&T+DB datasets demonstrate a significant performance improvement over previous methods.
  arXiv  Detail & Related papers  (2025-11-18T05:22:22Z)
• PreLoRA: Hybrid Pre-training of Vision Transformers with Full Training and Low-Rank Adapters [2.5547655072779]
  We propose an approach to identify states of partial convergence and switch from full parameter training to Low-Rank Adaptation (LoRA) on the ViT-Large model.<n> Experimental results show that this approach preserves model accuracy while reducing the number of trainable parameters to 10% of its original size.
  arXiv  Detail & Related papers  (2025-09-25T21:34:17Z)
• Efficient Federated Learning with Heterogeneous Data and Adaptive Dropout [62.73150122809138]
  Federated Learning (FL) is a promising distributed machine learning approach that enables collaborative training of a global model using multiple edge devices.<n>We propose the FedDHAD FL framework, which comes with two novel methods: Dynamic Heterogeneous model aggregation (FedDH) and Adaptive Dropout (FedAD)<n>The combination of these two methods makes FedDHAD significantly outperform state-of-the-art solutions in terms of accuracy (up to 6.7% higher), efficiency (up to 2.02 times faster), and cost (up to 15.0% smaller)
  arXiv  Detail & Related papers  (2025-07-14T16:19:00Z)
• Ambient Diffusion Posterior Sampling: Solving Inverse Problems with Diffusion Models trained on Corrupted Data [56.81246107125692]
  Ambient Diffusion Posterior Sampling (A-DPS) is a generative model pre-trained on one type of corruption. We show that A-DPS can sometimes outperform models trained on clean data for several image restoration tasks in both speed and performance. We extend the Ambient Diffusion framework to train MRI models with access only to Fourier subsampled multi-coil MRI measurements.
  arXiv  Detail & Related papers  (2024-03-13T17:28:20Z)
• Adaptive Federated Learning Over the Air [108.62635460744109]
  We propose a federated version of adaptive gradient methods, particularly AdaGrad and Adam, within the framework of over-the-air model training. Our analysis shows that the AdaGrad-based training algorithm converges to a stationary point at the rate of $mathcalO( ln(T) / T 1 - frac1alpha ).
  arXiv  Detail & Related papers  (2024-03-11T09:10:37Z)
• DaFoEs: Mixing Datasets towards the generalization of vision-state deep-learning Force Estimation in Minimally Invasive Robotic Surgery [6.55111164866752]
  We present a new vision-haptic dataset (DaFoEs) with variable soft environments for the training of deep neural models. We also present a variable encoder-decoder architecture to predict the forces done by the laparoscopic tool using single input or sequence of inputs.
  arXiv  Detail & Related papers  (2024-01-17T14:39:55Z)
• Enhancing Cross-Dataset Performance of Distracted Driving Detection With Score Softmax Classifier And Dynamic Gaussian Smoothing Supervision [6.891556476231427]
  Deep neural networks enable real-time monitoring of in-vehicle drivers, facilitating the timely prediction of distractions, fatigue, and potential hazards. Recent research has exposed unreliable cross-dataset driver behavior recognition due to a limited number of data samples and background noise. We propose a Score-Softmax classifier, which reduces the model overconfidence by enhancing category independence.
  arXiv  Detail & Related papers  (2023-10-08T15:28:01Z)
• Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Model [89.8764435351222]
  We propose a new family of unbiased estimators called WTA-CRS, for matrix production with reduced variance. Our work provides both theoretical and experimental evidence that, in the context of tuning transformers, our proposed estimators exhibit lower variance compared to existing ones.
  arXiv  Detail & Related papers  (2023-05-24T15:52:08Z)
• Stable Target Field for Reduced Variance Score Estimation in Diffusion Models [5.9115407007859755]
  Diffusion models generate samples by reversing a fixed forward diffusion process. We argue that the source of such variance lies in the handling of intermediate noise-variance scales. We propose to remedy the problem by incorporating a reference batch which we use to calculate weighted conditional scores as more stable training targets.
  arXiv  Detail & Related papers  (2023-02-01T18:57:01Z)
• Post-Processing Temporal Action Detection [134.26292288193298]
  Temporal Action Detection (TAD) methods typically take a pre-processing step in converting an input varying-length video into a fixed-length snippet representation sequence. This pre-processing step would temporally downsample the video, reducing the inference resolution and hampering the detection performance in the original temporal resolution. We introduce a novel model-agnostic post-processing method without model redesign and retraining.
  arXiv  Detail & Related papers  (2022-11-27T19:50:37Z)
• Loop Unrolled Shallow Equilibrium Regularizer (LUSER) -- A Memory-Efficient Inverse Problem Solver [26.87738024952936]
  In inverse problems we aim to reconstruct some underlying signal of interest from potentially corrupted and often ill-posed measurements. We propose an LU algorithm with shallow equilibrium regularizers (L) These implicit models are as expressive as deeper convolutional networks, but far more memory efficient during training.
  arXiv  Detail & Related papers  (2022-10-10T19:50:37Z)
• GradViT: Gradient Inversion of Vision Transformers [83.54779732309653]
  We demonstrate the vulnerability of vision transformers (ViTs) to gradient-based inversion attacks. We introduce a method, named GradViT, that optimize random noise into naturally looking images. We observe unprecedentedly high fidelity and closeness to the original (hidden) data.
  arXiv  Detail & Related papers  (2022-03-22T17:06:07Z)
• Plug-and-Play Anomaly Detection with Expectation Maximization Filtering [9.642625267699488]
  A plug-and-play smart camera for crowd surveillance has numerous constraints different from typical anomaly detection. We propose a Core Anomaly-Detection (CAD) neural network which learns the motion behavior of objects in the scene with an unsupervised method. We believe our work is the first step towards using deep learning methods with autonomous plug-and-play smart cameras for crowd anomaly detection.
  arXiv  Detail & Related papers  (2020-06-16T05:28:40Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)
• Software-Level Accuracy Using Stochastic Computing With Charge-Trap-Flash Based Weight Matrix [2.580765958706854]
  Charge Trap Flash (CTF) memory was shown to have a large number of levels before saturation, but variable non-linearity. We show, through simulations, that at an optimum choice of the range, our system performs nearly as well as the models trained using exact floating point operations. We also show its use in reinforcement learning, where it is used for value function approximation in Q-Learning, and learns to complete an episode the mountain car control problem in around 146 steps.
  arXiv  Detail & Related papers  (2020-03-09T02:45:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.