Studying Various Activation Functions and Non-IID Data for Machine Learning Model Robustness

• URL: http://arxiv.org/abs/2512.04264v1
• Date: Wed, 03 Dec 2025 21:03:45 GMT
• Title: Studying Various Activation Functions and Non-IID Data for Machine Learning Model Robustness
• Authors: Long Dang, Thushari Hapuarachchi, Kaiqi Xiong, Jing Lin,
• Abstract summary: We study the machine learning (ML) model robustness using ten different activation functions through adversarial training.<n>Our proposed centralized adversarial training approach achieves a natural and robust accuracy of 77.08% and 67.96%.<n>In the federated learning environment, however, the robust accuracy decreases significantly, especially on non-IID data.
• Score: 3.641683644638084
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Adversarial training is an effective method to improve the machine learning (ML) model robustness. Most existing studies typically consider the Rectified linear unit (ReLU) activation function and centralized training environments. In this paper, we study the ML model robustness using ten different activation functions through adversarial training in centralized environments and explore the ML model robustness in federal learning environments. In the centralized environment, we first propose an advanced adversarial training approach to improving the ML model robustness by incorporating model architecture change, soft labeling, simplified data augmentation, and varying learning rates. Then, we conduct extensive experiments on ten well-known activation functions in addition to ReLU to better understand how they impact the ML model robustness. Furthermore, we extend the proposed adversarial training approach to the federal learning environment, where both independent and identically distributed (IID) and non-IID data settings are considered. Our proposed centralized adversarial training approach achieves a natural and robust accuracy of 77.08% and 67.96%, respectively on CIFAR-10 against the fast gradient sign attacks. Experiments on ten activation functions reveal ReLU usually performs best. In the federated learning environment, however, the robust accuracy decreases significantly, especially on non-IID data. To address the significant performance drop in the non-IID data case, we introduce data sharing and achieve the natural and robust accuracy of 70.09% and 54.79%, respectively, surpassing the CalFAT algorithm, when 40% data sharing is used. That is, a proper percentage of data sharing can significantly improve the ML model robustness, which is useful to some real-world applications.

Related papers

• Efficient Machine Unlearning via Influence Approximation [75.31015485113993]
  Influence-based unlearning has emerged as a prominent approach to estimate the impact of individual training samples on model parameters without retraining.<n>This paper establishes a theoretical link between memorizing (incremental learning) and forgetting (unlearning)<n>We introduce the Influence Approximation Unlearning algorithm for efficient machine unlearning from the incremental perspective.
  arXiv  Detail & Related papers  (2025-07-31T05:34:27Z)
• Efficient Federated Learning with Heterogeneous Data and Adaptive Dropout [62.73150122809138]
  Federated Learning (FL) is a promising distributed machine learning approach that enables collaborative training of a global model using multiple edge devices.<n>We propose the FedDHAD FL framework, which comes with two novel methods: Dynamic Heterogeneous model aggregation (FedDH) and Adaptive Dropout (FedAD)<n>The combination of these two methods makes FedDHAD significantly outperform state-of-the-art solutions in terms of accuracy (up to 6.7% higher), efficiency (up to 2.02 times faster), and cost (up to 15.0% smaller)
  arXiv  Detail & Related papers  (2025-07-14T16:19:00Z)
• LifelongPR: Lifelong point cloud place recognition based on sample replay and prompt learning [15.464706470200337]
  We propose LifelongPR, a novel continual learning framework for PCPR.<n>LifelongPR effectively extracts and fuses knowledge from sequential point cloud data.<n>Our method achieves 6.50% improvement in mIR@1, 7.96% improvement in mR@1, and an 8.95% reduction in F.
  arXiv  Detail & Related papers  (2025-07-14T08:13:33Z)
• Privacy-Preserved Automated Scoring using Federated Learning for Educational Research [1.2556373621040728]
  We propose a federated learning (FL) framework for automated scoring of educational assessments.<n>We benchmark our model against two state-of-the-art FL methods and a centralized learning baseline.<n>Results show that our model achieves the highest accuracy (94.5%) among FL approaches.
  arXiv  Detail & Related papers  (2025-03-12T19:06:25Z)
• On the Robustness of Distributed Machine Learning against Transfer Attacks [1.0787328610467801]
  No prior work has examined the combined robustness stemming from distributing both the learning and the inference process.<n>We show that properly distributed ML instantiations achieve across-the-board improvements in accuracy-robustness tradeoffs against state-of-the-art transfer-based attacks.
  arXiv  Detail & Related papers  (2024-12-18T17:27:17Z)
• Sample Efficient Robot Learning in Supervised Effect Prediction Tasks [0.0]
  MUSEL (Model Uncertainty for Sample-Efficient Learning) is a novel AL framework tailored for regression tasks in robotics.<n>We show that MUSEL improves both learning accuracy and sample efficiency, validating its effectiveness in learning action effects selecting informative samples.
  arXiv  Detail & Related papers  (2024-12-03T09:48:28Z)
• Streamlined Federated Unlearning: Unite as One to Be Highly Efficient [12.467630082668254]
  Recently, the enactment of right to be forgotten" laws and regulations has imposed new privacy requirements on federated learning (FL)<n>We propose a streamlined federated unlearning approach (SFU) aimed at effectively removing the influence of the target data while preserving the model performance on retained data without degradation.
  arXiv  Detail & Related papers  (2024-11-28T12:52:48Z)
• Uncertainty Aware Learning for Language Model Alignment [97.36361196793929]
  We propose uncertainty-aware learning (UAL) to improve the model alignment of different task scenarios. We implement UAL in a simple fashion -- adaptively setting the label smoothing value of training according to the uncertainty of individual samples. Experiments on widely used benchmarks demonstrate that our UAL significantly and consistently outperforms standard supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-06-07T11:37:45Z)
• Improving Machine Learning Robustness via Adversarial Training [3.7942983866014073]
  We investigate ML robustness using adversarial training in centralized and decentralized environments. In the centralized environment, we achieve a test accuracy of 65.41% and 83.0% when classifying adversarial examples. In the decentralized environment, we study Federated learning (FL) robustness by using adversarial training with independent and identically distributed (IID) and non-IID data.
  arXiv  Detail & Related papers  (2023-09-22T02:43:04Z)
• Learning Objective-Specific Active Learning Strategies with Attentive Neural Processes [72.75421975804132]
  Learning Active Learning (LAL) suggests to learn the active learning strategy itself, allowing it to adapt to the given setting. We propose a novel LAL method for classification that exploits symmetry and independence properties of the active learning problem. Our approach is based on learning from a myopic oracle, which gives our model the ability to adapt to non-standard objectives.
  arXiv  Detail & Related papers  (2023-09-11T14:16:37Z)
• Robust Learning with Progressive Data Expansion Against Spurious Correlation [65.83104529677234]
  We study the learning process of a two-layer nonlinear convolutional neural network in the presence of spurious features. Our analysis suggests that imbalanced data groups and easily learnable spurious features can lead to the dominance of spurious features during the learning process. We propose a new training algorithm called PDE that efficiently enhances the model's robustness for a better worst-group performance.
  arXiv  Detail & Related papers  (2023-06-08T05:44:06Z)
• Towards Robust Dataset Learning [90.2590325441068]
  We propose a principled, tri-level optimization to formulate the robust dataset learning problem. Under an abstraction model that characterizes robust vs. non-robust features, the proposed method provably learns a robust dataset.
  arXiv  Detail & Related papers  (2022-11-19T17:06:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.