TrainSDC: Characterizing and Mitigating Silent Data Corruption in Large Language Model Training

• URL: http://arxiv.org/abs/2608.30769v1
• Date: Mon, 31 Aug 2026 13:33:26 GMT
• Title: TrainSDC: Characterizing and Mitigating Silent Data Corruption in Large Language Model Training
• Abstract summary: We present the first systematic characterization of silent data corruption (SDC) vulnerability across major computation interfaces in Transformer training.<n>We propose TrainSDC, a characterization-guided protection framework consisting of Q/K-path recomputation, residual-gain monitoring, and exponent-aware gradient scaling.
• Score: 9.623697282987083
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM training is increasingly vulnerable to silent data corruption (SDC), yet existing protection methods largely treat Transformer computations uniformly because their vulnerability remains poorly understood. We present the first systematic characterization of SDC vulnerability across major computation interfaces in both the forward and backward passes of Transformer training. Our analysis reveals two distinct error propagation mechanisms: forward-pass vulnerability is highly location dependent, with faults on the Q/K path producing persistent training deviations, whereas backward-pass vulnerability is largely governed by gradient exponent distributions rather than computation locations. Motivated by these observations, we propose TrainSDC, a characterization-guided protection framework consisting of Q/K-path recomputation, residual-gain monitoring, and exponent-aware gradient scaling. Experiments on Llama 3.2-1B and Qwen3-0.6B show that TrainSDC maintains training behavior close to fault-free execution under both sparse and dense fault injection while introducing only 1.65%-6.76% runtime overhead.

Related papers

• Benchmarking Sensor-Fault Robustness in Forecasting [34.25988781693566]
  We introduce SensorFault-Bench, a shared CPS-grounded sensor-fault stress-test protocol for evaluating forecasting architectures and robustness-improvement methods.<n>It reports worst-scenario degradation, clean mean squared error (MSE), and worst-scenario fault-time MSE, separating relative robustness from absolute error.<n> SensorFault-Bench provides open-source code, documented data access, and reproduction and extension guides.
  arXiv  Detail & Related papers  (2026-05-11T16:41:14Z)
• Balancing Stability and Plasticity in Sequentially Trained Early-Exiting Neural Networks [11.704570155438887]
  Early-exiting neural networks enable adaptive inference by allowing inputs to exit at intermediate classifiers.<n>We propose two alternative approaches that operate at different levels of the model.<n> Experiments on standard benchmarks show that our approaches consistently improve early-exit performance.
  arXiv  Detail & Related papers  (2026-05-06T18:36:32Z)
• Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration [72.0672328514289]
  Multimodal learning often grapples with the challenge of low-quality data.<n>We propose a unified framework, termed Conformal Predictive Self-Calibration.<n>Our framework consistently outperforms existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-05T14:48:52Z)
• Scaling Exposes the Trigger: Input-Level Backdoor Detection in Text-to-Image Diffusion Models via Cross-Attention Scaling [12.573852448122716]
  Text-to-image (T2I) diffusion models have achieved remarkable success in image synthesis, but their reliance on large-scale data and open ecosystems introduces serious backdoor security risks.<n>Existing defenses, particularly input-level methods, are more practical for deployment but often rely on observable anomalies that become unreliable under stealthy, semantics-preserving trigger designs.<n>We propose SET, an input-level backdoor detection framework that constructs response-offset features under multi-scale perturbations and learns a compact benign response space from a small set of clean samples.
  arXiv  Detail & Related papers  (2026-04-14T08:31:37Z)
• Exploring Silent Data Corruption as a Reliability Challenge in LLM Training [2.720409520634766]
  A major challenge arises from Silent Data Corruption (SDC): hardware-induced faults that bypass system-level detection mechanisms.<n>This work provides a controlled study of how intermittent SDC affects Large Language Models pretraining.<n>Building on the observed corruption signatures, we propose a lightweight detection method that identifies potentially harmful parameter updates.
  arXiv  Detail & Related papers  (2026-04-01T10:37:16Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• ThreatFormer-IDS: Robust Transformer Intrusion Detection with Zero-Day Generalization and Explainable Attribution [0.0]
  Intrusion detection in IoT and industrial networks requires models that can detect rare attacks at low false-positive rates while remaining reliable under evolving traffic and limited labels.<n>We propose ThreatFormer- IDS, a Transformer-based sequence modeling framework that converts flow records into time-ordered windows and learns contextual representations for robust intrusion screening.<n>On the ToN IoT benchmark with chronological evaluation, ThreatFormer-IDS achieves AUCROC 0.994, AUC-PR 0.956, and Recall@1%FPR 0.910, outperforming strong tree-based and sequence baselines.
  arXiv  Detail & Related papers  (2026-02-26T23:20:42Z)
• Lie Detector: Unified Backdoor Detection via Cross-Examination Framework [68.45399098884364]
  We propose a unified backdoor detection framework in the semi-honest setting.<n>Our method achieves superior detection performance, improving accuracy by 5.4%, 1.6%, and 11.9% over SoTA baselines.<n> Notably, it is the first to effectively detect backdoors in multimodal large language models.
  arXiv  Detail & Related papers  (2025-03-21T06:12:06Z)
• Improving LLM Safety Alignment with Dual-Objective Optimization [81.98466438000086]
  Existing training-time safety alignment techniques for large language models (LLMs) remain vulnerable to jailbreak attacks.<n>We propose an improved safety alignment that disentangles DPO objectives into two components: (1) robust refusal training, which encourages refusal even when partial unsafe generations are produced, and (2) targeted unlearning of harmful knowledge.
  arXiv  Detail & Related papers  (2025-03-05T18:01:05Z)
• Efficient local linearity regularization to overcome catastrophic overfitting [59.463867084204566]
  Catastrophic overfitting (CO) in single-step adversarial training results in abrupt drops in the adversarial test accuracy (even down to 0%) We introduce a regularization term, called ELLE, to mitigate CO effectively and efficiently in classical AT evaluations.
  arXiv  Detail & Related papers  (2024-01-21T22:55:26Z)
• Robust Pre-Training by Adversarial Contrastive Learning [120.33706897927391]
  Recent work has shown that, when integrated with adversarial training, self-supervised pre-training can lead to state-of-the-art robustness. We improve robustness-aware self-supervised pre-training by learning representations consistent under both data augmentations and adversarial perturbations.
  arXiv  Detail & Related papers  (2020-10-26T04:44:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.