Exploring Silent Data Corruption as a Reliability Challenge in LLM Training

• URL: http://arxiv.org/abs/2604.00726v1
• Date: Wed, 01 Apr 2026 10:37:16 GMT
• Title: Exploring Silent Data Corruption as a Reliability Challenge in LLM Training
• Abstract summary: A major challenge arises from Silent Data Corruption (SDC): hardware-induced faults that bypass system-level detection mechanisms.<n>This work provides a controlled study of how intermittent SDC affects Large Language Models pretraining.<n>Building on the observed corruption signatures, we propose a lightweight detection method that identifies potentially harmful parameter updates.
• Score: 2.720409520634766
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As Large Language Models (LLMs) scale in size and complexity, the consequences of failures during training become increasingly severe. A major challenge arises from Silent Data Corruption (SDC): hardware-induced faults that bypass system-level detection mechanisms. SDC may behave like benign numerical noise, but can also cause harmful gradient corruption that leads to loss spikes, divergence, or stalled progress. This work provides a controlled study of how intermittent SDC affects LLM pretraining. Using targeted fault injection at the level of GPU matrix-multiply instructions, we characterize the sensitivity of different bit positions, kernel functions, and execution stages. Our analysis shows that locally originating faults can produce impactful corruption, including NaN propagation, short-lived spikes in loss, gradient norm, and attention logits, as well as persistent parameter divergence. Building on the observed corruption signatures, we propose a lightweight detection method that identifies potentially harmful parameter updates. Experiments on LLaMA models with 60M, 350M, and 1.3B parameters demonstrate that recomputing the most recent training step upon detection can effectively mitigate the impact of these events.

Related papers

• TrainSDC: Characterizing and Mitigating Silent Data Corruption in Large Language Model Training [9.623697282987083]
  We present the first systematic characterization of silent data corruption (SDC) vulnerability across major computation interfaces in Transformer training.<n>We propose TrainSDC, a characterization-guided protection framework consisting of Q/K-path recomputation, residual-gain monitoring, and exponent-aware gradient scaling.
  arXiv  Detail & Related papers  (2026-08-31T13:33:26Z)
• Error Amplification Limits ANN-to-SNN Conversion in Continuous Control [64.99656514469972]
  Spiking Neural Networks (SNNs) can achieve competitive performance by converting already existing well-trained Artificial Neural Networks (ANNs)<n>Existing conversion methods perform poorly in continuous control, where suitable baselines are largely absent.<n>We propose Cross-Step Residual Potential Initialization (CRPI), a lightweight training-free mechanism that carries over residual membrane potentials across decision steps to suppress temporally correlated errors.
  arXiv  Detail & Related papers  (2026-01-29T14:28:00Z)
• Noise-Robust Tiny Object Localization with Flows [63.60972031108944]
  We propose a noise-robust localization framework leveraging normalizing flows for flexible error modeling and uncertainty-guided optimization.<n>Our method captures complex, non-Gaussian prediction distributions through flow-based error modeling, enabling robust learning under noisy supervision.<n>An uncertainty-aware gradient modulation mechanism further suppresses learning from high-uncertainty, noise-prone samples, mitigating overfitting while stabilizing training.
  arXiv  Detail & Related papers  (2026-01-02T09:16:55Z)
• DiffuGuard: How Intrinsic Safety is Lost and Found in Diffusion Large Language Models [50.21378052667732]
  We conduct an in-depth analysis of dLLM vulnerabilities to jailbreak attacks across two distinct dimensions: intra-step and inter-step dynamics.<n>We propose DiffuGuard, a training-free defense framework that addresses vulnerabilities through a dual-stage approach.
  arXiv  Detail & Related papers  (2025-09-29T05:17:10Z)
• Where Did It Go Wrong? Attributing Undesirable LLM Behaviors via Representation Gradient Tracing [12.835224376066769]
  Large Language Models (LLMs) have demonstrated remarkable capabilities, yet their deployment is frequently undermined by undesirable behaviors.<n>We introduce a novel and efficient framework that diagnoses a range of undesirable LLM behaviors by analyzing representation and its gradients.<n>We systematically evaluate our method for tasks that include tracking harmful content, detecting backdoor poisoning, and identifying knowledge contamination.
  arXiv  Detail & Related papers  (2025-09-26T12:07:47Z)
• Defending against Indirect Prompt Injection by Instruction Detection [109.30156975159561]
  InstructDetector is a novel detection-based approach that leverages the behavioral states of LLMs to identify potential IPI attacks.<n>InstructDetector achieves a detection accuracy of 99.60% in the in-domain setting and 96.90% in the out-of-domain setting, and reduces the attack success rate to just 0.03% on the BIPIA benchmark.
  arXiv  Detail & Related papers  (2025-05-08T13:04:45Z)
• Understanding Silent Data Corruption in LLM Training [22.679273469491754]
  We investigate the impact of silent data corruption (SDC) on large language training by comparing model training between healthy production nodes and unhealthy nodes exhibiting SDCs.<n>Our results reveal that the impact of SDCs on computation varies on different unhealthy nodes.
  arXiv  Detail & Related papers  (2025-02-17T22:07:49Z)
• ATTNChecker: Highly-Optimized Fault Tolerant Attention for Large Language Model Training [14.178223242134166]
  Large Language Models (LLMs) have demonstrated remarkable performance in various natural language processing tasks.<n>LLMs are susceptible to faults, particularly in the attention mechanism, which is a critical component of transformer-based LLMs.<n>We propose ATTNChecker, the first Algorithm-Based Fault Tolerance (ABFT) technique tailored for the attention mechanism in LLMs.
  arXiv  Detail & Related papers  (2024-10-15T15:52:45Z)
• Analyzing Adversarial Inputs in Deep Reinforcement Learning [53.3760591018817]
  We present a comprehensive analysis of the characterization of adversarial inputs, through the lens of formal verification. We introduce a novel metric, the Adversarial Rate, to classify models based on their susceptibility to such perturbations. Our analysis empirically demonstrates how adversarial inputs can affect the safety of a given DRL system with respect to such perturbations.
  arXiv  Detail & Related papers  (2024-02-07T21:58:40Z)
• A Micro Architectural Events Aware Real-Time Embedded System Fault Injector [0.12187048691454236]
  This paper introduces a novel fault injector designed to facilitate the monitoring, aggregation, and examination of micro-architectural events. The methodology targets bit-flipping within the memory system, affecting CPU registers and RAM. The outcomes of these fault injections enable a thorough analysis of the impact of soft errors and establish a robust correlation between the identified faults and the essential timing predictability demanded by SACRES.
  arXiv  Detail & Related papers  (2024-01-16T14:41:20Z)
• Robust Tiny Object Detection in Aerial Images amidst Label Noise [50.257696872021164]
  This study addresses the issue of tiny object detection under noisy label supervision. We propose a DeNoising Tiny Object Detector (DN-TOD), which incorporates a Class-aware Label Correction scheme. Our method can be seamlessly integrated into both one-stage and two-stage object detection pipelines.
  arXiv  Detail & Related papers  (2024-01-16T02:14:33Z)
• Diffusion Denoising Process for Perceptron Bias in Out-of-distribution Detection [67.49587673594276]
  We introduce a new perceptron bias assumption that suggests discriminator models are more sensitive to certain features of the input, leading to the overconfidence problem. We demonstrate that the diffusion denoising process (DDP) of DMs serves as a novel form of asymmetric, which is well-suited to enhance the input and mitigate the overconfidence problem. Our experiments on CIFAR10, CIFAR100, and ImageNet show that our method outperforms SOTA approaches.
  arXiv  Detail & Related papers  (2022-11-21T08:45:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.