All is Not Lost: LLM Recovery without Checkpoints

• URL: http://arxiv.org/abs/2506.15461v1
• Date: Wed, 18 Jun 2025 13:48:33 GMT
• Title: All is Not Lost: LLM Recovery without Checkpoints
• Abstract summary: CheckFree is an efficient recovery method where a failing stage is substituted by a weighted average of the closest neighboring stages.<n>CheckFree and CheckFree+ outperform checkpointing and redundant computation in terms of convergence in wall-clock time by over 12%.
• Score: 0.1638581561083717
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training LLMs on decentralized and wimpy computation nodes, e.g., multiple on-spot instances, lowers the training cost and enables model democratization. The inevitable challenge here is the churn of nodes due to failures and the operator's scheduling policies, leading to losing a stage - a part of the model. The conventional approaches to recover from failures are to either use checkpointing, where periodically a copy of the entire model is sent to an additional storage, or redundant computation. These approaches yield significant communication and/or computation overhead even in non-failure cases and scale poorly in settings with large models. In this paper, we propose, CheckFree, an efficient recovery method where a failing stage is substituted by a weighted average of the closest neighboring stages. In contrast to the state of the art, CheckFree requires no additional computation or storage. However, because of the nature of averaging neighbouring stages, it can only recover failures of intermediate stages. We further extend our method to CheckFree+ with out-of-order pipeline execution to tolerate crashes of the first and last stages. Thanks to out-of-order pipelining, behaviour of those stages is mimicked by their neighboring ones, which allows CheckFree+ to recover them by simply copying the weights from the immediate neighbour. To be able to recover the (de)embedding layers, CheckFree+ copies those layers to the neighboring stages, which requires relatively small storage overhead. We extensively evaluate our method on LLaMa models of model sizes from 124M to 1.5B with varying failure frequencies. In the case of low and medium failure rates (5-10%), CheckFree and CheckFree+ outperform both checkpointing and redundant computation in terms of convergence in wall-clock time by over 12%. Both of our proposals can be run via our code available at: https://github.com/gensyn-ai/CheckFree.

Related papers

• ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation [72.54604107217669]
  textscpass@$k$ nearly vanishes after compression, yet textscpass@$k$ recovers substantially under repeated sampling.<n>Recovery should train on the compressed model's own on-policy states with dense token-level supervision.<n>We propose textbfshortopd, a short-to-long OPD schedule that detects teacher-confirmed repetitive suffixes.
  arXiv  Detail & Related papers  (2026-07-14T17:50:50Z)
• The Orthogonalized Read Is a Removable Training Scaffold for Recurrent Memory [0.0]
  The effect replicates, but it is not a memory improvement.<n>Training on this task is a long chance plateau followed by a sharp escape.<n>Much of the published gain needs no architecture at all.
  arXiv  Detail & Related papers  (2026-07-02T18:54:23Z)
• Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving [69.5155152169752]
  We propose a two-stage cascaded solution for large language models (LLMs)<n>Stage 1 clusters incoming queries and assigns each cluster to its most cost-effective model.<n>Stage 2 adds a quality estimation (QE) cascade; when an output from Stage 1 is judged low-quality, the query is escalated to a stronger model.<n>On the test datasets, the cascaded system retains 97-99% of the strongest model's accuracy while reducing Time Per Output Token (TPOT)<n>It requires only task-correctness labels and adapts to changes in the model pool without manual reconfiguration.
  arXiv  Detail & Related papers  (2026-06-25T18:29:24Z)
• REVES: REvision and VErification--Augmented Training for Test-Time Scaling [53.197756110943395]
  We propose a two-stage iterative framework that alternates between online data/prompt augmentation and policy optimization.<n>We observe gains of +6.5 points over the RL baseline and +4.0 points over standard multi-turn training.
  arXiv  Detail & Related papers  (2026-06-17T10:37:23Z)
• Stop Early, Spend Less: Hidden-State Probes as a Practical Recipe for Streaming Moderation of LLM Outputs [0.0]
  Existing approaches rely on a separate moderation model applied after generation.<n>We observe that the signal needed for moderation is already present in the model hidden states.<n>We train lightweight token-level probes that operate directly on internal activations.
  arXiv  Detail & Related papers  (2026-06-09T07:01:43Z)
• Fast Unlearning at Scale via Margin Self-Correction [52.46927918952516]
  Language-model unlearning updates a trained model to behave as if it had not seen selected training examples.<n>MASC achieves a competitive forget-retain trade-off at a fraction of the computational cost of existing baselines.
  arXiv  Detail & Related papers  (2026-06-01T21:49:54Z)
• One Step Forward and K Steps Back: Better Reasoning with Denoising Recursion Models [4.903188186588148]
  We introduce Denoising Recursion Models, a method that corrupts data with noise but trains the model to reverse the corruption over multiple steps.<n>This strategy provides a tractable curriculum of intermediate states, while better aligning training with testing and incentivizing non-greedy, forward-looking generation.<n>We show this approach outperforms the Tiny Recursion Model on ARC-AGI, where it recently achieved breakthrough performance.
  arXiv  Detail & Related papers  (2026-04-20T21:06:12Z)
• MERGETUNE: Continued fine-tuning of vision-language models [77.8627788911249]
  Fine-tuning vision-language models (VLMs) often leads to catastrophic forgetting of pretrained knowledge.<n>We introduce a novel paradigm, continued fine-tuning (CFT), which seeks to recover pretrained knowledge after a zero-shot model has already been adapted.
  arXiv  Detail & Related papers  (2026-01-15T15:15:53Z)
• Adapt in the Wild: Test-Time Entropy Minimization with Sharpness and Feature Regularization [85.50560211492898]
  Test-time adaptation (TTA) may fail to improve or even harm the model performance when test data have mixed distribution shifts.<n>This is often a key obstacle preventing existing TTA methods from being deployed in the real world.<n>We propose a sharpness-aware and reliable entropy minimization method, called SAR, for stabilizing TTA from two aspects.
  arXiv  Detail & Related papers  (2025-09-05T10:03:00Z)
• Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Sampling [90.86991492288487]
  evaluating constraint on every token can be prohibitively expensive.<n> LCD can distort the global distribution over strings, sampling tokens based only on local information.<n>We show that our approach is superior to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-04-07T18:30:18Z)
• SkipPipe: Partial and Reordered Pipelining Framework for Training LLMs in Heterogeneous Networks [0.1638581561083717]
  SkipPipe is a partial pipeline framework to reduce the end-to-end training time for Large Language Models.<n>Our results show that SkipPipe reduces training time by up to $55%$ compared to full pipeline.
  arXiv  Detail & Related papers  (2025-02-27T09:34:23Z)
• Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting [15.251425165987987]
  Fine-tuning a pre-trained model on a downstream task often degrades its original capabilities.<n>We propose a sample weighting scheme for the fine-tuning data based on the pre-trained model's losses.<n>We empirically demonstrate the efficacy of our method on both language and vision tasks.
  arXiv  Detail & Related papers  (2025-02-05T00:49:59Z)
• ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think [22.333776599402754]
  We introduce ZeroFlow, the first benchmark designed to evaluate gradient-free optimization algorithms for overcoming forgetting.<n>Our results show that forward passes alone can be sufficient to mitigate forgetting.<n>We propose new enhancements that further improve forgetting resistance using only forward passes.
  arXiv  Detail & Related papers  (2025-01-02T04:10:17Z)
• Truncated Consistency Models [57.50243901368328]
  Training consistency models requires learning to map all intermediate points along PF ODE trajectories to their corresponding endpoints.<n>We empirically find that this training paradigm limits the one-step generation performance of consistency models.<n>We propose a new parameterization of the consistency function and a two-stage training procedure that prevents the truncated-time training from collapsing to a trivial solution.
  arXiv  Detail & Related papers  (2024-10-18T22:38:08Z)
• A Voter-Based Stochastic Rejection-Method Framework for Asymptotically Safe Language Model Outputs [0.0]
  We propose an approach for preventing unsafe or otherwise low-quality large language model (LLM) outputs by leveraging theity of LLMs.<n>In this system, LLM checkers vote on the acceptability of a generated output, regenerating it if a threshold of disapproval is reached.
  arXiv  Detail & Related papers  (2024-07-24T04:27:55Z)
• DataStates-LLM: Lazy Asynchronous Checkpointing for Large Language Models [3.3484462092188005]
  We introduce a lazy asynchronous multi-level approach that takes advantage of the fact that the tensors making up the model and state shards remain immutable for extended periods of time. The results show up to 48$times$ faster checkpointing and 2.2$times$ faster end-to-end training compared with state-of-art checkpointing approaches.
  arXiv  Detail & Related papers  (2024-06-15T18:30:40Z)
• An Efficient Rehearsal Scheme for Catastrophic Forgetting Mitigation during Multi-stage Fine-tuning [55.467047686093025]
  A common approach to alleviate such forgetting is to rehearse samples from prior tasks during fine-tuning.<n>We propose a sampling scheme, textttbf mix-cd, that prioritizes rehearsal of collateral damage'' samples.<n>Our approach is computationally efficient, easy to implement, and outperforms several leading continual learning methods in compute-constrained settings.
  arXiv  Detail & Related papers  (2024-02-12T22:32:12Z)
• Waypoint-Based Imitation Learning for Robotic Manipulation [56.4287610994102]
  Waypoint labeling is underspecified, and requires additional human supervision. We propose Automatic Waypoint Extraction (AWE) for imitation learning. AWE can increase the success rate of state-of-the-art algorithms by up to 25% in simulation and by 4-28% on real-world bimanual manipulation tasks.
  arXiv  Detail & Related papers  (2023-07-26T17:45:55Z)
• Post-Processing Temporal Action Detection [134.26292288193298]
  Temporal Action Detection (TAD) methods typically take a pre-processing step in converting an input varying-length video into a fixed-length snippet representation sequence. This pre-processing step would temporally downsample the video, reducing the inference resolution and hampering the detection performance in the original temporal resolution. We introduce a novel model-agnostic post-processing method without model redesign and retraining.
  arXiv  Detail & Related papers  (2022-11-27T19:50:37Z)
• Breaking the Sample Complexity Barrier to Regret-Optimal Model-Free Reinforcement Learning [52.76230802067506]
  A novel model-free algorithm is proposed to minimize regret in episodic reinforcement learning. The proposed algorithm employs an em early-settled reference update rule, with the aid of two Q-learning sequences. The design principle of our early-settled variance reduction method might be of independent interest to other RL settings.
  arXiv  Detail & Related papers  (2021-10-09T21:13:48Z)
• Unsupervised 3D Human Mesh Recovery from Noisy Point Clouds [30.401088478228235]
  We present an unsupervised approach to reconstruct human shape and pose from noisy point cloud. Our network is trained from scratch with no need to warm-up the network with supervised data.
  arXiv  Detail & Related papers  (2021-07-15T18:07:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.