stable-pretraining-v1: Foundation Model Research Made Simple

• URL: http://arxiv.org/abs/2511.19484v1
• Date: Sun, 23 Nov 2025 04:39:48 GMT
• Title: stable-pretraining-v1: Foundation Model Research Made Simple
• Abstract summary: We present stable-pretraining, a modular, and performance-optimized library built on top of PyTorch, Lightning, Hugging Face, and TorchMetrics.<n>Unlike prior toolkits focused narrowly on state-of-the-art results, stable-pretraining is designed for flexibility and speed.<n>We validate the library by demonstrating its ability to generate new research insights with minimal overhead.
• Score: 17.10240529611107
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Foundation models and self-supervised learning (SSL) have become central to modern AI, yet research in this area remains hindered by complex codebases, redundant re-implementations, and the heavy engineering burden of scaling experiments. We present stable-pretraining, a modular, extensible, and performance-optimized library built on top of PyTorch, Lightning, Hugging Face, and TorchMetrics. Unlike prior toolkits focused narrowly on reproducing state-of-the-art results, stable-pretraining is designed for flexibility and iteration speed: it unifies essential SSL utilities--including probes, collapse detection metrics, augmentation pipelines, and extensible evaluation routines--within a coherent and reliable framework. A central design principle is logging everything, enabling fine-grained visibility into training dynamics that makes debugging, monitoring, and reproducibility seamless. We validate the library by demonstrating its ability to generate new research insights with minimal overhead, including depthwise representation probing and the analysis of CLIP degradation under synthetic data finetuning. By lowering barriers to entry while remaining scalable to large experiments, stable-pretraining aims to accelerate discovery and expand the possibilities of foundation model research.

Related papers

• Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It [23.538941671680806]
  Tool use enables large language models to perform complex tasks.<n>Recent agentic reinforcement learning methods show promise for enhancing model capabilities.<n>Some models exhibit catastrophic collapse, where performance abruptly drops and tool-invocation structures fail.
  arXiv  Detail & Related papers  (2026-06-24T16:55:56Z)
• daVinci-LLM:Towards the Science of Pretraining [58.6458452580545]
  daVinci-LLM combines industrial-scale resources with full research freedom to advance the science of pretraining.<n>We release complete data processing pipelines, full training processes, and systematic exploration results.
  arXiv  Detail & Related papers  (2026-03-28T07:00:49Z)
• Understanding by Reconstruction: Reversing the Software Development Process for LLM Pretraining [66.89012795621349]
  Large Language Models (LLMs) often struggle with the deep, long-horizon reasoning required for complex software engineering.<n>We propose a novel paradigm: understanding via reconstruction.<n>We introduce a framework that synthesizes latent agentic trajectories using a multi-agent simulation.
  arXiv  Detail & Related papers  (2026-03-11T09:23:20Z)
• ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning [75.73135757250806]
  Agentic reinforcement learning (ARL) has rapidly gained attention as a promising paradigm for training agents to solve complex, multi-step interactive tasks.<n>Despite encouraging early results, ARL remains highly unstable, often leading to training collapse.<n>In this paper, we first propose ARLArena, a stable training recipe and systematic analysis framework that examines training stability in a controlled and reproducible setting.
  arXiv  Detail & Related papers  (2026-02-25T03:43:34Z)
• AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Research [42.296088824743684]
  AdversariaLLM is a toolbox for conducting jailbreak robustness research.<n>It implements twelve adversarial attack algorithms, integrates seven benchmark datasets spanning harmfulness, over-refusal, and utility evaluation, and provides access to a wide range of open-weight LLMs via Hugging Face.<n>name also integrates judging through the companion package JudgeZoo, which can also be used independently.
  arXiv  Detail & Related papers  (2025-11-06T12:38:09Z)
• Don't Just Fine-tune the Agent, Tune the Environment [25.7349297100143]
  Supervised fine-tuning on synthetic data leads to overfitting.<n>Standard reinforcement learning struggles with a critical cold-start problem and training instability.<n>Our work presents a paradigm shift from supervised fine-tuning on static trajectories to dynamic, environment-based exploration.
  arXiv  Detail & Related papers  (2025-10-11T12:35:15Z)
• Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environments [70.42705564227548]
  We propose an automated environment construction pipeline for large language models (LLMs)<n>This enables the creation of high-quality training environments that provide detailed and measurable feedback without relying on external tools.<n>We also introduce a verifiable reward mechanism that evaluates both the precision of tool use and the completeness of task execution.
  arXiv  Detail & Related papers  (2025-08-12T09:45:19Z)
• Machine Learning Pipeline for Software Engineering: A Systematic Literature Review [0.0]
  This systematic literature review examines state-of-the-art Machine Learning pipelines designed for software engineering (SE)<n>Our findings show that robust preprocessing, such as SMOTE for data balancing, improves model reliability.<n> Ensemble methods like Random Forest and Gradient Boosting dominate performance across tasks.<n>New metrics like Best Arithmetic Mean (BAM) are emerging in niche applications.
  arXiv  Detail & Related papers  (2025-07-31T15:37:30Z)
• Boosting All-in-One Image Restoration via Self-Improved Privilege Learning [72.35265021054471]
  Self-Improved Privilege Learning (SIPL) is a novel paradigm that overcomes limitations by extending the utility of privileged information (PI) beyond training into the inference stage.<n>Central to SIPL is Proxy Fusion, a lightweight module incorporating a learnable Privileged Dictionary.<n>Extensive experiments demonstrate that SIPL significantly advances the state-of-the-art on diverse all-in-one image restoration benchmarks.
  arXiv  Detail & Related papers  (2025-05-30T04:36:52Z)
• Seek in the Dark: Reasoning via Test-Time Instance-Level Policy Gradient in Latent Space [92.6187727249868]
  We introduce LatentSeek, a framework that enhances reasoning through Test-Time Instance-level Adaptation (TTIA) within the model's latent space.<n>LatentSeek is evaluated on a range of reasoning benchmarks, including GSM8K, MATH-500, and AIME2024.<n>Results show that LatentSeek consistently outperforms strong baselines.
  arXiv  Detail & Related papers  (2025-05-19T16:26:02Z)
• Thinking Longer, Not Larger: Enhancing Software Engineering Agents via Scaling Test-Time Compute [61.00662702026523]
  We propose a unified Test-Time Compute scaling framework that leverages increased inference-time instead of larger models.<n>Our framework incorporates two complementary strategies: internal TTC and external TTC.<n>We demonstrate our textbf32B model achieves a 46% issue resolution rate, surpassing significantly larger models such as DeepSeek R1 671B and OpenAI o1.
  arXiv  Detail & Related papers  (2025-03-31T07:31:32Z)
• Will Pre-Training Ever End? A First Step Toward Next-Generation Foundation MLLMs via Self-Improving Systematic Cognition [89.50068130832635]
  Self-Improving cognition (SIcog) is a self-learning framework for constructing next-generation foundation MLLMs by multimodal knowledge.<n>We propose Chain-of-Description for step-by-step visual understanding and integrate structured Chain-of-Thought (CoT) reasoning to support in-depth multimodal reasoning.<n>Experiments demonstrate SIcog's effectiveness in developing MLLMs with enhanced multimodal cognition.
  arXiv  Detail & Related papers  (2025-03-16T00:25:13Z)
• LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models [7.724874639022896]
  We introduce LibMoE, a unified framework for reproducible, efficient, and MoE research.<n>We conduct a comprehensive analysis along three dimensions: (i) routing dynamics, covering expert selection patterns, routing stability and optimality, and how routing entropy reveals task specialization and expert diversity.<n>By lowering the barrier to entry and standardizing evaluation, along with our comprehensive analysis, LibMoE broadens access to MoE research and establishes a reliable benchmark to guide future innovations.
  arXiv  Detail & Related papers  (2024-11-01T14:04:36Z)
• Towards Stable Machine Learning Model Retraining via Slowly Varying Sequences [6.067007470552307]
  We propose a model-agnostic framework for finding sequences of models that are stable across retraining iterations.<n>We develop a mixed-integer optimization formulation that is guaranteed to recover optimal models.<n>We find that, on average, a 2% reduction in predictive power leads to a 30% improvement in stability.
  arXiv  Detail & Related papers  (2024-03-28T22:45:38Z)
• Branch-Tuning: Balancing Stability and Plasticity for Continual Self-Supervised Learning [33.560003528712414]
  Self-supervised learning (SSL) has emerged as an effective paradigm for deriving general representations from vast amounts of unlabeled data. This poses a challenge in striking a balance between stability and plasticity when adapting to new information. We propose Branch-tuning, an efficient and straightforward method that achieves a balance between stability and plasticity in continual SSL.
  arXiv  Detail & Related papers  (2024-03-27T05:38:48Z)
• Self-Supervised Learning with Generative Adversarial Networks for Electron Microscopy [0.0]
  We show how self-supervised pretraining facilitates efficient fine-tuning for a spectrum of downstream tasks. We demonstrate the versatility of self-supervised pretraining across various downstream tasks in the context of electron microscopy.
  arXiv  Detail & Related papers  (2024-02-28T12:25:01Z)
• Toward Fast, Flexible, and Robust Low-Light Image Enhancement [87.27326390675155]
  We develop a new Self-Calibrated Illumination (SCI) learning framework for fast, flexible, and robust brightening images in real-world low-light scenarios. Considering the computational burden of the cascaded pattern, we construct the self-calibrated module which realizes the convergence between results of each stage. We make comprehensive explorations to SCI's inherent properties including operation-insensitive adaptability and model-irrelevant generality.
  arXiv  Detail & Related papers  (2022-04-21T14:40:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.