Trust, but Verify: Peeling Low-Bit Transformer Networks for Training Monitoring

• URL: http://arxiv.org/abs/2605.02853v1
• Date: Mon, 04 May 2026 17:30:50 GMT
• Title: Trust, but Verify: Peeling Low-Bit Transformer Networks for Training Monitoring
• Abstract summary: We propose a layer-wise peeling framework for training dynamics, in which each layer is locally optimized against representations of the trained model.<n>Experiments on decoder-only transformer models show that these layer-wise reference bounds can match or even surpass the trained model at various stages.
• Score: 20.854288216118423
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding whether deep neural networks are effectively optimized remains challenging, as training occurs in highly nonconvex landscapes and standard metrics provide limited visibility into layer-wise learning quality. This challenge is particularly acute for transformer-based language models, where training is expensive, models are often reused in frozen form, and poorly optimized layers can silently degrade performance. We propose a layer-wise peeling framework for monitoring training dynamics, in which each transformer layer is locally optimized against intermediate representations of the trained model. By constructing lightweight, layer-specific reference solutions and projecting layers onto multiple intermediate outputs via different permutations, we obtain achievable baselines that enable fine-grained diagnosis of under-optimized layers. Experiments on decoder-only transformer models show that these layer-wise reference bounds can match or even surpass the trained model at various stages of training, exposing inefficiencies that remain hidden in aggregate loss curves. We further demonstrate that this analysis remains effective under binarization and quantized settings, where training dynamics are particularly fragile. Across all numerical results, the proposed bounds consistently separate apparent convergence from effective optimality, highlighting optimization opportunities that are invisible when relying on training loss alone.

Related papers

• Simplifying Neural Networks During Training [1.7966001353008778]
  We propose an NC-inspired training framework for simplifying deep networks during training.<n>Our method monitors representation dynamics through the Inverse Fisher Criterion.<n>We then replace the trailing proxy layers with a lightweight classification head and continue training the reduced model.
  arXiv  Detail & Related papers  (2026-07-30T08:32:40Z)
• Freeze Deep, Train Shallow: Interpretable Layer Allocation for Continued Pre-Training [5.338568138489752]
  We propose LayerTracer to locate task execution positions and quantify layer sensitivity.<n>We show that deep layers act as critical regions for task execution and maintain high stability against disruptive updates.<n>We also present a hybrid model case study, which validates that placing high-quality pre-trained modules in deep layers effectively preserves inherent knowledge of the model.
  arXiv  Detail & Related papers  (2026-05-12T02:11:31Z)
• A Layer Separation Optimization Framework for Cross-Entropy Training in Deep Learning [17.37954084665439]
  This paper investigates the deep learning optimization problem with softmax cross-entropy loss.<n>We propose a layer separation strategy to alleviate the strong nonity subproblems.<n> Numerical experiments validate the effectiveness of the proposed methods.
  arXiv  Detail & Related papers  (2026-04-25T09:33:24Z)
• Neural Network Optimization Reimagined: Decoupled Techniques for Scratch and Fine-Tuning [49.751529745537546]
  We propose DualOpt, a novel approach that decouples optimization techniques specifically tailored for training from scratch.<n>For training from scratch, we introduce real-time layer-wise weight decay, designed to enhance both convergence and generalization.<n>We extend the layer-wise weight decay to dynamically adjust the rollback levels across layers, adapting to the varying demands of different downstream tasks.
  arXiv  Detail & Related papers  (2026-04-21T06:27:18Z)
• Enhancing Gradient Inversion Attacks in Federated Learning via Hierarchical Feature Optimization [56.95448807869383]
  Federated Learning (FL) has emerged as a compelling paradigm for privacy-preserving distributed machine learning.<n>Recent studies find that the gradients exchanged in the FL system are also vulnerable to privacy leakage.<n>We propose textbfGradient textbfInversion over textbfFeature textbfDomains (GIFD)
  arXiv  Detail & Related papers  (2026-04-01T14:32:15Z)
• A Constrained Optimization Perspective of Unrolled Transformers [77.12297732942095]
  We introduce a constrained optimization framework for training transformers that behave like optimization descent algorithms.<n>We observe constrained transformers achieve stronger to perturbations robustness and maintain higher out-of-distribution generalization.
  arXiv  Detail & Related papers  (2026-01-24T02:12:39Z)
• Noise-Adaptive Layerwise Learning Rates: Accelerating Geometry-Aware Optimization for Deep Neural Network Training [31.259303817974693]
  We introduce a noise-adaptive layerwise learning rate scheme on top of geometry-aware optimization algorithms.<n>Our method estimates gradient variance in the dual norm induced by the chosen LMO on the fly.<n>Our algorithm achieves a sharp convergence rate.
  arXiv  Detail & Related papers  (2025-10-15T18:42:13Z)
• Input Conditioned Layer Dropping in Speech Foundation Models [11.05223262950967]
  layer dropping ($mathcalLD$) skips fraction of the layers of a backbone network during inference to reduce the computational load.<n>We propose input-driven $mathcalLD$ that employs the network's input features and a lightweight layer selecting network to determine optimum combination of processing layers.
  arXiv  Detail & Related papers  (2025-07-10T17:39:03Z)
• Hierarchical Feature-level Reverse Propagation for Post-Training Neural Networks [24.442592456755698]
  End-to-end autonomous driving has emerged as a dominant paradigm, yet its highly entangled black-box models pose challenges in terms of interpretability and safety assurance.<n>This paper proposes a hierarchical and decoupled post-training framework tailored for pretrained neural networks.
  arXiv  Detail & Related papers  (2025-06-08T15:19:03Z)
• Accurate Neural Network Pruning Requires Rethinking Sparse Optimization [87.90654868505518]
  We show the impact of high sparsity on model training using the standard computer vision and natural language processing sparsity benchmarks. We provide new approaches for mitigating this issue for both sparse pre-training of vision models and sparse fine-tuning of language models.
  arXiv  Detail & Related papers  (2023-08-03T21:49:14Z)
• Layer-wise Linear Mode Connectivity [52.6945036534469]
  Averaging neural network parameters is an intuitive method for the knowledge of two independent models. It is most prominently used in federated learning. We analyse the performance of the models that result from averaging single, or groups.
  arXiv  Detail & Related papers  (2023-07-13T09:39:10Z)
• Neural Networks with Quantization Constraints [111.42313650830248]
  We present a constrained learning approach to quantization training. We show that the resulting problem is strongly dual and does away with gradient estimations. We demonstrate that the proposed approach exhibits competitive performance in image classification tasks.
  arXiv  Detail & Related papers  (2022-10-27T17:12:48Z)
• Initialization and Regularization of Factorized Neural Layers [23.875225732697142]
  We show how to initialize and regularize Factorized layers in deep nets. We show how these schemes lead to improved performance on both translation and unsupervised pre-training.
  arXiv  Detail & Related papers  (2021-05-03T17:28:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.