Most Transformer Modifications Still Do Not Transfer at 1-3B: A 2020-2026 Update to Narang et al. (2021) with Downstream Evaluation and a Noise Floor

• URL: http://arxiv.org/abs/2605.20798v1
• Date: Wed, 20 May 2026 06:43:34 GMT
• Title: Most Transformer Modifications Still Do Not Transfer at 1-3B: A 2020-2026 Update to Narang et al. (2021) with Downstream Evaluation and a Noise Floor
• Abstract summary: We test 20 Transformer modifications at 1.2B and 3B under strict iso-data, iso-compute, iso-recipe control, with a multi-seed baseline noise floor and CLIMB-12 downstream evaluation as the primary metric.<n>Of the 20 modifications, only two clear Bonferroni correction at 1.2B; one of those two further fails to train stably at 3B under the shared recipe.<n>We conclude that noise-floor reporting, downstream evaluation, and cross-scale stability testing are now prerequisites for architecture comparisons at 1-3B.
• Score: 32.18134535997269
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Narang et al. (2021) evaluated 40+ Transformer modifications at T5-base scale and concluded that most did not transfer. Five years later, the typical working regime has moved to 1-3B parameters, downstream evaluation has replaced pretraining perplexity, and a substantially different catalogue of modifications has emerged. We revisit their question by testing 20 post-2021 Transformer modifications at 1.2B and 3B under strict iso-data, iso-compute, iso-recipe control, with a multi-seed baseline noise floor and CLIMB-12 downstream evaluation as the primary metric. The central finding reproduces theirs at this curated set: most modifications do not transfer. Of the 20 modifications, only two clear Bonferroni correction at 1.2B; one of those two further fails to train stably at 3B under the shared recipe. We also find that the loss-downstream gap reported by Tay et al. (2023) enlarges several-fold for attention-output modifications: two significant failures converge to within 2-3% of baseline validation loss yet drop 6-16 CLIMB-points. We conclude that noise-floor reporting, downstream evaluation, and cross-scale stability testing are now prerequisites for architecture comparisons at 1-3B.

Related papers

• Pre- and Post-Treatment Brain Metastases Segmentation Using nnU-Net with Post-Processing for BraTS 2026 [3.6641231031729173]
  Brain metastases exhibit high inter-lesion variability in size, enhancement pattern, and post-treatment appearance.<n>We build a pragmatic pipeline on a 5-fold nnU-Net ResEnc-L ensemble.<n>We report thirteen negative results spanning loss engineering, alternative backbones, and inference-time settings.
  arXiv  Detail & Related papers  (2026-09-10T12:43:04Z)
• Can You Trust Frozen Hematology Foundation Models under Acquisition Shift? [0.0]
  We audit 15 frozen encoders across four public single-cell acquisition domains.<n>In-domain linear-probe macro-F1 is saturated (0.98-0.997), yet cross-dataset macro-F1 drops 34-72%.<n>DinoBloom-L, the in-domain best, falls to 10th of 15 on the most-shifted target (MLL23), behind RedDino and several general and pathology encoders.
  arXiv  Detail & Related papers  (2026-08-25T20:55:57Z)
• How Much Capacity Does EEG Denoising Need? Ultra-Compact Networks reveal Benchmark Saturation and Metric-Utility Gap [0.0]
  Deep learning EEG denoising architectures have scaled from tens of thousands to tens of millions of parameters, yet no prior study has isolated model capacity as the experimental variable.<n>We address both gaps by fixing architecture, loss, data split, and training recipe while sweeping only channel width from 1.05K to 40.26K parameters in a minimal depthwiseseparable convolutional U-Net.
  arXiv  Detail & Related papers  (2026-06-07T12:17:25Z)
• EVA-0: Test-Time Model Evolution with Only Two Forward Passes per Sample [55.284129005947484]
  We study test-time model evolution under a strict two-forward budget.<n>We reveal three key obstacles in zeroth-order test-time optimization.<n>We propose EVA-0, a minimal zeroth-order adaptation framework.
  arXiv  Detail & Related papers  (2026-05-15T09:26:02Z)
• Feature-Augmented Transformers for Robust AI-Text Detection Across Domains and Generators [0.10923877073891443]
  We train transformer-based detectors on HC3 PLUS and calibrate a single decision threshold by maximising balanced accuracy on held-out validation.<n>We evaluate in-domain on HC3 PLUS, under cross-dataset transfer to the multi-domain, multi-generator M4 benchmark, and on the external AI-Text-Detection-Pile.<n>Our best model (DeBERTa-v3-base+FeatAttn) achieves 85.9% balanced accuracy on M4.
  arXiv  Detail & Related papers  (2026-05-05T16:52:26Z)
• TEMPO: Scaling Test-time Training for Large Reasoning Models [87.61789183311856]
  Test-time training (TTT) adapts model parameters on unlabeled test instances during inference time.<n>We propose TEMPO, a TTT framework that interleaves policy refinement on unlabeled questions with periodic critic recalibration on a labeled dataset.
  arXiv  Detail & Related papers  (2026-04-21T10:01:04Z)
• Hybrid Gated Flow (HGF): Stabilizing 1.58-bit LLMs via Selective Low-Rank Correction [0.766310831583367]
  Hybrid Gated Flow (HGF) is a dual-stream architecture that couples a 1.58-bit ternary backbone with a learnable, low-rank FP16 correction path.<n>We show that HGF 5.4 achieves a validation loss of 0.9306 compared to BitNet's 1.0294, recovering approximately 55% of the quality gap between pure ternary quantization and the FP16 baseline.
  arXiv  Detail & Related papers  (2026-02-05T03:47:17Z)
• RefineBench: Evaluating Refinement Capability of Language Models via Checklists [71.02281792867531]
  We evaluate two refinement modes: guided refinement and self-refinement.<n>In guided refinement, both proprietary LMs and large open-weight LMs can leverage targeted feedback to refine responses to near-perfect levels within five turns.<n>These findings suggest that frontier LMs require breakthroughs to self-refine their incorrect responses.
  arXiv  Detail & Related papers  (2025-11-27T07:20:52Z)
• CBAM-SwinT-BL: Small Rail Surface Defect Detection Method Based on Swin Transformer with Block Level CBAM Enhancement [2.48550866840644]
  This study utilizes Swin Transformer (SwinT) as baseline and incorporates the Convolutional Block Attention Module (CBAM) for enhancement. Our proposed method integrates CBAM within the swin transformer blocks, resulting in significant performance improvement in rail defect detection. Experiment and ablation study have proven the effectiveness of the framework.
  arXiv  Detail & Related papers  (2024-09-30T09:11:13Z)
• Towards Accurate Post-Training Quantization for Vision Transformer [48.779346466374406]
  Existing post-training quantization methods still cause severe performance drops. APQ-ViT surpasses the existing post-training quantization methods by convincing margins.
  arXiv  Detail & Related papers  (2023-03-25T03:05:26Z)
• DeepNet: Scaling Transformers to 1,000 Layers [106.33669415337135]
  We introduce a new normalization function (DeepNorm) to modify the residual connection in Transformer. In-depth theoretical analysis shows that model updates can be bounded in a stable way. We successfully scale Transformers up to 1,000 layers without difficulty, which is one order of magnitude deeper than previous deep Transformers.
  arXiv  Detail & Related papers  (2022-03-01T15:36:38Z)
• Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers [57.931830650323]
  This paper presents scaling insights from pretraining and finetuning Transformers. We show that aside from only the model size, model shape matters for downstream fine-tuning. We present improved scaling protocols whereby our redesigned models achieve similar downstream fine-tuning quality.
  arXiv  Detail & Related papers  (2021-09-22T12:29:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.