LegoLM: Structured Weight Sharing for Large Language Models

• URL: http://arxiv.org/abs/2608.08652v1
• Date: Sun, 09 Aug 2026 11:51:14 GMT
• Title: LegoLM: Structured Weight Sharing for Large Language Models
• Abstract summary: We present LegoLM, a structured weight-sharing compression framework for large language models.<n>LegoLM resolves two failure modes: distributional mismatch and outlier dominance.<n>Across GPT-2 small (124M) and Mistral-7B, LegoLM achieves +0.03% PPL degradation at 4.41X compression on Mistral-7B.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present \LegoLM{}, a structured weight-sharing compression framework for large language models grounded in a systematic study of why global weight sharing fails and how to fix it. We identify two distinct failure modes. Distributional mismatch: for vector blocks of dimension d <= 2, transformer layers with heterogeneous weight scales impose a scale-mismatch penalty that grows linearly with d and cannot be resolved by increasing K, producing perplexity in the millions.Outlier dominance: for scalar blocks, a fraction ~1/K of weights lies beyond the outermost Lloyd-Max decision threshold and cannot be represented by any centroid; their misrepresentation accumulates across layers, causing catastrophic quality loss. \LegoLM{} resolves both failure modes via three data-free adaptations: 1 scalar-block encoding to eliminate the $d$-linear mismatch component, 2 percentile-selective replacement that identifies and preserves outlier weights verbatim, and 3 boundary-layer protection for the first and last transformer blocks. Across GPT-2 small (124M) and Mistral-7B, \LegoLM{} achieves +0.03% PPL degradation at 4.41X compression on Mistral-7B - outperforming PTQ-8bit in both quality and compression ratio - and -0.02% at 2.67X. Downstream evaluation on LAMBADA and HellaSwag confirms that \LegoLM{} at K=64, p=99% preserves accuracy within noise at 5.12 X compression, exceeding PTQ-8bit's compression ratio while matching its accuracy. We further discover that outlier dominance grows with model scale: full replacement at K=128 degrades GPT-2 small by only +23% but catastrophically degrades Mistral-7B by +1,134,279%, while selective replacement at p=99% rescues both models to under +15%. A controlled ablation confirms that selective replacement is the dominant mechanism: adding it to per-layer K-means also yields near-lossless quality, matching \LegoLM{} within 0.02%.

Related papers

• Deterministic LLM Inference Across GPU Kernels: Power-of-Two INT8 Quantization Scales and the Limits of Tolerance-Based Conformance [0.10152838128195468]
  Conformance suites for quantized GEMM kernels ask whether two implementations agree within a tolerance.<n>We find that every one of five epilogue faults -- scale precision, double rounding, multiplication order, output truncation, fused ordering -- moves the output by at most a single bfloat16 spacing.<n>Four of the five faults are detected by no check in the suite, and the fifth only under power-of-two scales.
  arXiv  Detail & Related papers  (2026-08-25T03:05:54Z)
• Bounded Precision-Geometry Scaling for Robust Multi-Task Learning under Loss Scale Mismatch [0.0]
  Multi-task learning often combines losses that span several orders of magnitude, causing homoscedastic uncertainty weighting to degrade severely.<n>We propose Bounded Precision-Geometry Scaling (BPGS), a method that maps each task's log-variance through a bounded sigmoid parameterisation anchored to detached batch loss statistics.<n>We evaluate BPGS on synthetic stress tests and three real-world benchmarks: NYUv2 dense prediction, Yeast multi-label classification, and RF1 multi-target regression.
  arXiv  Detail & Related papers  (2026-08-21T21:48:51Z)
• Wiring Beats Blending: What Transfers Between Transformer Sizes -- and What Doesn't [0.0]
  We characterize the 1.4B->410M conversion in the Pythia family end-to-end.<n>We decompose conversion into two independent levers -- least-squares compensation and variance-preserving rescale.<n>Compensation is a token-efficient, low-budget win rather than a universal one.
  arXiv  Detail & Related papers  (2026-08-03T19:44:08Z)
• Modeling Normal Is All You Need: Joint Latent Clustering for Anomaly Detection in Multimodal Cyber-Physical Systems [47.027290803102666]
  Normal behaviour is the union of many imbalanced, curved, thin-fringed operating regimes.<n>We model the normal law with a jointly learned latent representation plus explicit Gaussian-mixture mode clustering.<n>We evaluate under a deliberately fair protocol: raw point-wise metrics with no point adjustment, a trivial-detector difficulty split, prevalence-matched F1, and train-normal-only calibration.
  arXiv  Detail & Related papers  (2026-07-07T10:10:00Z)
• Layer-Isolated Evaluation: Gating the Deterministic Scaffold of a Production LLM Agent with a No-LLM, Regression-Locked Test Harness [45.148328075418156]
  A deployed ordering agent is decomposed into a fixed taxonomy of layers.<n>The pure suite runs in CI on every change against a locked per-slice baseline.<n>We validate by controlled regression injection, degrading one layer at a time across seven non-safety layers.
  arXiv  Detail & Related papers  (2026-06-10T05:55:13Z)
• Normalized Architectures are Natively 4-Bit [49.13186675123547]
  nGPT, an architecture that constrains weights and hidden representations to the unit hypersphere, is inherently more robust to low-precision arithmetic.<n>We validate this approach on both a 1.2B dense model and hybrid (Mamba-Transformer) MoE models of up to 3B/30B parameters.
  arXiv  Detail & Related papers  (2026-05-07T11:54:07Z)
• Predicting LLM Compression Degradation from Spectral Statistics [2.76240219662896]
  Matrix-level low-rank compression is a promising way to reduce the cost of large language models.<n>Can compression-induced degradation be predicted before committing to this compute?<n>We find that stable rank and information density, measured in bits per parameter, dominate performance degradation.
  arXiv  Detail & Related papers  (2026-04-20T11:01:19Z)
• Quantization Dominates Rank Reduction for KV-Cache Compression [0.0]
  quantization consistently outperforms rank reduction by 4-364 PPL depending on model and compression level.<n>We formalize this via a result showing projection damage exceeds quantization damage by 3 x 2 (2b) per direction under the softmax Fisher metric.
  arXiv  Detail & Related papers  (2026-04-13T14:06:18Z)
• Compressing Transformer Language Models via Matrix Product Operator Decomposition: A Case Study on PicoGPT [0.0]
  Transformer-based language models achieve strong performance across NLP tasks, but their quadratic parameter scaling with hidden dimension makes deployment on resource-constrained hardware expensive.<n>We study Matrix Product Operator decomposition as a principled compression method for transformers.<n>MPO factorises weight matrices into chains of low-rank cores, with approximation quality controlled by the bond dimension chi.
  arXiv  Detail & Related papers  (2026-03-30T14:57:47Z)
• Only relative ranks matter in weight-clustered large language models [0.0]
  Large language models (LLMs) contain billions of parameters, yet many exact values are not essential.<n>We show that what matters most is the relative rank of weights-whether one connection is stronger or weaker than another-rather than precise magnitudes.<n>For Llama 3.1-8B-Instruct and SmolLM2-135M, reducing each matrix to only 16-64 distinct values preserves strong accuracy without retraining.
  arXiv  Detail & Related papers  (2026-03-18T16:55:13Z)
• LoTA-QAF: Lossless Ternary Adaptation for Quantization-Aware Fine-Tuning [50.89500210372827]
  Quantization and fine-tuning are crucial for deploying large language models (LLMs) on resource-constrained edge devices.<n>LoTA-QAF is a novel fine-tuning method specifically designed for quantized LLMs.<n>On the MMLU benchmark, our method effectively recovers performance for quantized models, surpassing 16-bit LoRA by up to 5.14%.
  arXiv  Detail & Related papers  (2025-05-24T14:47:28Z)
• Label Distributionally Robust Losses for Multi-class Classification: Consistency, Robustness and Adaptivity [55.29408396918968]
  We study a family of loss functions named label-distributionally robust (LDR) losses for multi-class classification. Our contributions include both consistency and robustness by establishing top-$k$ consistency of LDR losses for multi-class classification. We propose a new adaptive LDR loss that automatically adapts the individualized temperature parameter to the noise degree of class label of each instance.
  arXiv  Detail & Related papers  (2021-12-30T00:27:30Z)
• Training with Quantization Noise for Extreme Model Compression [57.51832088938618]
  We tackle the problem of producing compact models, maximizing their accuracy for a given model size. A standard solution is to train networks with Quantization Aware Training, where the weights are quantized during training and the gradients approximated with the Straight-Through Estimator. In this paper, we extend this approach to work beyond int8 fixed-point quantization with extreme compression methods.
  arXiv  Detail & Related papers  (2020-04-15T20:10:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.