A general tensor-structured compression scheme for efficient large language models

• URL: http://arxiv.org/abs/2605.25344v1
• Date: Mon, 25 May 2026 02:00:41 GMT
• Title: A general tensor-structured compression scheme for efficient large language models
• Abstract summary: We present a general tensor-structured compression scheme that replaces targeted dense linear layers with executable mixtures of tensor operators.<n>MixT is potentially applicable across Transformer-based large language models (LLMs) and other dense neural mappings.<n>At the LLaMA2-7B transition boundary, MixT reduces full-model parameters by 47.5%, inference FLOPs by 37.1%, training FLOPs by 52.1% and peak inference memory by 60.4%.
• Score: 8.85443988406591
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are dominated by dense linear transformations, whose storage, memory and computational overheads hinder efficient adaptation and deployment while masking the functional impacts of structural simplification. Here we present Tensor Mixture (MixT), a general tensor-structured compression scheme that replaces targeted dense linear layers with natively executable mixtures of tensor operators. Operating directly on generic linear projections instead of model-specific components, MixT is potentially applicable across Transformer-based LLMs and other dense neural mappings. We evaluate MixT on Qwen3-8B and LLaMA2-7B under a unified recovery protocol, identifying a broad compressible regime in which MMLU accuracy is largely preserved before an abrupt transition at model-specific boundaries. This transition coincides with coordinated shifts in output entropy, prediction entropy and inter-layer geometry. At the LLaMA2-7B transition boundary, MixT reduces full-model parameters by 47.5\%, inference FLOPs by 37.1\%, training FLOPs by 52.1\% and peak inference memory by 60.4\%, demonstrating its practical potential for lower-cost LLM compression.

Related papers

• Simplifying Flow Matching Transformations with Low-Rank Mixture Models [58.797704398690165]
  We propose using mixtures of probabilistic principal component analyzers (MPPCA) as the latent density for normalizing flows.<n>We simplify the learned flow transformation by learning a latent distribution that more closely aligns with the data distribution in terms of KL divergence.<n>MPPCA models can be fit quickly and cheaply using the expectation-maximization algorithm, making them a practical choice for initializing latent distributions even in high-dimensional generative tasks.
  arXiv  Detail & Related papers  (2026-06-29T03:03:06Z)
• Fast Tensorization of Neural Networks via Slice-wise Feature Distillation [0.0]
  We propose a scalable tensorization framework for neural network compression based on slice-wise feature distillation.<n>We show significant gains over conventional global tensorization, achieving near-lossless compression at moderate compression rates.<n>Results on GPT-2 XL further demonstrate the scalability of the method and its applicability to large-scale models.
  arXiv  Detail & Related papers  (2026-05-19T13:37:53Z)
• Generalization and Scaling Laws for Mixture-of-Experts Transformers [0.0]
  We develop a theory of generalization and scaling for Mixture-of-Experts (MoE) Transformers.<n>We derive a sup-norm covering-number bound whose metric entropy scales with the active parameter budget and incurs a MoE-specific routing overhead.<n>We prove a constructive approximation theorem for MoE architectures, showing that, under the approximation construction, error can decrease either by scaling active capacity or by increasing the number of experts.
  arXiv  Detail & Related papers  (2026-04-10T09:59:48Z)
• Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers [0.0]
  We propose replacing the dense output projection in multi-head attention with a fixed, parameter-free Walsh Hadamard Transform.<n>Across different model sizes, we demonstrate that this structured substitution maintains comparable or slightly superior downstream task performance.<n>We observe that structured Hadamard-based models exhibit a steeper validation loss curve relative to training FLOPs, suggesting more favorable compute utilization during training.
  arXiv  Detail & Related papers  (2026-03-09T13:05:08Z)
• JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulation [46.64215658042213]
  We introduce Joint-Token (JTok) and Mixture of Joint-Token (JTok-M), which augment Transformer layers with modulation vectors retrieved from auxiliary embedding tables.<n>These vectors modulate the backbone via lightweight, element-wise operations, incurring negligible FLOPs overhead.<n>Our approach consistently reduces validation loss and significantly improves downstream task performance.
  arXiv  Detail & Related papers  (2026-01-31T16:15:18Z)
• WUSH: Near-Optimal Adaptive Transforms for LLM Quantization [52.77441224845925]
  Quantization to low bitwidth is a standard approach for deploying large language models.<n>A few extreme weights and activations stretch the dynamic range and reduce the effective resolution of the quantizer.<n>We derive, for the first time, closed-form optimal linear blockwise transforms for joint weight-activation quantization.
  arXiv  Detail & Related papers  (2025-11-30T16:17:34Z)
• MLPMoE: Zero-Shot Architectural Metamorphosis of Dense LLM MLPs into Static Mixture-of-Experts [0.0]
  Large Language Models (LLMs) are predominantly deployed as dense transformers, where every parameter in every feed-forward block is activated for every token.<n>Recent upcycling methods such as MoEfication, CMoE, ToMoE, and MoORE reveal that much of the useful computation lives in sparse, semi-modular substructures inside dense feed-forward networks.<n>This paper introducesMoE (MLP-Experts), a training-free transformation that restructures the dense in transformer blocks into a static, high-cardinality mixture of experts.
  arXiv  Detail & Related papers  (2025-11-26T06:14:26Z)
• MoEUT: Mixture-of-Experts Universal Transformers [75.96744719516813]
  Universal Transformers (UTs) have advantages over standard Transformers in learning compositional generalizations. Layer-sharing drastically reduces the parameter count compared to the non-shared model with the same dimensionality. No previous work has succeeded in proposing a shared-layer Transformer design that is competitive in parameter count-dominated tasks such as language modeling.
  arXiv  Detail & Related papers  (2024-05-25T03:24:32Z)
• SDPose: Tokenized Pose Estimation via Circulation-Guide Self-Distillation [53.675725490807615]
  We introduce SDPose, a new self-distillation method for improving the performance of small transformer-based models. SDPose-T obtains 69.7% mAP with 4.4M parameters and 1.8 GFLOPs, while SDPose-S-V2 obtains 73.5% mAP on the MSCOCO validation dataset.
  arXiv  Detail & Related papers  (2024-04-04T15:23:14Z)
• AffineQuant: Affine Transformation Quantization for Large Language Models [58.45460102764]
  Post-Training Quantization (PTQ) has emerged as a subject of considerable interest due to its compression efficiency and cost-effectiveness in the context of training. Existing PTQ methods for Large-scale Language Models (LLMs) limit the optimization scope to scaling transformations between pre- and post-quantization weights. In this paper, we advocate for the direct optimization using equivalent Affine transformations in PTQ (AffineQuant)
  arXiv  Detail & Related papers  (2024-03-19T08:40:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.