Lillama: Large Language Models Compression via Low-Rank Feature Distillation

• URL: http://arxiv.org/abs/2412.16719v2
• Date: Sat, 28 Dec 2024 17:45:12 GMT
• Title: Lillama: Large Language Models Compression via Low-Rank Feature Distillation
• Authors: Yaya Sy, Christophe Cerisara, Irina Illina,
• Abstract summary: Lillama is a compression method that distills activations with low-rank weights.<n>It compresses Mixtral-8x7B within minutes on a single A100 GPU, removing 10 billion parameters while retaining over 95% of its original performance.<n>It generalizes well to non-transformer architectures, compressing Mamba-3B by 20% while maintaining 99% performance.
• Score: 8.090496457850852
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current LLM structured pruning methods typically involve two steps: (1) compression with calibration data and (2) costly continued pretraining on billions of tokens to recover lost performance. This second step is necessary as the first significantly impacts model accuracy. Prior research suggests pretrained Transformer weights aren't inherently low-rank, unlike their activations, which may explain this drop. Based on this observation, we propose Lillama, a compression method that locally distills activations with low-rank weights. Using SVD for initialization and a joint loss combining teacher and student activations, we accelerate convergence and reduce memory use with local gradient updates. Lillama compresses Mixtral-8x7B within minutes on a single A100 GPU, removing 10 billion parameters while retaining over 95% of its original performance. Phi-2 3B can be compressed by 40% with just 13 million calibration tokens, resulting in a small model that competes with recent models of similar size. The method generalizes well to non-transformer architectures, compressing Mamba-3B by 20% while maintaining 99% performance.

Related papers

• 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float [71.43026659686679]
  Large Language Models (LLMs) have grown rapidly in size, creating challenges for efficient deployment on resource-constrained hardware. We introduce Dynamic-Length Float (DFloat11), a compression framework that reduces LLM size by 30% while preserving outputs that are bit-for-bit identical to the original model.
  arXiv  Detail & Related papers  (2025-04-15T22:38:38Z)
• Compression Laws for Large Language Models [20.62274005080048]
  We introduce compression laws for language language models (LLMs) We empirically examine the effects of structured model compression on LLMs through over $1000$ experiments. Our findings indicate that the test cross-entropy loss increases quadratically with the compression ratio.
  arXiv  Detail & Related papers  (2025-04-06T03:39:34Z)
• X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compression [23.023849840907594]
  Multi-head latent attention (MLA) is designed to optimize KV cache memory through low-rank key-value joint compression. We show that our proposed method can effectively compress the KV cache while preserving the performance on the benchmarks.
  arXiv  Detail & Related papers  (2025-03-14T06:49:37Z)
• DeltaLLM: Compress LLMs with Low-Rank Deltas between Shared Weights [11.047879241587315]
  We introduce DeltaLLM, a new post-training compression technique to reduce the memory footprint of LLMs. For training, we adopt the progressing module replacement method and show that the lightweight training of the low-rank modules is sufficient to achieve performance on par with LLMs of comparable sizes trained from scratch. Our method also outperforms compression techniques JointDrop, LaCo, ShortGPT and SliceGPT with the same number of parameters removed.
  arXiv  Detail & Related papers  (2025-01-30T18:59:55Z)
• MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection [14.073722038551125]
  KV cache has become a de facto technique for the inference of large language models. This paper uses low-rank projection matrices to transform the cache features into spaces with reduced dimensions. We find that our method can sustain over 90% performance with an average KV cache compression rate of 60%.
  arXiv  Detail & Related papers  (2024-10-16T08:34:51Z)
• LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy [59.1298692559785]
  Key-Value ( KV) cache is crucial component in serving transformer-based autoregressive large language models (LLMs) Existing approaches to mitigate this issue include: (1) efficient attention variants integrated in upcycling stages; (2) KV cache compression at test time; and (3) KV cache compression at test time. We propose a low-rank approximation of KV weight matrices, allowing plug-in integration with existing transformer-based LLMs without model retraining. Our method is designed to function without model tuning in upcycling stages or task-specific profiling in test stages.
  arXiv  Detail & Related papers  (2024-10-04T03:10:53Z)
• Token Compensator: Altering Inference Cost of Vision Transformer without Re-Tuning [63.43972993473501]
  Token compression expedites the training and inference of Vision Transformers (ViTs) However, when applied to downstream tasks, compression degrees are mismatched between training and inference stages. We propose a model arithmetic framework to decouple the compression degrees between the two stages.
  arXiv  Detail & Related papers  (2024-08-13T10:36:43Z)
• From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients [86.40635601953446]
  We study the emergence of low-rank structures across different layers of Modern Large Language Models. We present Weight Low-Rank Projection (WeLore) that unifies weight compression and memory-efficient fine-tuning as ONE.
  arXiv  Detail & Related papers  (2024-07-15T21:05:20Z)
• TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance [97.01406871579525]
  We propose a novel cross-modal distillation method, called TinyCLIP, for large-scale language-image pre-trained models. We show that TinyCLIP can reduce the size of the pre-trained CLIP ViT-B/32 by 50%, while maintaining comparable zero-shot performance. Our TinyCLIP ViT-8M/16, trained on YFCC-15M, achieves an impressive zero-shot top-1 accuracy of 41.1% on ImageNet.
  arXiv  Detail & Related papers  (2023-09-21T17:59:53Z)
• DiffRate : Differentiable Compression Rate for Efficient Vision Transformers [98.33906104846386]
  Token compression aims to speed up large-scale vision transformers (e.g. ViTs) by pruning (dropping) or merging tokens. DiffRate is a novel token compression method that has several appealing properties prior arts do not have.
  arXiv  Detail & Related papers  (2023-05-29T10:15:19Z)
• Just CHOP: Embarrassingly Simple LLM Compression [27.64461490974072]
  Large language models (LLMs) enable unparalleled few- and zero-shot reasoning capabilities but at a high computational footprint. We show that simple layer pruning coupled with an extended language model pretraining produces state-of-the-art results against structured and even semi-structured compression of models at a 7B scale. We also show how distillation, which has been super effective in task-agnostic compression of smaller BERT-style models, becomes inefficient against our simple pruning technique.
  arXiv  Detail & Related papers  (2023-05-24T08:18:35Z)
• CrAM: A Compression-Aware Minimizer [103.29159003723815]
  We propose a new compression-aware minimizer dubbed CrAM that modifies the optimization step in a principled way. CrAM produces dense models that can be more accurate than the standard SGD/Adam-based baselines, but which are stable under weight pruning. CrAM can produce sparse models which perform well for transfer learning, and it also works for semi-structured 2:4 pruning patterns supported by GPU hardware.
  arXiv  Detail & Related papers  (2022-07-28T16:13:28Z)
• Compression-aware Continual Learning using Singular Value Decomposition [2.4283778735260686]
  We propose a compression based continual task learning method that can dynamically grow a neural network. Inspired by the recent model compression techniques, we employ compression-aware training and perform low-rank weight approximations. Our method achieves compressed representations with minimal performance degradation without the need for costly fine-tuning.
  arXiv  Detail & Related papers  (2020-09-03T23:29:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.