LLM Compression with Jointly Optimizing Architectural and Quantization choices

• URL: http://arxiv.org/abs/2606.04063v1
• Date: Tue, 02 Jun 2026 12:57:28 GMT
• Title: LLM Compression with Jointly Optimizing Architectural and Quantization choices
• Abstract summary: Large language models (LLMs) are challenging due to their significant memory and computational requirements.<n>We introduce a differentiable NAS framework that explores the entire space and jointly optimize architectural configurations.<n>Our models achieve up to 1.4x faster inference than sequential NAS-then-quantization baselines at comparable accuracy, or up to 6% higher average accuracy across seven reasoning tasks at equivalent latency.
• Score: 1.4857783122641302
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Deploying large language models (LLMs) is challenging due to their significant memory and computational requirements. While some methods address this by developing small or tiny language models from scratch, these approaches demand extensive GPU training. Compressing pre-trained LLMs for edge devices offers a compelling alternative. Beyond pruning and quantization, Neural Architecture Search (NAS) enables effective compression, yet prior NAS approaches often limit the search space and decouple architecture from quantization. We introduce a differentiable NAS framework that explores the entire space and jointly optimizes architectural configurations alongside mixed-precision quantization for linear layers of LLMs. Experiments demonstrate superior accuracy-latency trade-offs: our models achieve up to 1.4x faster inference than sequential NAS-then-quantization baselines at comparable accuracy, or up to 6% higher average accuracy across seven reasoning tasks at equivalent latency.

Related papers

• PEL-NAS: Search Space Partitioned Architecture Prompt Co-Evolutionary LLM-driven Hardware-Aware Neural Architecture Search [3.59587275290785]
  Hardware-Aware Neural Architecture Search (HW-NAS) requires joint optimization of accuracy and latency under device constraints.<n>We propose PEL-NAS: a search space Partitioned, architecture prompt co-Evolutionary and LLM-driven Neural Architecture Search.
  arXiv  Detail & Related papers  (2025-10-01T21:29:20Z)
• Flexiffusion: Training-Free Segment-Wise Neural Architecture Search for Efficient Diffusion Models [50.260693393896716]
  Diffusion models (DMs) are powerful generative models capable of producing high-fidelity images but constrained by high computational costs.<n>We propose Flexiffusion, a training-free NAS framework that jointly optimize generation schedules and model architectures without modifying pre-trained parameters.<n>Our work pioneers a resource-efficient paradigm for searching high-speed DMs without sacrificing quality.
  arXiv  Detail & Related papers  (2025-06-03T06:02:50Z)
• Compressing Large Language Models with Automated Sub-Network Search [41.452512557226335]
  We consider model compression for Large Language Models to reduce model size while improving downstream task performance.<n>We phrase this as a neural architecture search problem that automatically prunes structural components.<n>Our method achieves upto 9.85% improvement on average on 11 diverse downstream tasks, while achieving up to 22% improvement of on-device latency.
  arXiv  Detail & Related papers  (2024-10-09T02:14:39Z)
• Search for Efficient Large Language Models [52.98684997131108]
  Large Language Models (LLMs) have long held sway in the realms of artificial intelligence research. Weight pruning, quantization, and distillation have been embraced to compress LLMs, targeting memory reduction and inference acceleration. Most model compression techniques concentrate on weight optimization, overlooking the exploration of optimal architectures.
  arXiv  Detail & Related papers  (2024-09-25T21:32:12Z)
• LLaMA-NAS: Efficient Neural Architecture Search for Large Language Models [3.4070166891274263]
  Large language models (LLMs) solve natural language processing, complex reasoning, sentiment analysis and other tasks. These abilities come with very high memory and computational costs which precludes the use of LLMs on most hardware platforms. We propose an effective method of finding Pareto-optimal network architectures based on LLaMA2-7B using one-shot NAS. We show that, for certain standard benchmark tasks, the pre-trained LLaMA2-7B network is unnecessarily large and complex.
  arXiv  Detail & Related papers  (2024-05-28T17:20:44Z)
• LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit [55.73370804397226]
  Quantization, a key compression technique, can effectively mitigate these demands by compressing and accelerating large language models. We present LLMC, a plug-and-play compression toolkit, to fairly and systematically explore the impact of quantization. Powered by this versatile toolkit, our benchmark covers three key aspects: calibration data, algorithms (three strategies), and data formats.
  arXiv  Detail & Related papers  (2024-05-09T11:49:05Z)
• Multi-objective Differentiable Neural Architecture Search [58.67218773054753]
  We propose a novel NAS algorithm that encodes user preferences to trade-off performance and hardware metrics.<n>Our method outperforms existing MOO NAS methods across a broad range of qualitatively different search spaces and datasets.
  arXiv  Detail & Related papers  (2024-02-28T10:09:04Z)
• Combining Multi-Objective Bayesian Optimization with Reinforcement Learning for TinyML [4.2019872499238256]
  We propose a novel strategy for deploying deep neural networks on microcontrollers (TinyML) based on multi-objective Bayesian optimization (MOBOpt)<n>Our methodology aims at efficiently finding tradeoffs between a DNN's predictive accuracy, memory requirements on a given target system, and computational complexity.
  arXiv  Detail & Related papers  (2023-05-23T14:31:52Z)
• Lightweight Neural Architecture Search for Temporal Convolutional Networks at the Edge [21.72253397805102]
  This work focuses in particular on Temporal Convolutional Networks (TCNs), a convolutional model for time-series processing. We propose the first NAS tool that explicitly targets the optimization of the most peculiar architectural parameters of TCNs. We test the proposed NAS on four real-world, edge-relevant tasks, involving audio and bio-signals.
  arXiv  Detail & Related papers  (2023-01-24T19:47:40Z)
• BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule [95.56873042777316]
  Differentiable Architecture Search (DARTS) has received massive attention in recent years, mainly because it significantly reduces the computational cost. This paper formulates the neural architecture search as a distribution learning problem through relaxing the architecture weights into Gaussian distributions. We demonstrate how the differentiable NAS benefits from Bayesian principles, enhancing exploration and improving stability.
  arXiv  Detail & Related papers  (2021-11-25T18:13:42Z)
• NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search [100.71365025972258]
  We propose NAS-BERT, an efficient method for BERT compression. NAS-BERT trains a big supernet on a search space and outputs multiple compressed models with adaptive sizes and latency. Experiments on GLUE and SQuAD benchmark datasets demonstrate that NAS-BERT can find lightweight models with better accuracy than previous approaches.
  arXiv  Detail & Related papers  (2021-05-30T07:20:27Z)
• Geometry-Aware Gradient Algorithms for Neural Architecture Search [41.943045315986744]
  We argue for the study of single-level empirical risk minimization to understand NAS with weight-sharing. We present a geometry-aware framework that exploits the underlying structure of this optimization to return sparse architectural parameters. We achieve state-of-the-art accuracy on the latest NAS benchmarks in computer vision.
  arXiv  Detail & Related papers  (2020-04-16T17:46:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.