RAMP: Reinforcement Adaptive Mixed Precision Quantization for Efficient On Device LLM Inference

• URL: http://arxiv.org/abs/2603.17891v1
• Date: Wed, 18 Mar 2026 16:16:28 GMT
• Title: RAMP: Reinforcement Adaptive Mixed Precision Quantization for Efficient On Device LLM Inference
• Abstract summary: RAMP (Reinforcement Adaptive Mixed Precision) learns per layer bit width assignments to minimize perplexity under a global bit budget.<n>On Llama 2 7B, RAMP achieves 5.54 perplexity at 3.68GB (3.65 effective bits), outperforming uniform 4 bit AWQ (5.60 at 3.90 GB) and GPTQ by 6% in size and 1% to3% in quality.
• Score: 1.1100764382749708
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Post training quantization is essential for deploying large language models (LLMs) on resource constrained hardware, yet state of the art methods enforce uniform bit widths across layers, yielding suboptimal accuracy efficiency trade offs. We present RAMP (Reinforcement Adaptive Mixed Precision), an off policy Soft Actor Critic framework that learns per layer bit width assignments to minimize perplexity under a global bit budget. The policy conditions on an 11 dimensional embedding of activation statistics, weight properties, and structural descriptors, enabling zero shot transfer across model families and scales. To enable stable sub 4 bit quantization, we introduce Scale Folding, a preconditioning technique that migrates activation outliers into weights via per channel scaling and normalization layer compensation. A quality prioritized reward with asymmetric penalties and budget cliffs drives rapid convergence. On Llama 2 7B, RAMP achieves 5.54 perplexity at 3.68GB (3.65 effective bits), outperforming uniform 4 bit AWQ (5.60 at 3.90 GB) and GPTQ by 6% in size and 1% to3% in quality. Critically, a policy trained only on Llama 2 7B generalizes zero shot to Llama 2 13B and Mistral 7B, often surpassing target specific training, supporting the hypothesis that quantization sensitivity is primarily architectural. The HALO pipeline exports allocations to GGUF format for kernel free inference on CPUs, GPUs, and edge devices, retaining 99.5% of FP16 commonsense reasoning performance.

Related papers

• Scaling Post-Training Ternarisation to Qwen3-8B Capability Retention, Reproduction, Lossless Packing, and Packed Execution [0.0]
  We study a scale-up of an aggressive post-training conversion pipeline from Qwen3-4B to Qwen3-8B.<n>The conversion uses KOTMS rotation, E2M-ATQ adaptive ternarisation, and GPTQ-style error compensation in a weight-only A16 configuration.
  arXiv  Detail & Related papers  (2026-09-08T06:21:44Z)
• APQF: Agentic Profiling-Guided Structured Pruning and Mixed-Precision Quantization with Adaptive Fine-Tuning [12.834835889446177]
  APQF is a profiling-guided framework for deep neural networks.<n>It combines structured pruning, mixed-precision quantization-aware training, and accuracy recovery in one automated pipeline.<n>We evaluate it on ResNet, VGG7, ViT, DeiT, and Swin using ImageNet-1k and CIFAR-10.
  arXiv  Detail & Related papers  (2026-08-06T01:09:50Z)
• LoRaQ: Optimized Low Rank Approximation for 4-bit Quantization [39.25504627568235]
  Post-training quantization (PTQ) is essential for deploying large diffusion transformers on resource-constrained hardware.<n>LoRaQ (Low-Rank Approximated Quantization) is a simple, data-free calibration approach that optimize quantization error compensation.<n>We demonstrate that LoRaQ outperforms the state-of-the-art methods in their native implementations on Pixart-$$ and SANA.
  arXiv  Detail & Related papers  (2026-04-20T11:37:10Z)
• BWTA: Accurate and Efficient Binarized Transformer by Algorithm-Hardware Co-design [71.97035034203275]
  We analyze zero-point distortion in binarization and propose a Binary Weights & Ternary Activations (BWTA) quantization scheme.<n>For training, we propose Smooth Multi-Stage Quantization, combining a Levelwise Degradation Strategy and a MagnitudeAlignment Projection Factor.<n>Experiments show that BWTA approaches full-precision performance for Transformer-based models, with an average 3.5% drop on GLUE and less than 2% drop on five tasks.
  arXiv  Detail & Related papers  (2026-04-05T04:25:07Z)
• Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation [46.34608916687127]
  Low-Rank Decomposed Scaling (LoRDS) is a unified framework that rethinks quantization granularity through this low-rank decomposition.<n>By "breaking the blocks" of spatial constraints, LoRDS establishes a seamless efficiency lifecycle.<n>LoRDS consistently outperforms state-of-the-art baselines across various model families in both quantization and downstream fine-tuning tasks.
  arXiv  Detail & Related papers  (2026-01-30T08:46:02Z)
• Exploring Layer-wise Information Effectiveness for Post-Training Quantization in Small Language Models [4.238165821317982]
  LieQ is a metric-driven framework that addresses the challenge of maintaining accuracy in sub-7B models under extreme low-bit compression.<n>Our method introduces three complementary layer-wise diagnostics-Perplexity Drop, Representational Compactness, and Top-k Energy Gain.<n>On Qwen3-4B, LieQ recovers 95.9% of FP16 baseline performance at 2.05-bit quantization, outperforming GPTQ by 19.7% and AWQ by 18.1% on average across seven zero-shot reasoning tasks.
  arXiv  Detail & Related papers  (2025-08-05T11:17:04Z)
• FireQ: Fast INT4-FP8 Kernel and RoPE-aware Quantization for LLM Inference Acceleration [1.6127639408026697]
  FireQ is a co-designed PTQ framework and an INT4-FP8 matrix multiplication kernel.<n>FireQ quantizes linear layer weights and key-values to INT4, and activations and queries to FP8.<n>Three-stage pipelining for the prefill phase reduces time-to-first-token in the prefill phase.
  arXiv  Detail & Related papers  (2025-05-27T07:58:35Z)
• EfficientLLM: Efficiency in Large Language Models [64.3537131208038]
  Large Language Models (LLMs) have driven significant progress, yet their growing counts and context windows incur prohibitive compute, energy, and monetary costs.<n>We introduce EfficientLLM, a novel benchmark and the first comprehensive empirical study evaluating efficiency techniques for LLMs at scale.
  arXiv  Detail & Related papers  (2025-05-20T02:27:08Z)
• Optimizing Large Language Model Training Using FP4 Quantization [73.55459961002371]
  Quantized training presents a promising solution by enabling low-bit arithmetic operations to reduce costs.<n>This work introduces the first FP4 training framework for large language models (LLMs)
  arXiv  Detail & Related papers  (2025-01-28T18:04:50Z)
• GAQAT: gradient-adaptive quantization-aware training for domain generalization [54.31450550793485]
  We propose a novel Gradient-Adaptive Quantization-Aware Training (GAQAT) framework for DG.<n>Our approach begins by identifying the scale-gradient conflict problem in low-precision quantization.<n>Extensive experiments validate the effectiveness of the proposed GAQAT framework.
  arXiv  Detail & Related papers  (2024-12-07T06:07:21Z)
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization [67.74400574357472]
  LLMs are seeing growing use for applications which require large context windows, and with these large context windows KV cache activations surface as the dominant contributor to memory consumption during inference.<n> Quantization is a promising approach for compressing KV cache activations; however, existing solutions fail to represent activations accurately in sub-4-bit precision.<n>Our work, KVQuant, facilitates low precision KV cache quantization by incorporating several novel methods.
  arXiv  Detail & Related papers  (2024-01-31T18:58:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.