Measuring Maximum Activations in Open Large Language Models

• URL: http://arxiv.org/abs/2605.15572v1
• Date: Fri, 15 May 2026 03:31:51 GMT
• Title: Measuring Maximum Activations in Open Large Language Models
• Abstract summary: We measure global and layerwise maxima on 27 checkpoints from 8 open families spanning dense, MoE, vision-language, intermediate-training, and instruction-tuned variants.<n>We conclude that maximum activation magnitude is a model property tied to family, architecture, and training stage - not a simple byproduct of size.
• Score: 60.3514350516308
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The dynamic range of activations is a first-order constraint for low-bit quantization, activation scaling, and stable LLM inference. Prior work characterized outlier features and massive activations on pre-2024 LLaMA-style models, and the downstream activation-quantization stack inherits that picture without revisiting it for the post-LLaMA open-model boom. We ask the deployment-oriented question: how large can activations get in modern open LLMs, and how does this magnitude vary across families, generations, and training stages? Under a unified pipeline (5,000-sample multi-domain corpus, family-specific tokenization, identical hooks across embeddings, hidden states, attention, MLP/MoE, SwiGLU gates, and final norm), we measure global and layerwise maxima on 27 checkpoints from 8 open families spanning dense, MoE, vision-language, intermediate-training, and instruction-tuned variants. We find that (i) global maxima span over nearly four orders of magnitude at comparable parameter counts, with Qwen3.5 and MoE checkpoints in the 10^2 to 10^3 range and Gemma3-27B-it reaching ~7 x 10^5; (ii) cross-family and cross-generation comparisons break simple monotonic scaling; and (iii) MoE checkpoints exhibit 14.0-23.4x lower peaks than matched-scale dense counterparts, while the residual stream carries the global maximum in 22/24 checkpoints. A lightweight INT-8 sanity check shows that measured maxima co-vary with low-bit reconstruction error via activation-scale selection. We conclude that maximum activation magnitude is a model property tied to family, architecture, and training stage - not a simple byproduct of size - and should be measured and reported alongside any open-weight release before low-bit deployment. The code is publicly available at https://github.com/clx1415926/Max_act_llm.

Related papers

• Calibration is the Bottleneck: An Action-Class Diagnostic of Multi-Turn Tool-Calling [65.03665015319457]
  We propose an action-class-oriented diagnostic framework that decomposes multi-turn failures into two modes: action-class miscalibration and action-execution failure.<n>We validate it on a panel of tool-calling models across multiple multi-turn benchmarks.
  arXiv  Detail & Related papers  (2026-09-01T09:08:15Z)
• ActQuant: Sub-4-bit Action-Guided Quantization for Vision-Language-Action Models [45.62029693245481]
  We introduce ActQuant, an action-guided mixed-precision PTQ framework.<n>ActQuant operates in two stages: (1) an inter-tensor bit allocator that assigns each weight matrix a single bitwidth based on how much it contributes to predicting the agent's actions.<n>We also introduce OmniModel, an agentic conversion pipeline that ports into a native C/C++ with efficient low-bit kernels.
  arXiv  Detail & Related papers  (2026-05-19T19:57:26Z)
• Cross-Dialect Generalization Without Retraining: Benchmarks and Evaluation of Schema-Derived Constrained Decoding for MLIR [0.0]
  Multi-Level Intermediate Representation (MLIR) underlies modern ML compiler infrastructure (TensorFlow, JAX/StableHLO, PyTorch Inductor, IREE)<n>MLIR is also by design: new dialects ship per application domain, so a fine-tuned model per dialect does not scale.<n>We release four natural-language-to-MLIR benchmarks across three dialects.
  arXiv  Detail & Related papers  (2026-05-14T02:08:13Z)
• Human Values in a Single Sentence: Moral Presence, Hierarchies, and Transformer Ensembles on the Schwartz Continuum [3.0339361014383397]
  We study sentence-level detection of the 19 human values in the refined Schwartz continuum in about 74k English sentences.<n>Each sentence is annotated with value presence, yielding a binary moral-presence label.<n>We show that moral presence is learnable from single sentences.
  arXiv  Detail & Related papers  (2026-01-20T17:25:33Z)
• Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs [78.09559830840595]
  We present the first systematic study on quantizing diffusion-based language models.<n>We identify the presence of activation outliers, characterized by abnormally large activation values.<n>We implement state-of-the-art PTQ methods and conduct a comprehensive evaluation.
  arXiv  Detail & Related papers  (2025-08-20T17:59:51Z)
• Chain-of-Model Learning for Language Model [91.81240728426994]
  We propose a novel learning paradigm, termed Chain-of-Model (CoM), which incorporates the causal relationship into the hidden states of each layer as a chain style.<n>We introduce the concept of Chain-of-Representation (CoR), which formulates the hidden states at each layer as a combination of multiple sub-representations (i.e., chains) at the hidden dimension level.<n>Based on CoLM, we further introduce CoLM-Air by introducing a KV sharing mechanism, that computes all keys and values within the first chain and then shares across all chains.
  arXiv  Detail & Related papers  (2025-05-17T04:06:12Z)
• S*: Test Time Scaling for Code Generation [55.11863577956177]
  We propose S*, the first hybrid test-time scaling framework for code generation.<n>S* substantially improves the coverage and selection accuracy of generated code.
  arXiv  Detail & Related papers  (2025-02-20T09:18:53Z)
• The Uniqueness of LLaMA3-70B Series with Per-Channel Quantization [5.7672452948056545]
  Quantization is a crucial technique for deploying large language models (LLMs) efficiently. The impact of W8A8 post-training quantization on model accuracy remains contentious. We investigate what makes the LLaMA3-70B model series uniquely vulnerable to quantization.
  arXiv  Detail & Related papers  (2024-08-27T15:03:01Z)
• TernaryLLM: Ternarized Large Language Model [29.29122031050894]
  Large language models (LLMs) have achieved remarkable performance on Natural Language Processing (NLP) tasks. We introduce Dual Learnable Ternarization (DLT), which enables both scales and shifts to be learnable. We also propose Outlier-Friendly Feature Knowledge Distillation (OFF) to recover the information lost in extremely low-bit quantization.
  arXiv  Detail & Related papers  (2024-06-11T11:40:12Z)
• CLAQ: Pushing the Limits of Low-Bit Post-Training Quantization for LLMs [44.03692512352445]
  Column-Level Adaptive weight Quantization (CLAQ) is a novel and effective framework for Large Language Models (LLMs) quantization. In this paper, we present a novel and effective CLAQ framework by introducing three different types of adaptive strategies for LLM quantization. Experiments on various mainstream open source LLMs including LLaMA-1, LLaMA-2 and Yi demonstrate that our methods achieve the state-of-the-art results across different bit settings.
  arXiv  Detail & Related papers  (2024-05-27T14:49:39Z)
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization [67.74400574357472]
  LLMs are seeing growing use for applications which require large context windows, and with these large context windows KV cache activations surface as the dominant contributor to memory consumption during inference.<n> Quantization is a promising approach for compressing KV cache activations; however, existing solutions fail to represent activations accurately in sub-4-bit precision.<n>Our work, KVQuant, facilitates low precision KV cache quantization by incorporating several novel methods.
  arXiv  Detail & Related papers  (2024-01-31T18:58:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.