Prompt Compression via Activation Aggregation

• URL: http://arxiv.org/abs/2607.08399v1
• Date: Thu, 09 Jul 2026 12:21:44 GMT
• Title: Prompt Compression via Activation Aggregation
• Abstract summary: Large language models process prompts by propagating activations through dozens of layers before generating a response.<n>We ask whether the task-relevant information contained in an instruction prompt can be compressed into a single activation vector and re-injected into the model.<n>We show this is achievable using a learned weighted sum of activations extracted at an intermediate layer and injected at an early layer of the target LLM.
• Score: 2.739231894601598
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models process prompts by propagating activations through dozens of layers before generating a response. We ask whether the task-relevant information contained in an instruction prompt can be compressed into a single activation vector and re-injected into the model, replacing the original token sequence? We show this is achievable using a learned weighted sum of activations extracted at an intermediate layer and injected at an early layer of the target LLM. The compressed vector preserves task-relevant information, incurring an accuracy drop of under $2\%$ relative to full prompt processing. Beyond its practical implications, including reducing per-query computation for fixed instruction prompts without reprocessing the original token sequence, our analysis reveals structure in the activation space of LLMs: (i) mid-layer representations transfer meaningfully to early layers, suggesting a degree of cross-layer compatibility in how information is encoded; (ii) a single activation vector encodes a quantifiable and recoverable amount of semantic information; (iii) a weighted sum of activations is a robust representation compressor.

Related papers

• A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Models [56.03096341469333]
  We investigate the origins of massive activations in large language models (LLMs)<n>We identify a specific layer named the textbfMassive Emergence Layer (ME Layer), that is consistently observed across model families.<n>We show that, within the ME Layer both the RMSNorm and the FFN parameters jointly contribute to the emergence of massive activations.<n>Motivated by this limitation, we propose a simple and effective method to reduce the rigidity of the massive activation token.
  arXiv  Detail & Related papers  (2026-05-08T21:37:27Z)
• MiA-Signature: Approximating Global Activation for Long-Context Understanding [48.9604042895266]
  We introduce the concept of textbfMindscape Activation Signature (MiA-Signature), a compressed representation of the global activation pattern induced by a query.<n>The resulting MiA-Signature serves as a conditioning signal that approximates the effect of the full activation state while remaining computationally tractable.
  arXiv  Detail & Related papers  (2026-05-07T15:25:02Z)
• Detecting Overflow in Compressed Token Representations for Retrieval-Augmented Generation [49.48204107529758]
  We define token overflow as a regime in which compressed representations no longer contain sufficient information to answer a given query.<n>In this paper, we find that query-agnostic saturation statistics reliably separate compressed from uncompressed token representations.<n>Lightweight probing classifiers over both query and context xRAG representations detect overflow with 0.72 AUC-ROC on average.<n>These results advance from query-independent diagnostics to query-aware detectors, enabling low-cost pre-LLM gating to mitigate compression-induced errors.
  arXiv  Detail & Related papers  (2026-02-12T18:15:08Z)
• To Copy or Not to Copy: Copying Is Easier to Induce Than Recall [5.057026826740146]
  Language models must arbitrate between parametric knowledge stored in their weights and contextual information in the prompt.<n>This work presents a mechanistic study of that choice by extracting an empharbitration vector from model activations on a curated dataset.
  arXiv  Detail & Related papers  (2026-01-17T14:46:29Z)
• Behavior-Equivalent Token: Single-Token Replacement for Long Prompts in LLMs [55.827877498548965]
  We propose a lightweight training framework that learns a single prompt-specific Behavior-Equivalent token ([BE])<n>The framework first trains [BE] to encode the natural-language content of the original system prompt via reconstruction, and then distills the prompt's downstream behavior into this single token.<n> Empirical evaluations on three datasets show that a single [BE] token achieves up to a 3000x reduction in prompt length, while retaining about 98% of the downstream performance of the original system prompts.
  arXiv  Detail & Related papers  (2025-11-28T15:22:52Z)
• Direct Multi-Token Decoding [24.347862297812977]
  We introduce Direct Multi-Token Decoding (DMTD) as an inference paradigm for large language models (LLMs)<n>Unlike speculative decoding, our method introduces no additional parameters, auxiliary routines, or post-generation verification.<n>A fine-tuned DMTD Qwen3-4B model has already demonstrated promising results, achieving up to a 2x speedup with only minor performance loss.
  arXiv  Detail & Related papers  (2025-10-13T21:42:37Z)
• Command-V: Pasting LLM Behaviors via Activation Profiles [67.07238260037839]
  Command-V is a backpropagation-free behavior transfer method.<n>It copies an existing residual activation adapter from a donor model and pastes its effect into a recipient model.
  arXiv  Detail & Related papers  (2025-06-23T21:21:49Z)
• GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignment [18.256369876037883]
  This paper introduces GMSA, a context compression framework based on the encoder-decoder architecture.<n>GMSA reduces input sequence length and redundant information.<n>It can achieve approximately a 2x speedup in end-to-end inference.
  arXiv  Detail & Related papers  (2025-05-18T03:21:30Z)
• Semantic Compression With Large Language Models [1.0874100424278175]
  Large language models (LLMs) are revolutionizing information retrieval, question answering, summarization, and code generation tasks. LLMs are inherently limited by the number of input and output tokens that can be processed at once. This paper presents three contributions to research on LLMs.
  arXiv  Detail & Related papers  (2023-04-25T01:47:05Z)
• MASTER: Multi-task Pre-trained Bottlenecked Masked Autoencoders are Better Dense Retrievers [140.0479479231558]
  In this work, we aim to unify a variety of pre-training tasks into a multi-task pre-trained model, namely MASTER. MASTER utilizes a shared-encoder multi-decoder architecture that can construct a representation bottleneck to compress the abundant semantic information across tasks into dense vectors.
  arXiv  Detail & Related papers  (2022-12-15T13:57:07Z)
• Representation Learning for Sequence Data with Deep Autoencoding Predictive Components [96.42805872177067]
  We propose a self-supervised representation learning method for sequence data, based on the intuition that useful representations of sequence data should exhibit a simple structure in the latent space. We encourage this latent structure by maximizing an estimate of predictive information of latent feature sequences, which is the mutual information between past and future windows at each time step. We demonstrate that our method recovers the latent space of noisy dynamical systems, extracts predictive features for forecasting tasks, and improves automatic speech recognition when used to pretrain the encoder on large amounts of unlabeled data.
  arXiv  Detail & Related papers  (2020-10-07T03:34:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.