AgentCompress: Task-Aware Compression for Affordable Large Language Model Agents

• URL: http://arxiv.org/abs/2601.05191v2
• Date: Mon, 12 Jan 2026 18:25:18 GMT
• Title: AgentCompress: Task-Aware Compression for Affordable Large Language Model Agents
• Abstract summary: A single session using a 70-billion- parameter model can cost around $127 in cloud computing fees.<n>We present AgentCompress, a framework that tackles this problem through task-aware dynamic compression.<n>Results show a 68.3% reduction in computational costs while preserving 96.2% of the original success rate.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models hold considerable promise for various applications, but their computational requirements create a barrier that many institutions cannot overcome. A single session using a 70-billion-parameter model can cost around $127 in cloud computing fees, which puts these tools out of reach for organizations operating on limited budgets. We present AgentCompress, a framework that tackles this problem through task-aware dynamic compression. The idea comes from a simple observation: not all tasks require the same computational effort. Complex reasoning, for example, is far more demanding than text reformatting, yet conventional compression applies the same reduction to both. Our approach uses a lightweight neural controller that looks at the first few tokens of each request, estimates how complex the task will be, and sends it to an appropriately quantized version of the model. This routing step adds only about 12 milliseconds of overhead. We tested the framework on 290 multi-stage workflows from domains including computer science, physics, chemistry, and biology. The results show a 68.3% reduction in computational costs while preserving 96.2% of the original success rate. These findings suggest that routing queries intelligently can make powerful language models substantially more affordable without sacrificing output quality

Related papers

• Large Models for Small Devices: Recent Advances and Empirical Analysis of Edge AI Deployment [4.582004238093247]
  Running large AI models on resource-constrained edge devices requires model compression to reduce model size and computation.<n>We deploy compact language and image models on GPU, CPU, and Raspberry Pi platforms across question answering and image segmentation.<n>We explain these effects through neural-flow graph analysis and prefill-decode-level latency decomposition.
  arXiv  Detail & Related papers  (2026-08-16T11:52:13Z)
• $π_{0.7}$: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities [77.01386332235107]
  $_0.7$ can enable strong out-of-the-box performance in a wide range of scenarios.<n>$_0.7$ can follow diverse language instructions in unseen environments.<n>$_0.7$ can also provide zero-shot cross-embodiment generalization.
  arXiv  Detail & Related papers  (2026-04-16T19:18:07Z)
• Trainable Bitwise Soft Quantization for Input Feature Compression [0.7559720049837458]
  We propose a task-specific, trainable feature quantization layer that compresses the input features of a neural network.<n>This can significantly reduce the amount of data that needs to be transferred from the device to a remote server.
  arXiv  Detail & Related papers  (2026-03-05T13:40:55Z)
• Stacked from One: Multi-Scale Self-Injection for Context Window Extension [69.24689919827817]
  modelname is a novel framework based on multi-grained context compression and query-aware information acquisition.<n>modelnameachieves performance superior or comparable to strong baselines.
  arXiv  Detail & Related papers  (2026-03-05T03:16:16Z)
• Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference [0.0]
  "Pyramid MoA" is a hierarchical Mixture-of-Agents architecture that uses a lightweight Router to dynamically escalate queries only when necessary.<n>We demonstrate that the system introduces negligible latency overhead (+0.82s) and allows for a tunable trade-off between performance and budget.
  arXiv  Detail & Related papers  (2026-02-23T04:47:47Z)
• Scaling Up Efficient Small Language Models Serving and Deployment for Semantic Job Search [12.891315513255527]
  Large Language Models (LLMs) have demonstrated impressive quality when applied to predictive tasks such as relevance ranking and semantic search.<n>However, deployment of such LLMs remains prohibitively expensive for industry applications with strict latency and throughput requirements.<n>We present lessons and efficiency insights from developing a purely text-based decoder-only Small Language Model (SLM) for a semantic search application at LinkedIn.
  arXiv  Detail & Related papers  (2025-10-25T00:56:06Z)
• Compressing Many-Shots in In-Context Learning [61.231471139896506]
  We study an approach to improve the memory and computational efficiency of ICL inference by compressing the many-shot prompts.<n>We first show that existing prompt compression methods are ineffective for many-shot compression.<n>We propose MemCom, a layer-wise compression method.
  arXiv  Detail & Related papers  (2025-10-17T16:57:42Z)
• Mitigating Copy Bias in In-Context Learning through Neuron Pruning [74.91243772654519]
  Large language models (LLMs) have demonstrated impressive few-shot in-context learning abilities. They are sometimes prone to a copying bias', where they copy answers from provided examples instead of learning the underlying patterns. We propose a novel and simple method to mitigate such copying bias.
  arXiv  Detail & Related papers  (2024-10-02T07:18:16Z)
• Language models scale reliably with over-training and on downstream tasks [121.69867718185125]
  Scaling laws are useful guides for derisking expensive training runs. However, there remain gaps between current studies and how language models are trained. In contrast, scaling laws mostly predict loss on inference, but models are usually compared on downstream task performance.
  arXiv  Detail & Related papers  (2024-03-13T13:54:00Z)
• MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation [96.71370747681078]
  We introduce MLAgentBench, a suite of 13 tasks ranging from improving model performance on CIFAR-10 to recent research problems like BabyLM. For each task, an agent can perform actions like reading/writing files, executing code, and inspecting outputs. We benchmark agents based on Claude v1.0, Claude v2.1, Claude v3 Opus, GPT-4, GPT-4-turbo, Gemini-Pro, and Mixtral and find that a Claude v3 Opus agent is the best in terms of success rate.
  arXiv  Detail & Related papers  (2023-10-05T04:06:12Z)
• MonoByte: A Pool of Monolingual Byte-level Language Models [4.491765479948667]
  We release 10 monolingual byte-level models rigorously pretrained under the same configuration. Because they are tokenizer-free, the problem of unseen token embeddings is eliminated. Experiments on QA and NLI tasks show that our monolingual models achieve competitive performance to the multilingual one.
  arXiv  Detail & Related papers  (2022-09-22T14:32:48Z)
• AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models [121.22644352431199]
  We use Neural Architecture Search (NAS) to automatically distill several compressed students with variable cost from a large model. Current works train a single SuperLM consisting of millions ofworks with weight-sharing. Experiments on GLUE benchmark against state-of-the-art KD and NAS methods demonstrate AutoDistil to outperform leading compression techniques.
  arXiv  Detail & Related papers  (2022-01-29T06:13:04Z)
• ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training [65.68511423300812]
  We propose ProgFed, a progressive training framework for efficient and effective federated learning. ProgFed inherently reduces computation and two-way communication costs while maintaining the strong performance of the final models. Our results show that ProgFed converges at the same rate as standard training on full models.
  arXiv  Detail & Related papers  (2021-10-11T14:45:00Z)
• Exploring Sparse Expert Models and Beyond [51.90860155810848]
  Mixture-of-Experts (MoE) models can achieve promising results with outrageous large amount of parameters but constant computation cost. We propose a simple method called expert prototyping that splits experts into different prototypes and applies $k$ top-$1$ routing. This strategy improves the model quality but maintains constant computational costs, and our further exploration on extremely large-scale models reflects that it is more effective in training larger models.
  arXiv  Detail & Related papers  (2021-05-31T16:12:44Z)
• NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search [100.71365025972258]
  We propose NAS-BERT, an efficient method for BERT compression. NAS-BERT trains a big supernet on a search space and outputs multiple compressed models with adaptive sizes and latency. Experiments on GLUE and SQuAD benchmark datasets demonstrate that NAS-BERT can find lightweight models with better accuracy than previous approaches.
  arXiv  Detail & Related papers  (2021-05-30T07:20:27Z)
• Self-Supervised GAN Compression [32.21713098893454]
  We show that a standard model compression technique, weight pruning, cannot be applied to GANs using existing methods. We then develop a self-supervised compression technique which uses the trained discriminator to supervise the training of a compressed generator. We show that this framework has a compelling performance to high degrees of sparsity, can be easily applied to new tasks and models, and enables meaningful comparisons between different pruning granularities.
  arXiv  Detail & Related papers  (2020-07-03T04:18:54Z)
• General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference [34.47592026375839]
  We show that some of the computational cost during inference can be amortized over the different tasks using a shared text encoder. We also compare approaches for training such an encoder and show that encoders pre-trained over multiple tasks generalize well to unseen tasks.
  arXiv  Detail & Related papers  (2020-04-29T16:11:26Z)
• Information-Theoretic Probing for Linguistic Structure [74.04862204427944]
  We propose an information-theoretic operationalization of probing as estimating mutual information. We evaluate on a set of ten typologically diverse languages often underrepresented in NLP research.
  arXiv  Detail & Related papers  (2020-04-07T01:06:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.