A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compression

• URL: http://arxiv.org/abs/2412.17483v1
• Date: Mon, 23 Dec 2024 11:24:04 GMT
• Title: A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compression
• Authors: Chenlong Deng, Zhisong Zhang, Kelong Mao, Shuaiyi Li, Xinting Huang, Dong Yu, Zhicheng Dou,
• Abstract summary: We show that gist-based compression can achieve near-lossless performance on tasks like retrieval-augmented generation and long-document QA. We identify three key failure patterns: lost by the boundary, lost if surprise, and lost along the way. We propose two effective strategies: fine-grained autoencoding, which enhances the reconstruction of original token information, and segment-wise token importance estimation.
• Score: 41.71994217868039
• License:
• Abstract: In this work, we provide a thorough investigation of gist-based context compression methods to improve long-context processing in large language models. We focus on two key questions: (1) How well can these methods replace full attention models? and (2) What potential failure patterns arise due to compression? Through extensive experiments, we show that while gist-based compression can achieve near-lossless performance on tasks like retrieval-augmented generation and long-document QA, it faces challenges in tasks like synthetic recall. Furthermore, we identify three key failure patterns: lost by the boundary, lost if surprise, and lost along the way. To mitigate these issues, we propose two effective strategies: fine-grained autoencoding, which enhances the reconstruction of original token information, and segment-wise token importance estimation, which adjusts optimization based on token dependencies. Our work provides valuable insights into the understanding of gist token-based context compression and offers practical strategies for improving compression capabilities.

Related papers

• Robust and Transferable Backdoor Attacks Against Deep Image Compression With Selective Frequency Prior [118.92747171905727]
  This paper introduces a novel frequency-based trigger injection model for launching backdoor attacks with multiple triggers on learned image compression models. We design attack objectives tailored to diverse scenarios, including: 1) degrading compression quality in terms of bit-rate and reconstruction accuracy; 2) targeting task-driven measures like face recognition and semantic segmentation. Experiments show that our trigger injection models, combined with minor modifications to encoder parameters, successfully inject multiple backdoors and their triggers into a single compression model.
  arXiv  Detail & Related papers  (2024-12-02T15:58:40Z)
• TACO-RL: Task Aware Prompt Compression Optimization with Reinforcement Learning [11.167198972934736]
  Large language models (LLMs) such as GPT-4 have led to a surge in the size of prompts required for optimal performance. We propose a novel and efficient reinforcement learning (RL) based task-aware prompt compression method. We demonstrate that our RL-guided compression method improves the task performance by 8% - 189% over state-of-the-art compression techniques.
  arXiv  Detail & Related papers  (2024-09-19T18:11:59Z)
• LanguaShrink: Reducing Token Overhead with Psycholinguistics [8.123272461141815]
  LanguaShrink is a prompt compression framework for large language models. It reduces prompt length while preserving essential information. Compared to existing prompt compression methods, LanguaShrink improves end-to-end latency by 1.43 times.
  arXiv  Detail & Related papers  (2024-09-01T22:09:20Z)
• QUITO-X: A New Perspective on Context Compression from the Information Bottleneck Theory [66.01597794579568]
  We introduce information bottleneck theory (IB) to model the problem. We propose a cross-attention-based approach to approximate mutual information in IB. Our method achieves a 25% increase in compression rate compared to the state-of-the-art.
  arXiv  Detail & Related papers  (2024-08-20T02:44:45Z)
• Concise and Precise Context Compression for Tool-Using Language Models [60.606281074373136]
  We propose two strategies for compressing tool documentation into concise and precise summary sequences for tool-using language models. Results on API-Bank and APIBench show that our approach reaches a performance comparable to the upper-bound baseline under up to 16x compression ratio.
  arXiv  Detail & Related papers  (2024-07-02T08:17:00Z)
• Compressing Lengthy Context With UltraGist [22.054232261437186]
  We propose a new method called UltraGist, which is distinguished for its high-quality compression of lengthy context. UltraGist contributes to the flexibility of compression, as it can be effectively learned to support a broad range of context lengths and compression ratios. It makes the training process sample-efficient and thus maximizes the use of training data.
  arXiv  Detail & Related papers  (2024-05-26T17:23:56Z)
• Long Context Compression with Activation Beacon [22.054232261437186]
  Activation Beacon is a plug-in module for transformer-based LLMs. It targets effective, efficient, and flexible compression of long contexts. It achieves a 2x acceleration in inference time and an 8x reduction of memory costs for KV cache.
  arXiv  Detail & Related papers  (2024-01-07T11:57:40Z)
• An Information Theory-inspired Strategy for Automatic Network Pruning [88.51235160841377]
  Deep convolution neural networks are well known to be compressed on devices with resource constraints. Most existing network pruning methods require laborious human efforts and prohibitive computation resources. We propose an information theory-inspired strategy for automatic model compression.
  arXiv  Detail & Related papers  (2021-08-19T07:03:22Z)
• Learning End-to-End Lossy Image Compression: A Benchmark [90.35363142246806]
  We first conduct a comprehensive literature survey of learned image compression methods. We describe milestones in cutting-edge learned image-compression methods, review a broad range of existing works, and provide insights into their historical development routes. By introducing a coarse-to-fine hyperprior model for entropy estimation and signal reconstruction, we achieve improved rate-distortion performance.
  arXiv  Detail & Related papers  (2020-02-10T13:13:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.