Daily Paper Briefing

Abstract Overview

This paper revisits encoder-decoder context compression for long-context language model inference, targeting the memory bottleneck created by growing KV caches. The authors introduce Latent Context Language Models (LCLMs), which compress long token sequences into shorter latent embeddings that a decoder consumes in place of the original context. They combine an architecture search with a staged training recipe and then continually pre-train 0.6B-encoder, 4B-decoder models on over 350B tokens each at 4x, 8x, and 16x compression. Across long-context benchmarks and system measurements, the work argues that end-to-end latent compression can improve the trade-off among quality, compression speed, and memory usage while remaining compatible with standard inference frameworks.

Novelty

The main novelty is showing that encoder-decoder soft-token compression, which had previously lagged KV-cache methods, can become competitive through large-scale end-to-end training and a carefully optimized architecture. The paper also extends latent compression to an agent setting where the model can inspect compressed context globally and expand selected segments on demand.

Results

The reported results place LCLMs on a new Pareto frontier over baselines on RULER, LongBench, and LongHealth, with faster time to first token and lower peak GPU memory at long context lengths while maintaining strong accuracy. The paper further reports that LCLMs achieve the best accuracy across compression ratios on GSM8K when compressing the full prompt and that an expansion-enabled agent substantially improves needle-in-a-haystack retrieval, sometimes matching uncompressed-context performance.

Key Points

1. LCLMs compress context before decoder prefill, which directly reduces decoder-side sequence length and leads to better latency and memory scaling than KV-cache eviction approaches in the reported experiments.
2. The authors identify an effective design through architecture search and staged training, favoring a 0.6B encoder with a 4B decoder, causal masking, an MLP adapter, and compression-ratio-dependent pooling choices.
3. In an agentic setup, the model can read compressed context broadly and selectively expand 512-token segments, improving exact retrieval on needle-in-a-haystack tasks over a fixed 16x compressed context alone.

References

• arXiv: https://arxiv.org/abs/2606.09659v1
• Fugu-MT: https://fugumt.com/fugumt/paper_check/2606.09659v1
• Hugging Face Papers: https://huggingface.co/papers/2606.09659