Context Language Models
Abstract Overview
This paper introduces Context Language Models (CLMs), which make context management a native model capability by representing the live context as an editable file that the model can modify directly rather than only extending it append-only. The study argues that this unrestricted, model-controlled context editing allows more adaptive strategies than fixed harness policies or predefined context-management actions. The authors evaluate zero-shot CLMs on diagnostic context-management tasks, coding and deep-research benchmarks, and longer-horizon open discovery settings including repository optimization and mathematical search problems. They also show that CLMs can be improved through natural-language steering, textual skill evolution, reinforcement learning, and a serving optimization called Suffix Cache Reuse.
Novelty
The distinctive contribution is to recast context management itself as part of the language model's action space, implemented through editable context files with unrestricted updates rather than hand-designed summarization, retrieval, or compaction tools. The work also extends this idea to learning and serving: CLMs can acquire better context strategies via in-context evolution or RL, and their serving stack is co-designed with Suffix Cache Reuse to handle non-prefix-preserving edits more efficiently.
Results
Across multiple evaluations, zero-shot CLMs outperform or match strong context-management baselines while often using less compute, including higher accuracy on BrowseComp-Plus, matched TerminalBench 2.1 performance at lower FLOPs, and improved EdgeBench and multi-repository optimization outcomes. On mathematical optimization tasks, CLMs achieve the best reported best-of-run scores among the compared workflows. The paper also reports that RL improves a Qwen3.5-9B CLM from 28.8% to 42.5% on BrowseComp-Plus, and that Suffix Cache Reuse reduces server-side compute by 35% relative to standard SGLang at matched performance.
Key Points
- CLMs treat the live context as an editable file, giving the model direct control over what to retain, rewrite, or remove over long trajectories.
- In zero-shot evaluations, CLMs outperform or match harness-based and action-based baselines on deep research, coding, and open discovery tasks while often requiring fewer prefix-reuse FLOPs.
- The framework supports further gains through natural-language steering, textual skill evolution, reinforcement learning, and Suffix Cache Reuse for more efficient serving after context edits.