CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs
Abstract Overview
This paper introduces CIAware-Bench, a benchmark for measuring control intervention awareness in frontier language models: whether a model can tell when part of its trajectory has been replaced or edited by a control protocol. The benchmark covers four domains—essay writing, BigCodeBench, Bash Arena, and SHADE-Arena—and varies intervention settings including watermarking, side tasks, and alternative protocols such as defer-to-trusted, trusted editing, and defer-to-resample. Across evaluations of eleven frontier models, the authors find that intervention detection is usually low to moderate and depends strongly on the specific acting model, intervening model, and environment rather than being a stable model-level trait. The results suggest that stylistic and behavioral cues matter, especially in text- and code-heavy settings, and that CI awareness should be measured for each deployment scenario rather than assumed from general capability.
Novelty
The main novelty is a dedicated benchmark for control intervention awareness, a safety-relevant capability that prior AI control evaluations had not systematically measured. The work is also distinctive in combining multiple task domains, model pairings, watermarking conditions, and intervention protocols to isolate when interventions become detectable.
Results
Under the default setting, most acting/intervening/environment combinations are near chance, but some pairs reach substantially higher detection accuracy, with reported values up to 0.87. Cross-family interventions are easier to detect than within-family interventions on Essay, GPT-5 is the most detectable intervening model on average, and larger interventions are easier to notice. Hidden-scratchpad watermarking markedly increases detection on BigCodeBench and Essay, while same-model resampling on Essay is indistinguishable from chance and full cross-model substitution is more detectable than editing or resampling.
Key Points
- CIAware-Bench measures whether an acting model can distinguish its own trajectories from ones modified by a control intervention, using four domains and multiple protocol variants.
- CI awareness is not a simple function of overall model capability; it varies mainly by the specific acting model, intervening model, and environment combination.
- Detection is amplified by stronger stylistic signals such as cross-family outputs, larger replacements, and watermarking, especially in essay and code settings rather than constrained agentic environments.