LinuxArena: A Control Setting for AI Agents in Live Production Software Environments

• URL: http://arxiv.org/abs/2604.15384v1
• Date: Thu, 16 Apr 2026 03:01:11 GMT
• Title: LinuxArena: A Control Setting for AI Agents in Live Production Software Environments
• Abstract summary: LinuxArena is a control setting in which agents operate directly on live, multi-service production environments.<n> LinuxArena contains 20 environments, 1,671 main tasks representing legitimate software engineering work, and 184 side tasks representing safety failures.<n>We validate LinuxArena is useful for control research by running sabotage evaluations, and monitor evaluations, which measure a monitor model's ability to detect sabotage attempts.
• Score: 11.33890221858001
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce LinuxArena, a control setting in which agents operate directly on live, multi-service production environments. LinuxArena contains 20 environments, 1,671 main tasks representing legitimate software engineering work, and 184 side tasks representing safety failures such as data exfiltration and backdooring, making it the largest and most diverse control setting for software engineering to date. We validate LinuxArena is useful for control research by running sabotage evaluations, which measure whether attackers can complete side tasks while working on main tasks, and monitor evaluations, which measure a monitor model's ability to detect sabotage attempts. Against a GPT-5-nano trusted monitor at a 1\% step-wise false positive rate, Claude Opus 4.6 achieves roughly a 23% undetected sabotage success rate. We additionally release LaStraj, a dataset of human-crafted attack trajectories that evade monitors at substantially higher rates than any model-generated attacks we elicited, showing that current attack policies do not saturate LinuxArena. These results suggest that LinuxArena has meaningful headroom for both attackers and defenders, making it a strong testbed for developing and evaluating future control protocols.

Related papers

• HazardAuditor: From Executable Threats to Safer Computer-Use Agents [50.06351661912873]
  Existing guard models target static prompts and responses.<n>Existing safety platforms produce evaluation verdicts rather than the normalized supervision a guard model needs to learn.<n>We introduce HazardAuditor, an execution-grounded framework that closes both gaps.
  arXiv  Detail & Related papers  (2026-09-14T07:09:33Z)
• StepGuard: Learning Step-Level Guardrails with Scalable Supervision and Safety-Utility Balancing [80.39644823192113]
  We propose StepGuard, a step-level guard model that can audit completed agent trajectories and check tool actions before they are executed.<n>StepGuard achieves the highest average accuracy among open-weight guard models, with performance comparable to GPT-5.4.<n>When used to guard agents on AgentDojo and AgentDyn, StepGuard reduces mean attack success rate by 77.3%.
  arXiv  Detail & Related papers  (2026-08-25T16:17:27Z)
• Distributed Attacks in Persistent-State AI Control [0.8411355178908656]
  Iterative VibeCoding is the study of safely deploying capable but potentially untrusted AI.<n>Evasion remains high across state-of-the-art monitor models.<n>We introduce a stateful link-tracker monitor that tracks suspicious buildup across PRs.
  arXiv  Detail & Related papers  (2026-07-02T17:59:56Z)
• ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? [92.21756459993695]
  Exploitation is a challenging task because it requires low-level program reasoning.<n>Despite its importance and diagnostic value, exploitation remains under-evaluated.<n>We introduce ExploitGym, a large-scale, diverse, realistic benchmark on the exploitation capabilities of AI agents.
  arXiv  Detail & Related papers  (2026-05-11T18:00:14Z)
• MonitoringBench: Semi-Automated Red-Teaming for Agent Monitoring [0.0]
  We introduce a red-teaming methodology that exposes harder-to-catch attacks for monitoring agents.<n>We find that frontier monitors often detect suspicious actions but fall for persuasion or fail to calibrate suspiciousness scores appropriately.<n>MonitoringBench is a benchmark of 2,644 attack trajectories for evaluating monitor capabilities and failure modes.
  arXiv  Detail & Related papers  (2026-05-10T18:04:31Z)
• TraceGuard: Structured Multi-Dimensional Monitoring as a Collusion-Resistant Control Protocol [1.0195618602298684]
  TraceGuard is a structured multi-dimensional monitoring protocol that evaluates agent actions across five dimensions.<n>It is implemented as a new monitor type for the open-source ControlArena framework.
  arXiv  Detail & Related papers  (2026-04-05T05:05:59Z)
• Detecting Object Tracking Failure via Sequential Hypothesis Testing [80.7891291021747]
  Real-time online object tracking in videos constitutes a core task in computer vision.<n>We propose interpreting object tracking as a sequential hypothesis test, wherein evidence for or against tracking failures is gradually accumulated over time.<n>We propose both supervised and unsupervised variants by leveraging either ground-truth or solely internal tracking information.
  arXiv  Detail & Related papers  (2026-02-13T14:57:15Z)
• Outrunning LLM Cutoffs: A Live Kernel Crash Resolution Benchmark for All [57.23434868678603]
  Live-kBench is an evaluation framework for self-evolving benchmarks that scrapes and evaluates agents on freshly discovered kernel bugs.<n> kEnv is an agent-agnostic crash-resolution environment for kernel compilation, execution, and feedback.<n>Using kEnv, we benchmark three state-of-the-art agents, showing that they resolve 74% of crashes on the first attempt.
  arXiv  Detail & Related papers  (2026-02-02T19:06:15Z)
• MirrorGuard: Toward Secure Computer-Use Agents via Simulation-to-Real Reasoning Correction [16.58862217164395]
  We present MirrorGuard, a plug-and-play defense framework that uses simulation-based training to improve CUA security in the real world.<n>MirrorGuard learns to intercept and rectify insecure reasoning chains of CUAs before they produce and execute unsafe actions.<n>Our work proves that simulation-derived defenses can provide robust, real-world protection while maintaining the fundamental utility of the agent.
  arXiv  Detail & Related papers  (2026-01-19T08:32:09Z)
• ReasAlign: Reasoning Enhanced Safety Alignment against Prompt Injection Attack [52.17935054046577]
  We present ReasAlign, a model-level solution to improve safety alignment against indirect prompt injection attacks.<n>ReasAlign incorporates structured reasoning steps to analyze user queries, detect conflicting instructions, and preserve the continuity of the user's intended tasks.
  arXiv  Detail & Related papers  (2026-01-15T08:23:38Z)
• CaMeLs Can Use Computers Too: System-level Security for Computer Use Agents [60.98294016925157]
  AI agents are vulnerable to prompt injection attacks, where malicious content hijacks agent behavior to steal credentials or cause financial loss.<n>We introduce Single-Shot Planning for CUAs, where a trusted planner generates a complete execution graph with conditional branches before any observation of potentially malicious content.<n>Although this architectural isolation successfully prevents instruction injections, we show that additional measures are needed to prevent Branch Steering attacks.
  arXiv  Detail & Related papers  (2026-01-14T23:06:35Z)
• BashArena: A Control Setting for Highly Privileged AI Agents [1.685485565763117]
  We introduce BashArena, a setting for studying AI control techniques in security-critical environments.<n>BashArena contains 637 Linux system administration and infrastructure engineering tasks in complex, realistic environments.<n>We evaluate multiple frontier LLMs on their ability to complete tasks, perform sabotage undetected, and detect sabotage attempts.
  arXiv  Detail & Related papers  (2025-12-17T18:45:25Z)
• Async Control: Stress-testing Asynchronous Control Measures for LLM Agents [2.7146936326590425]
  We investigate asynchronous monitoring, in which a monitoring system reviews agent actions after the fact.<n>Unlike synchronous monitoring, this approach does not impose runtime latency, while still attempting to disrupt attacks before irreversible harm occurs.<n>We treat monitor development as an adversarial game between a blue team (who design monitors) and a red team (who create sabotaging agents)
  arXiv  Detail & Related papers  (2025-12-15T16:56:28Z)
• Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols [80.68060125494645]
  We study adaptive attacks by an untrusted model that knows the protocol and the monitor model.<n>We instantiate a simple adaptive attack vector by which the attacker embeds publicly known or zero-shot prompt injections in the model outputs.
  arXiv  Detail & Related papers  (2025-10-10T15:12:44Z)
• Measuring Harmfulness of Computer-Using Agents [14.509501604704127]
  Computer-using agents (CUAs) can autonomously control computers to perform multi-step actions.<n> CUAHarm consists of 104 expert-written realistic misuse risks.<n>We evaluate frontier LMs including GPT-5, Claude 4 Sonnet, Gemini 2.5 Pro, Llama-3.3-70B, and Mistral Large 2.
  arXiv  Detail & Related papers  (2025-07-31T07:02:19Z)
• OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents [60.78202583483591]
  We introduce OS-Harm, a new benchmark for measuring safety of computer use agents.<n> OS-Harm is built on top of the OSWorld environment and aims to test models across three categories of harm: deliberate user misuse, prompt injection attacks, and model misbehavior.<n>We evaluate computer use agents based on a range of frontier models and provide insights into their safety.
  arXiv  Detail & Related papers  (2025-06-17T17:59:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.