Quantifying Frontier LLM Capabilities for Container Sandbox Escape

• URL: http://arxiv.org/abs/2603.02277v1
• Date: Sun, 01 Mar 2026 22:47:39 GMT
• Title: Quantifying Frontier LLM Capabilities for Container Sandbox Escape
• Authors: Rahul Marchand, Art O Cathain, Jerome Wynne, Philippos Maximos Giavridis, Sam Deverett, John Wilkinson, Jason Gwartz, Harry Coppock,
• Abstract summary: Large language models (LLMs) increasingly act as autonomous agents, using tools to execute code, read and write files, and access networks.<n>To mitigate these risks, agents are commonly deployed and evaluated in isolated "sandbox" environments.<n>We introduce SANDBOXESCAPEBENCH, an open benchmark that safely measures an LLM's capacity to break out of these sandboxes.
• Score: 1.6245103041408155
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) increasingly act as autonomous agents, using tools to execute code, read and write files, and access networks, creating novel security risks. To mitigate these risks, agents are commonly deployed and evaluated in isolated "sandbox" environments, often implemented using Docker/OCI containers. We introduce SANDBOXESCAPEBENCH, an open benchmark that safely measures an LLM's capacity to break out of these sandboxes. The benchmark is implemented as an Inspect AI Capture the Flag (CTF) evaluation utilising a nested sandbox architecture with the outer layer containing the flag and no known vulnerabilities. Following a threat model of a motivated adversarial agent with shell access inside a container, SANDBOXESCAPEBENCH covers a spectrum of sandboxescape mechanisms spanning misconfiguration, privilege allocation mistakes, kernel flaws, and runtime/orchestration weaknesses. We find that, when vulnerabilities are added, LLMs are able to identify and exploit them, showing that use of evaluation like SANDBOXESCAPEBENCH is needed to ensure sandboxing continues to provide the encapsulation needed for highly-capable models.

Related papers

• RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories [58.32028251925354]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, but their proficiency in producing secure code remains a critical, under-explored area.<n>We introduce RealSec-bench, a new benchmark for secure code generation meticulously constructed from real-world, high-risk Java repositories.
  arXiv  Detail & Related papers  (2026-01-30T08:29:01Z)
• LLM-in-Sandbox Elicits General Agentic Intelligence [142.7174116109795]
  We introduce LLM-in-Sandbox, enabling LLMs to explore within a code sandbox (i.e., a virtual computer) to elicit general intelligence in non-code domains.<n>We show that strong LLMs, without additional training, exhibit generalization capabilities to leverage the code sandbox for non-code tasks.<n>Experiments demonstrate that LLM-in-Sandbox, in both training-free and post-trained settings, achieves robust generalization spanning mathematics, physics, chemistry, biomedicine, long-context understanding, and instruction following.
  arXiv  Detail & Related papers  (2026-01-22T18:57:09Z)
• The Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree Search [58.8834056209347]
  Large language models (LLMs) remain vulnerable to jailbreak attacks that bypass safety guardrails to elicit harmful outputs.<n>We introduce the Correlated Knowledge Attack Agent (CKA-Agent), a dynamic framework that reframes jailbreaking as an adaptive, tree-structured exploration of the target model's knowledge base.
  arXiv  Detail & Related papers  (2025-12-01T07:05:23Z)
• ORCA: Unveiling Obscure Containers In The Wild [2.412902381004722]
  Software Composition Analysis (SCA) is a critical process that helps identify packages and dependencies inside a container.<n>In this paper, we examine the limitations of both cloud-based and open-source SCA tools when faced with such obscure images.<n>We propose an obscuration-resilient methodology for container analysis and introduce ORCA, its open-source implementation.
  arXiv  Detail & Related papers  (2025-09-11T10:12:56Z)
• Empirical Security Analysis of Software-based Fault Isolation through Controlled Fault Injection [21.57370108666908]
  Google's V8 heap sandbox protects billions of users across all Chromium-based browsers and countless applications built on Node$.$js and Electron.<n>Despite their widespread use, such SFI mechanisms have seen little security testing.<n>We propose a new testing technique that models the security boundary of modern SFI implementations.
  arXiv  Detail & Related papers  (2025-09-09T13:56:17Z)
• SafePTR: Token-Level Jailbreak Defense in Multimodal LLMs via Prune-then-Restore Mechanism [123.54980913741828]
  Multimodal Large Language Models (MLLMs) extend LLMs to support visual reasoning.<n>MLLMs are susceptible to multimodal jailbreak attacks and hindering their safe deployment.<n>We propose Safe Prune-then-Restore (SafePTR), a training-free defense framework that selectively prunes harmful tokens at vulnerable layers while restoring benign features at subsequent layers.
  arXiv  Detail & Related papers  (2025-07-02T09:22:03Z)
• CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities [6.752938800468733]
  Large language model (LLM) agents are increasingly capable of autonomously conducting cyberattacks.<n>Existing benchmarks fall short as they are limited to abstracted Capture the Flag competitions or lack comprehensive coverage.<n>We introduce CVE-Bench, a real-world cybersecurity benchmark based on critical-severity Common Vulnerabilities and Exposures.
  arXiv  Detail & Related papers  (2025-03-21T17:32:32Z)
• Automating Prompt Leakage Attacks on Large Language Models Using Agentic Approach [9.483655213280738]
  This paper presents a novel approach to evaluating the security of large language models (LLMs)<n>We define prompt leakage as a critical threat to secure LLM deployment.<n>We implement a multi-agent system where cooperative agents are tasked with probing and exploiting the target LLM to elicit its prompt.
  arXiv  Detail & Related papers  (2025-02-18T08:17:32Z)
• AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting [54.931241667414184]
  We propose textbfAdaptive textbfShield Prompting, which prepends inputs with defense prompts to defend MLLMs against structure-based jailbreak attacks. Our methods can consistently improve MLLMs' robustness against structure-based jailbreak attacks.
  arXiv  Detail & Related papers  (2024-03-14T15:57:13Z)
• Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models [79.0183835295533]
  We introduce the first benchmark for indirect prompt injection attacks, named BIPIA, to assess the risk of such vulnerabilities.<n>Our analysis identifies two key factors contributing to their success: LLMs' inability to distinguish between informational context and actionable instructions, and their lack of awareness in avoiding the execution of instructions within external content.<n>We propose two novel defense mechanisms-boundary awareness and explicit reminder-to address these vulnerabilities in both black-box and white-box settings.
  arXiv  Detail & Related papers  (2023-12-21T01:08:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.