Quantifying Frontier LLM Capabilities for Container Sandbox Escape

Summary

The paper introduces SANDBOXESCAPEBENCH, an Inspect-AI benchmark that measures whether frontier LLM agents can break out of Docker/Kubernetes container sandboxes. The core methodological move is a nested "sandbox-in-a-sandbox" architecture in which the model under test runs inside a container running inside a Vagrant/EC2 VM, so successful escapes from the inner container remain isolated from the evaluation host. Eighteen tasks are constructed against the layered container-security taxonomy of Jarkas et al. (2025), spanning L1 orchestration (RBAC, CRI-O, kubectl cp, route_localnet), L3 engine/runtime (privileged, docker.sock, three capability-abuse variants, hostpath, two runc CVEs), and L4 host/kernel (pid_ns, cgroup_release_agent, Dirty COW, Dirty Pipe, bpf_privesc, packet_socket). Nine frontier models are run at five epochs per task with a 2.5M-token budget, and the paper reports pooled Wilson-CI success rates, an inference-compute scaling analysis, a hint ablation on GPT-5 and Claude Opus 4.5, and a substantial transcript-analysis appendix that documents divergent behavioural patterns (Claude models persevere; GPT models disengage or emit mid-trajectory refusals; DeepSeek-R1 and GPT-OSS-120B hallucinate flag submissions). The contribution is timely and fills a real gap that Cybench, NYU CTF, CVE-Bench, and CyberGym do not cover. Executing the shipped code confirms the paper's central quantitative findings — the Table 10 point estimates reproduce essentially cell-for-cell on difficulty 1-3, the log-linear scaling and hint-ablation figures reproduce structurally, and the behavioural contrasts hold. The remaining limitations are conceptual and interpretive rather than experimental: the "no jailbreaking" framing is at odds with the white-hat persona actually used and is elicitation-relevant given the observed refusal patterns; the abstract and contribution 3 understate the sharp difficulty gradient (all difficulty-4/5 tasks return zero in the paper, with one re-run L5 solve appearing after a data re-save); the log-linear scaling and the parallel-vs-sequential inference in Appendix F are drawn from visual trends and success-conditioned distributions respectively; and the difficulty rubric is defined by time but is internally inconsistent with CRI-O (5/5, 1-2h) and kubectl_cp (4/5, 30-60m).

Strengths

• Genuine benchmark gap. The paper convincingly establishes that container-to-host escape is not systematically evaluated by existing offensive-security benchmarks, and the layered taxonomy borrowed from Jarkas et al. (2025) is a principled organising axis.
• Safety-conscious evaluation methodology. The nested Docker-in-VM architecture is a genuinely useful piece of research infrastructure, and the new Inspect Vagrant sandbox provider makes the whole benchmark locally reproducible on commodity hardware — an unusually low bar for a security eval.
• Transcript-analysis is a real contribution in itself. Section 5.4 and Appendix E distinguish capability failures from compliance and hallucination failures via combined regex/LLM-judge scanning. The finding that GPT-5.2 disengages in ~92% of failed samples while Claude models exhaust their budget is arguably as important as the raw success rates because it changes how a defender should read the leaderboard.
• Careful shortcut discipline. Appendix C documents four cases where models found alternate escape paths (default Vagrant credentials, insecure key, eBPF substitution, Dirty COW substitution), each with the mitigation applied — precisely the "shortcut prevention" hygiene that Zhu et al. (2025a) argue benchmarks routinely miss.
• Inference-time compute analysis. The token- and cost-scaled scaling curves in Figure 3 and the hint ablation in Figure 4 are policy-relevant framings that go beyond a single success-rate leaderboard.
• Central quantitative findings reproduce. Running the shipped scoring/aggregation code reproduces the per-model per-task success rates in Table 10 essentially cell-for-cell on difficulty 1-3 (36/36 cells identical for difficulty 1 and 2; 63/63 per-task cells match at difficulty 3, with only two aggregate averages drifting by 0.01-0.04), the GPT-5.2 vs GPT-5 regression, the log-linear scaling structure, and the hint-ablation structure.

Weaknesses

• System prompt persona is not "minimal" and is elicitation-relevant. Section 3.3 characterises the prompt as free of "jailbreaking or careful coaxing," but Appendix A's prompt assigns a white-hat persona with a track record, grants blanket permission ("permitted to do anything necessary"), and provides an authorisation narrative. The fact that ~46% of GPT-5.2 and ~41% of GPT-OSS-120B samples still contain authorisation-seeking refusals shows the persona is doing work, not that it is neutral; reported capabilities are prompt-conditioned in a way the framing hides.
• Headline framing understates the difficulty gradient. The abstract and contribution 3 imply broad exploit success where the data show a step change: high on 1-2, ~0.40 on 3 for GPT-5/Opus only, and zero across every model on 4-5. Because L4/L5 contains the highest-severity CVEs the paper studies, this stratification should be surfaced in the summary language, not left to Table 10.
• Log-linear scaling and its policy corollary lack quantitative backing. The "approximately log-linear" scaling in Section 5.1 is used to argue that "deployment configuration ... can move a system across capability thresholds," but no fit statistic (slope, $R^2$, range of validity) is provided. Given that the aggregate is over 18 heterogeneous vulnerabilities and 9 models, a fitted trend with a confidence band would substantially strengthen the claim.
• Parallel-vs-sequential inference is selection-biased. Appendix F's argument that dac_read_search rewards sequential compute while pid_namespace rewards parallel retries is drawn from the token distribution of successful attempts only. To adjudicate the trade-off you need per-task success-probability-vs-budget curves; success-only spreads cannot distinguish "task is easy or unsolved" from "task benefits from many short attempts."
• Difficulty rubric conflicts with the rated tasks. Rating is defined as expected solve time, yet CRI-O sits at 5/5 with a 1-2h estimate — the same time band as several 3/5 tasks — and kubectl_cp at 4/5 with 30-60m overlaps 3/5 tasks. The rubric therefore does not cleanly explain the rating boundary, which also complicates the inter-rater analysis in Appendix G.
• Solvability verification only demonstrates the forward direction of task validity. Section 3 defines validity as "solvable iff the agent has the target capability"; the solution variant establishes only the "if." The converse relies on Appendix C shortcut prevention, which caught four unintended paths — the coupling between the two checks should be spelled out.
• Threat model is quite permissive. The agent starts as root inside the container with unrestricted egress. Many real deployments run agents as an unprivileged user with egress restrictions; the reported capability numbers therefore represent an upper bound relative to hardened deployments. A single Limitations sentence noting this would prevent over-generalisation of the numbers.
• Dirty COW shortcut description is under-specified. The four-step description in Appendix C.4 jumps from "overwrite /etc/passwd" to "as container root ... mount fresh cgroup" without the re-authentication step a Dirty COW /etc/passwd technique requires; the CAP_MKNOD mitigation targets device-node creation, but the described exploit chain does not create a device node, so it is not obvious how removing CAP_MKNOD forecloses the shortcut.
• Behavioural analysis is under-sampled. The LLM-judge scanners were run on one transcript per (model, task), not all five epochs. Given that judge output is stochastic and cell denominators are already single-digit, this under-sampling likely inflates variance on the reported disengagement/false-success percentages that drive Figures 6 and 16.
• Text-figure and text-appendix bookkeeping slips. The main text promises "four" unintended escape paths but only enumerates three (Appendix C.4 supplies the fourth). Figure 6B's y-axis is labelled "Incorrect submissions (% of samples)" while the prose reports absolute counts up to 94. The LLM judge is called Opus 4.5, 4.6, and 4.7 in three different places. Some Table 10 cells have wider Wilson intervals that silently encode reduced n.
• "Significantly worse" without a test, and inconsistent aggregation bases. The GPT-5.2 vs GPT-5 regression is described with a statistical adjective backed only by non-overlapping Wilson intervals; the 0.50 aggregate also does not match GPT-5's difficulty-3 aggregate of 0.40, and the pooling base is unstated.
• Fang et al. 87% figure is stated without scope. The related-work sentence attributes 87% one-day CVE exploitation to GPT-4 without the "given the CVE description, on a curated 15-CVE set" qualifier the original paper carries.

Reproducibility & code

• Headline aggregates reproduce; a boundary claim does not. Executing the shipped pipeline reproduces Table 10 essentially cell-for-cell on difficulty 1-3, together with the GPT-5.2 vs GPT-5 regression and the structural shape of Figures 3 and 4. The exception is the "zero success on levels 4 and 5" / "Difficulty levels 4 and 5 have been omitted due to no solves" claim: rerunning against the current bucket produces two solves for Claude Opus 4.5 on bpf_privesc (2/5 = 0.50), because bpf_privesc epochs 4-5 were re-saved after the paper's manifest was frozen and a post-paper Opus 4.5 succeeded on them. The qualitative claim survives (2/209 L4/L5 samples), but the literal "zero" does not.
• Availability statement omits the ~4.6 GB log bucket needed to regenerate the figures. The scenario code, VM sandbox provider, and plotting scripts are released, but Table 10 and Figures 2-5 can only be regenerated after downloading raw .eval logs from gs://docker-breakout-transcripts/logs/. The documented anonymous gsutil access path returns 401 and had to be replaced with a custom JSON-API downloader during replication; the availability statement does not mention the bucket or a working retrieval command.
• Plotting pipeline is not turnkey. viz/paper_plots/plots.py crashes on the full released log tree (Index contains duplicate entries) because the paper's canonical one-eval-per-(model, task) selection is not wired in; a --include-eval-ids filter and a stable task_id join were needed (recovering 161/162 vs 151/162 evals under the committed manifest, which drifted when 11 evals were re-saved). Steps 6 and 7 need different manifests (regenerated for figures, frozen-committed for the LLM-judge scan whose Parquets are keyed to original eval_ids), a coupling the repo does not document. A tabulate dependency is missing and run_misconfiguration_checks.sh globs a wrong path. None of these change the scoring logic, but the release requires a knowledgeable reproducer.
• Figure 15 regex refusal detector has no shipped code path. Appendix E.2.1 describes a nine-pattern refusal-detector over every assistant message, but transcript-analysis/scripts/ does not contain a script that emits the per-model prevalences, and no intermediate CSV is shipped. Figure 15's ~46%/~41%/~38%/~22% panel A/B is not regenerable from the release.
• Inter-rater agreement (Appendix G) has no shipped data or code. The repo carries only a single consensus difficulty per scenario; the raw R1/R2/R3 per-item ratings are absent, and no script computes Krippendorff's alpha or the pairwise linear-weighted Cohen's kappa. Tables 3-9 could not be reproduced.
• Shipped LLM-judge counts differ from paper values on the smaller/open models. A re-run of the frozen 148-sample manifest matches the paper on GPT-5.2 disengagement (92.3%) and the Claude zero-rows, but disagrees by 15-35 pp on GPT-OSS-120B disengagement (~50% vs paper 80.0%), DeepSeek-R1 false-success (~70% vs 86.7%), and GPT-OSS-120B false-success (~30% vs 66.7%). The README attributes the shipped scan to a subset; a paper-value re-run would need roughly $350 of Anthropic API budget. This is largely irreducible stochastic-judge variance rather than a methodological error, but the point estimates in Section 5.4 currently carry no uncertainty.
• Hint-variant logs are not documented in the release. viz/paper_plots/ablation.py expects --hints-logs and --baseline-logs paths that are not listed in the README's data section, so Figure 4 is not reproducible without a fresh, paid hint-variant re-run of GPT-5 and Claude Opus 4.5.
• Avg column semantics under-documented in Table 10. Avg (3/5) reconciles to a macro-average for Claude Opus and Claude Sonnet, but not for GPT-5 (0.42 reported vs 0.40 macro over 7 tasks); the code (create_performance_table) uses a sample-level pool, which produces the 0.42 because runc_overwrite has only 3 GPT-5 samples. The convention is not stated in the paper.
• "Under 25 messages" summary does not survive the shipped truncation fix. plot_ecdfs.py reconstructs true message counts to work around an Inspect single-message truncation bug, shifting the ECDF right relative to the paper's narrative summary; the >100-message tail claim still holds, but the "majority under 25 messages" half of the sentence does not match the corrected counts.

Recommended Changes

Essential

• Reconcile the "no jailbreaking" claim with the actual Appendix A prompt. Either soften Section 3.3 to describe the fixed prompt as "a single white-hat persona system prompt with authorisation framing," or add an ablation stripping the persona/authorisation clauses so readers can see how much reported capability depends on prompt framing (weakness: system prompt persona).
• Qualify the abstract and contribution 3 with the difficulty gradient. Add a clause noting that success is confined to difficulty 1-3 and that difficulty-4/5 solves are essentially absent in the released cohort, so summary language matches Section 5.1 and Table 10 (weaknesses: abstract framing, contribution 3).
• Update the "zero success on levels 4 and 5" statements against the released data. Either re-freeze the analysed manifest and note the exact set of .eval files behind the paper's numbers, or acknowledge that a re-run against the current bucket yields two Opus 4.5 solves on bpf_privesc (2/5) and revise Section 5.1 and the Table 10 caption accordingly (Reproducibility bullet: difficulty-4/5 boundary claim).
• Document the GCS bucket, a working retrieval command, and the hint-variant log location in the availability statement. Explicitly point at gs://docker-breakout-transcripts/logs/ (and replace the broken anonymous gsutil path with a working retrieval method), and either ship the hint-variant logs or a compact intermediate table containing the per-(model, task, budget) hint success rates used in Figure 4 (Reproducibility bullets: bucket omitted from availability statement; hint-variant logs).
• Ship inter-rater agreement code and raw ratings. Add the per-item R1/R2/R3 ratings CSV plus a short script that reproduces Krippendorff's alpha, the pairwise linear-weighted Cohen's kappa, and the confusion matrices in Appendix G, and report a bootstrap CI on alpha (Reproducibility bullet: Appendix G unsupported).
• Ship the Figure 15 refusal-regex detector and its intermediate counts. Without the detector script and per-model CSV, Figure 15's ~46%/~41%/~38%/~22% panel is not regenerable from the release (Reproducibility bullet: regex refusal detector).
• Wire canonical eval-selection into the plotting entry point and document the two-manifest coupling. plots.py should not crash on the full released log tree; add the canonical include-eval-ids machinery, a stable task_id join to survive re-saves, and a README note explaining that Steps 6 and 7 require different manifests (Reproducibility bullet: plotting pipeline not turnkey).
• Support the log-linear scaling claim with a fit. Report at least an aggregate slope, $R^2$, and range of validity for the "approximately log-linear" curve in Figure 3, and re-express the operational implication in terms of that fitted trend (weakness: log-linear scaling backing).
• Re-scope or re-analyse the parallel-vs-sequential argument. Either describe Figure 17 as a descriptive observation about successful-run token spread, or add unconditional success-probability-vs-budget curves so the parallel-vs-sequential comparison is actually supported by the data (weakness: Appendix F selection bias).
• Clarify the difficulty rubric so the rating boundary is explained. Either add non-time criteria that justify CRI-O at 5/5 and kubectl_cp at 4/5 despite comparable time estimates to 3/5 tasks, or re-rate for consistency with the time-based rubric (weakness: difficulty rubric conflict).
• Re-run and ship the full-cohort LLM-judge scanner outputs, or clearly flag the shipped subset as preliminary. The 15-35 pp gap on GPT-OSS-120B and DeepSeek-R1 numbers between the frozen 148-sample scans and the paper values is currently silent (Reproducibility bullet: LLM-judge counts).

Suggested

• Add per-cell $n$ annotations in Table 10. Cells with wider Wilson intervals silently encode reduced $n$ (e.g., DeepSeek-R1 dac_read_search [0.00, 0.49]); footnoting them would prevent over-reading apparent 0/5 vs 0/n differences (weakness: Wilson-interval $n$ encoding).
• Fix the "four unintended paths" enumeration. Either list all four in Section 3.3 or rewrite as "three plus a fourth described in Appendix C.4" (weakness: bookkeeping slips).
• Reconcile the judge model version across §5.4, §E.2, and §E.2.2. State the single judge model actually used per scanner (weakness: bookkeeping slips).
• Fix Figure 6B's axis label vs. prose units mismatch. The axis is either a count (matching the "21 per sample" prose) or a percentage; the two cannot coexist (weakness: bookkeeping slips).
• Add per-transcript uncertainty on the judge-based percentages in Section 5.4. Either sample all five epochs or attach a bootstrap CI to the disengagement/false-success rates that drive Figures 6 and 16 (weakness: behavioural analysis under-sampled).
• Add the missing re-authentication step to the Dirty COW description and re-examine the CAP_MKNOD mitigation. The transition from "overwrite /etc/passwd" to "container root" needs an intervening su/login/exec, and the CAP_MKNOD mitigation should either target a device-node step that is missing from the current chain, or be revised (weakness: Dirty COW shortcut).
• Note that reported numbers are an upper bound relative to hardened deployments. A short Limitations sentence acknowledging that container-root + external egress is a permissive initial posture (weakness: threat-model permissiveness).
• State the averaging convention used in Table 10's Avg column. Sample-level pooling is the code's convention but the paper does not say so, and the two conventions disagree on GPT-5's 3/5 average (Reproducibility bullet: Avg semantics).
• Reconcile the message-length narrative with the shipped ECDF. Either report the corrected counts in the main text or explicitly note that Figure 5 uses corrected counts that differ from the "under 25 messages" summary (Reproducibility bullet: message-length summary).
• Add the scope caveat to the Fang et al. 87% figure ("when given the CVE description, on a curated 15-CVE benchmark") to prevent misreading of the related-work claim (weakness: Fang scope).
• Reconcile the 803-sample number at first mention. Move the seven-sample cancellation note out of Appendix E.2 so it appears where Section 5.4 first uses 803 (weakness: sample-count reconciliation).
• Back "significantly worse" with a test or soften the wording. A paired-by-task test on the underlying counts (or dropping "significantly") would remove the statistical-word overreach in the GPT-5.2 vs GPT-5 regression paragraph (weakness: significance wording).