Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking

Summary

Jailbreak Foundry (JBF) tackles a real and worsening problem in LLM safety evaluation: jailbreak papers proliferate faster than benchmarks can absorb them, so cross-paper robustness numbers drift as datasets, harnesses, and judges vary. The conceptual move is to industrialize paper-to-code translation with a planner–coder–auditor multi-agent workflow (JBF-FORGE) emitting modules that conform to a shared framework contract (JBF-LIB), so a standardized harness (JBF-EVAL) can run 30 reproduced attacks against 10 victim models on a single dataset with a fixed GPT-4o judge. The engineering scope is substantial: 30 attacks reproduced from 2023–2025 papers, 42% LOC compression, an 82.5% shared-code share, and a matched-setting reproduction with a signed-mean deviation of $+0.26$ pp. The cross-model heatmap is the most interesting empirical artifact — it exposes strong attack–model interactions (GPT-5.1 mean 29.5% but 94% on JAILCON-CIT; GPT-OSS-120B mean 9.13% but 82% on Mousetrap) that only surface when a broad, current attack suite is applied uniformly. Overall this is a useful systems paper that raises the floor for jailbreak reproducibility. My main concerns are that (i) the headline fidelity claim rests on a signed mean that hides two-sided error (mean absolute gap 5.7 pp, range $-16$ to $+20$), and did not reproduce out-of-environment because the shipped code is hard-wired to private infrastructure (internal vLLM endpoints, provider keys) that is undocumented, forcing model substitutions that pushed ASR down $\sim 35$ pp; (ii) the paper-matched and standardized GPT-4o-judge regimes are never reconciled with a within-attack judge sensitivity check; (iii) the automation narrative under-counts human choices (representative-setting selection, MATCHCFG, refinement triggering, repo retrieval); (iv) precedence between paper, plan, and repo is stated three different ways, so refined modules are effectively repo-faithful; and (v) most quantitative supplements — Table 1 ASRs, the 30x10 matrix, timing logs, no-repo runs, Kimi ablations — are described but not shipped. Reassuringly, when we did execute the shipped code, the engineering claims (LOC compression, framework-reuse share, QueryAttack structural compliance) reproduced almost exactly.

Strengths

• Timely engineering contribution. The paper identifies a genuine coverage-freshness problem in jailbreak benchmarking and delivers a concrete systems answer — a shared library, a paper-to-module agent workflow, and a standardized evaluator — rather than yet another attack.
• Empirical breadth. 30 attacks × 10 victims (including recent GPT-5.1 and GPT-OSS-120B) under a unified harness is the largest current-generation jailbreak matrix I have seen, and the heatmap surfaces attack–model interaction effects that a smaller suite would hide.
• Multi-tier fidelity evidence. The paper does not rely on ASR alone: Appendix F adds trace-level checks (Past Tense, QueryAttack) and a manual 8-criterion semantic audit (DeepInception, ABJ, ReNeLLM, TrojFill), and Appendix G separates whole-repo compression from core-attack LOC — this is more careful than most reproduction papers.
• Well-defined agent contract. The planner–coder–auditor decomposition with a bounded audit loop, the explicit non-invention rule, and a static-audit-plus-execution-gate design are cleanly specified in Appendix B and give reviewers something concrete to critique.
• Engineering claims reproduce cleanly. Executing the shipped code, we recover $\rho \approx 0.419$ against the paper's $0.42$ compression ratio, an $\approx 81.5\%$ framework-reuse share against the paper's $82.5\%$, and 50/50 QueryAttack SQL structural compliance — so the systems half of the paper holds up independently.
• Backend ablation is included at all. Even though the ablation is small (see below), running a targeted planner/coder/auditor substitution study with Kimi-K2.5 is more than most agent-workflow papers do.

Weaknesses

• Signed-mean fidelity headline. The $+0.26$ pp claim in the abstract is a mean of a signed deviation with two-sided errors up to $\pm 20$ pp and a mean absolute gap of 5.7 pp. Please lead with mean $|\Delta|$, median $|\Delta|$, and a per-attack distribution — a mean signed error near zero is not the same as "high fidelity."
• Two evaluation regimes never reconciled. $\text{ASR}_\text{paper}$–$\text{ASR}_\text{gen}$ fidelity is computed under paper-matched judges (varied), but Section 5's cross-attack story uses one fixed GPT-4o judge. Cross-attack comparability of Section 5 numbers therefore inherits an uncontrolled judge shift; a within-attack $\text{ASR}_\text{paper-judge}$ vs $\text{ASR}_\text{gpt4o-judge}$ check on the same generated outputs is missing.
• Autonomy claims overstated. "Without human involvement" / "without manual implementation effort" undersells the required choices: representative-setting selection, MATCHCFG, enhanced-refinement triggering, and reference-repo retrieval are all human decisions, and the coder-prompt template hard-codes provider="openai" while the surrounding rules say to read args.provider.
• Precedence between paper, plan, and repo is inconsistent. The main text uses $s_p \succ C \succ R$. The auditor prompt calls the repo the "gold standard" and says "cloned code taking priority when they differ." The enhanced-refinement Phase-1 prompt flatly instructs "treat the repo as the source of truth." Three different orderings coexist; the resulting modules are effectively repo-faithful, not paper-faithful, whenever the refinement pass runs.
• Selection bias in the reproduction set. Attacks are restricted to those reported on AdvBench or JailbreakBench — precisely the ones whose original evaluation is already aligned with JBF-EVAL. Attacks reported on HarmBench, StrongREJECT, or bespoke behaviors are excluded a priori, and the fidelity claim does not generalize to the broader jailbreak literature.
• With-repo δ arithmetic does not close. Text says paper ASR $77.0 \to$ no-repo $66.5 \to$ with-repo $86.3$ ($\delta = +19.8$), but the five per-attack $\delta$ (5.2, 5.3, 48.6, 34.8, 10.1) average to 20.8. Off by $\approx 1.0$ pp.
• Small-n ablation and small-n audit. The planner/coder/auditor architectural claim rests on $n = 3$ attacks $\times$ 5 ablation cells $\times$ 1 seed, and the semantic-fidelity audit is $n = 4$ (3/4 of which contain unsupported additions marked C8 = 1). Neither supports the strength of the architectural conclusions drawn.
• Statistically weak iteration-count analysis. Reporting Pearson $r = 0.186$, $p = 0.325$ on a bimodal, 1-to-5-integer predictor with $n = 2$ at three of the five points, and concluding "the loop does not systematically inflate or suppress ASR," is null-not-rejected framed as evidence of no effect. Spearman/permutation/ANOVA would be more appropriate.
• Variant confounding. The "30 attacks" include several variant pairs from the same paper (SATA-MLM/ELP, WordGame/WordGame+, RA-DRI/SRI, JAILCON-CIT/CVT); Section 4.3 de-dupes to 26 for LOC accounting but Section 5's threshold counts and family means treat variants as independent.
• Judge-choice bias risk. Using GPT-4o as the unified judge risks favoring formal/structured outputs (which look "answer-like" to an OpenAI judge) — plausibly part of the "formal wrappers are the most effective carrier class (66.0%)" finding. No sensitivity check to a second judge (e.g., a HarmBench classifier or StrongREJECT rubric) is reported.
• Static audit over-interprets TrojFill. The audit is billed as static, source-inspection only, so its conclusion that TrojFill's $-6.3$ pp gap is "more consistent with runtime/interface effects" is inference beyond the evidence type. Either add runtime traces or soften to "no visible mechanism-level deviation."
• Weak randomness accounting. The paper acknowledges nondeterminism in one sentence and defers to "fixed seeds whenever supported"; there is no per-attack rerun variance, no snapshot-date pinning for GPT-4/4o/5.1, and no exact Claude/Qwen/GPT-OSS checkpoint identifiers.
• 82.5% "reuse" is amortization, not per-attack maintainability. Adding more attacks mechanically inflates the ratio (framework LOC) / (framework LOC + $\sum$ attack LOC). A median-attack-specific-LOC figure would speak to maintainability directly.
• "Black-box and gray-box" scoping is not grounded in the setup. The LiteLLM adapter architecture is text-in/text-out only; there is no gray-box (partial-access) attack in the 30-attack set. Please drop the "gray-box" qualifier or add an example.
• Dual-use release policy is unspecified. The impact statement acknowledges dual-use but the mitigation is generic; the released repo is not gated, and the paper reports high per-attack ASRs against currently deployed commercial models.
• Minor notation and consistency issues. Planner and coder signatures differ between the role definitions, Algorithm 1, and the "Synthesis and validation" text; the enhanced-refinement threshold $-10.0$ is never explicitly bound to $\tau$; the fidelity-score sentence "We set" reads truncated; GPT-4o's row mean in the text (74.5%) disagrees with Figure 4's model average (74.4).

Reproducibility & code

Executing the released repository, my main finding is that the systems half of the paper reproduces cleanly, but the effectiveness half does not — and the reason is a private-infrastructure dependency that is not documented anywhere in the paper.

• Central fidelity claim does not reproduce out-of-environment. Running the shipped code, the reproduced Table 1 came out roughly $-35$ pp below the paper (vs. the reported $+0.26$ pp), and only 11 of 30 attacks emitted any valid ASR. The failure is largely environmental: the paper's exact 2023–2024 victim snapshots are not reachable from outside the authors' network, so newer, more-safety-aligned checkpoints were substituted and the standardized GPT-4o judge is stricter than several of the paper-matched per-attack judges. The number itself should therefore be labeled as a matched-setting estimate under the authors' internal infrastructure, and the model snapshots/endpoints/judge configurations required to reproduce it should be documented.
• Shipped code assumes private infrastructure the reader cannot reach. src/jbfoundry/llm/litellm.py routes every victim/judge/helper call through internal vLLM endpoints (10.210.22.x IPs) plus Azure/Anthropic/Infini provider keys that no external reproducer has. We had to author an OpenRouter routing shim with approximate model substitutions before any of the pipeline would run. For a paper whose central promise is reproducible living benchmarks, this is the most consequential release gap: even readers with API budget cannot run the pipeline as shipped.
• Engineering aggregates reproduce almost exactly. Recomputing the LOC compression against re-cloned upstream repos gave $\rho \approx 0.419$ vs. the paper's $0.42$; the framework-reuse share came out to $\approx 81.5\%$ vs. $82.5\%$; and QueryAttack SQL structural compliance was 50/50 on our rerun. These are strong signals that the engineering claims are credible; the fidelity story is what does not travel.
• Headline Table 1. No table_1.csv (or equivalent) with ASR_paper / ASR_gen per attack ships. Verifying $+0.26$ pp requires re-running each attack against its paper-matched victim (several proprietary) with a paper-matched judge; there is no rerun log, judge configuration record, or seed archive.
• 30 x 10 standardized heatmap. scripts/analysis/generate_heatmap.py exists but its input tree (results/*_comprehensive_results.json) is not shipped. Under an API budget we could only produce a sparse subset; the induced GPT-OSS-120B profile did not match the paper's 9.13%/15-of-30-at-0% figures, but the discrepancy is dominated by coverage, not disagreement. All of Section 5's per-model, per-family, and per-carrier aggregates therefore cannot be checked without the full 300-cell matrix.
• LOC-compression baseline. Every attacks_paper_info/<arxiv_id>/<ProjectName>/ reference-repo folder is shipped empty. The 22,714 $\to$ 9,549 LOC comparison, the $\rho$ column of Table 1, and Appendix G's core-LOC ratios cannot be independently recomputed without re-cloning 19 upstream repos and re-applying an implicit "attack-related code" selection rule.
• Synthesis timing. No per-run stage-tagged log is included, so the mean 28.2 min / median 25.0 min / 82%-under-60 min claim and Appendix D's stage decomposition are not directly checkable.
• No-repo vs with-repo ablation. Only with-repo modules ship under attacks/generated/; no paired paper-only variants, no ablation ASR log, and no Figure 2 plotting script are included. $\delta = +19.8$ pp cannot be verified.
• Enhanced refinement. *_improved.py variants for past_tense, scp, trial, and trojfill are shipped, but the AIR and ISA variants — needed to reproduce the 6-attack subset in Figure 3 and the ISA non-improving-case analysis — are absent.
• Stage-tagged token accounting. Table 3 requires per-forge-stage token logs; the shipped cost_tracker.py tracks per-attack-run cost, not per-stage forge cost, and no stratified-sample log is included. The overall cached total (3.24M) also does not equal the sum of stage cached totals ($0.28 + 2.44 + 0.57 \approx 3.29$M).
• Kimi ablation. The 15-run planner/coder/auditor substitution requires 15 alternative attack modules and their ASRs; none are shipped, and no "remove-planner" / "remove-auditor" switches appear in the workflow orchestrator.
• Past-Tense trace comparison and QueryAttack 50/50 compliance. Neither has a shipped trace log or prompt dump; only illustrative excerpts appear in text (we did reproduce QueryAttack 50/50 by re-running the module, but the prompt set is not archived).
• Snapshot pinning. No API-version dates for GPT-4/4o/5.1 and no exact Claude/Qwen/GPT-OSS checkpoints; a re-runner today will not see the same numbers even under identical seeds.

Overall, the shipped code is enough to verify what was implemented for each attack — and the engineering aggregates hold up — but it is not enough to independently verify what the effectiveness numbers were, and the private-infrastructure dependency means most readers cannot re-run the pipeline as shipped at all. For a paper whose central promise is "living benchmarks" and reproducibility, that combination deserves an explicit data/artifacts section and, ideally, a portable execution recipe.

Recommended Changes

Essential.

• Rewrite the fidelity headline. Replace the signed mean $+0.26$ pp in the abstract and introduction with mean $|\Delta|$, median $|\Delta|$, and a per-attack distribution or CI, and note the $+20$ / $-16$ pp range. Addresses Signed-mean fidelity headline.
• Document and portabilize the execution environment. Add a "how to reproduce end-to-end" section that names the internal vLLM endpoints and the Azure/Anthropic/Infini keys the shipped code currently assumes, and ship a portable fallback (e.g., a routing config for OpenRouter or a documented shim) so an external reader can actually run the pipeline. Addresses Shipped code assumes private infrastructure.
• Ship a real data/artifacts release. Add an explicit data-availability section that lists what is present and what is not, and commit (i) table_1.csv with ASR_paper / ASR_gen per attack, (ii) the aggregated 30 $\times$ 10 standardized ASR matrix, (iii) per-run synthesis-timing logs, (iv) paired no-repo/with-repo modules and ASRs behind Figure 2, (v) enhanced-refinement _improved.py for AIR and ISA, (vi) the 15 Kimi-ablation modules or their outputs, (vii) per-forge-stage token logs for Table 3, (viii) the Past-Tense trace and the QueryAttack 50-prompt dump, and (ix) snapshot dates / checkpoint IDs for GPT-4/4o/5.1 and Claude/Qwen/GPT-OSS. Addresses every bullet under Reproducibility & code except the private-infrastructure item above.
• Reconcile the two evaluation regimes. Add a within-attack sensitivity check that scores the same $\text{ASR}_\text{gen}$ generations under both the paper-matched judge and the standardized GPT-4o judge on at least a handful of attacks, so Section 4.2 fidelity and Section 5 comparability can be interpreted together. Addresses Two evaluation regimes never reconciled.
• Fix the precedence contradiction. Choose one precedence order (either $s_p \succ C \succ R$ with $R$ as tie-breaker, or R-as-gold-standard) and rewrite the auditor prompt (Appendix K) and the enhanced-refinement Phase-1 prompt so they no longer contradict Section 3.2 or each other. If the enhanced pass genuinely elevates $R$ over $s_p$, say so and label the enhanced modules "repo-faithful." Addresses Precedence between paper, plan, and repo is inconsistent (and Enhanced-refinement Phase-1 prompt overrides paper precedence).
• Soften the autonomy claims. Replace "without human involvement" / "without manual implementation effort" with concrete language enumerating the human choices (representative-setting selection, MATCHCFG, enhanced-refinement trigger, reference-repo retrieval), and fix the coder-prompt template so it reads args.provider instead of hard-coding provider="openai". Addresses Autonomy claims overstated.
• Address judge-choice bias. Cross-validate at least one column of Figure 4 (e.g., GPT-4o victim $\times$ 30 attacks) with a second, non-OpenAI judge such as a HarmBench classifier or a StrongREJECT rubric, and discuss whether the "formal wrappers 66.0% mean ASR" finding survives. Addresses Judge-choice bias risk.

Suggested.

• Widen the semantic audit and the ablation. Grow the semantic-fidelity audit beyond $n = 4$ and re-run each Kimi ablation cell across $\geq 3$ seeds; report variance. Addresses Small-n ablation and small-n audit.
• Use a non-Pearson iteration analysis. Replace the Pearson $r$ with Spearman, a permutation test, or a Kruskal–Wallis across iteration groups, and reframe the null result as "not detected at $n = 30$" rather than "not present." Addresses Statistically weak iteration-count analysis.
• Add per-paper aggregation. For Section 5 threshold counts and family/carrier means, report both attack-level ($n = 30$) and paper-level ($n \approx 26$) numbers so variant pairs (SATA-MLM/ELP, WordGame/+, RA-DRI/SRI, JAILCON-CIT/CVT) do not double-count. Addresses Variant confounding.
• Retitle 82.5% as amortization. Add a median per-attack-specific-LOC number alongside the 82.5% reuse share so readers can distinguish amortization from maintainability. Addresses 82.5% "reuse" is amortization.
• Soften TrojFill attribution or add runtime evidence. Either add runtime traces (max-token clipping, provider-specific system-prompt handling, retry policy) that establish the runtime cause of TrojFill's $-6.3$ pp gap, or rewrite the sentence as "no visible mechanism-level deviation." Addresses Static audit over-interprets TrojFill.
• Drop or ground "gray-box." Either add a concrete gray-box attack (logit or embedding access) or remove the qualifier from the future-work paragraph. Addresses "Black-box and gray-box" scoping is not grounded.
• Discuss and mitigate dual-use. Add a concrete release policy (gating, disclosure timeline relative to model-provider notification, eligibility criteria) to the impact statement. Addresses Dual-use release policy is unspecified.
• Fix the arithmetic and notation nits. Reconcile the with-repo $\delta = +19.8$ vs the per-attack sum 20.8, fix GPT-4o mean 74.5 vs heatmap 74.4, close the truncated "We set" sentence, harmonize planner/coder function signatures between the role definitions and Algorithm 1, and explicitly write $\tau = -10$ in the Require clause. Addresses Minor notation and consistency issues and With-repo δ arithmetic does not close.
• State the selection filter as a limitation. Add one sentence to Section 4.1 noting that restricting to AdvBench/JBB-reporting papers biases the population toward attacks whose original evaluation already aligns with JBF-EVAL, so fidelity does not generalize to the broader jailbreak literature. Addresses Selection bias in the reproduction set.