CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability

Summary

CVE-Factory presents a six-stage multi-agent pipeline that turns sparse CVE metadata into fully executable agentic tasks — an interactive Docker environment, test_func.py/test_vuln.py, a reference solution.sh, and a first-person bug-report task.yaml. The conceptual move is to decouple long-horizon CVE reproduction into three independent generation stages (Analyzer / Generator / Builder) and three progressive coupling stages (Validator / Solver / Checker), each instantiated as a Claude Code session with its own context, and coordinated by a central Orchestrator through continue / error / pause signals plus a file-ownership feedback map. The framework is exercised in four ways: (i) cross-validation against 215 expert PatchEval reproductions (95.35% solution correctness, 96.13% environment fidelity, 73.91% test-quality "equal or better"); (ii) reproduction of 554 — or 454, the paper reports both — real-world CVEs from May–December 2025 with a headline 66.2% "verified success"; (iii) the release of LiveCVEBench, 190 tasks over 14 languages, 74 CWEs, and 153 repositories (27.8% multi-container, 10% AI-tooling), with a 10-model × 4-scaffold evaluation; and (iv) large-scale synthesis of >1000 training tasks that lift Qwen3-32B from 5.3% to 35.8% on LiveCVEBench and generalize to Terminal Bench.

The contribution is genuine: an automated, containerized, multi-language pipeline for agentic CVE tasks is a real unlock over static CVE-fix datasets and over CVE-Genie–style pipelines, and a continuously-updated benchmark addresses a real contamination/drift problem for security agents. The main conceptual limitation is that the central quality claim rests heavily on LLM-as-judge evaluations (Comparer, Judger) whose calibration against humans is asserted rather than reported, while several key numbers are internally inconsistent (454 vs 554 candidate CVEs; 94 vs 52 mock failures; 42.33/38.10% in the LiveCVEBench text vs 41.27/34.39% in Table 3; 12.5→31.3% implying 2.5× yet reported as 2.3×; 15.50% vs 19.3% for AI success; XSS n = 850 in a 190-task benchmark). Executing what the release provides confirms that the benchmark artifact itself is well-formed — 30 shipped packages ran through the repo's own Docker verification loop with 24/29 verifying end-to-end — but every training and multi-model evaluation number rests on external artifacts (a fine-tuning driver, trajectory-collection scripts, and LCB/PE/TB harnesses are absent from the release), and the paper's Table 1 cross-execution could not be re-run at scale (only 1 of 8 staged CVEs reached a verdict). Combined with unpinned frontier-LLM configurations and absent per-CVE outcome/manual-verification labels, most aggregate numbers remain in a "paper-only" state on top of an otherwise well-scoped release.

Strengths

• Conceptual contribution. The decouple-then-couple design with owner-based feedback is a clean way to bound each agent's context and blast radius while still allowing revisions to route back to the responsible creator; the "context isolation via Markdown handoffs + orchestrator-owned signals" pattern is a reusable idea beyond CVEs.
• Scale and realism of the benchmark. LiveCVEBench spans 14 languages and 74 CWEs (up from 3/65 in PatchEval), with 27.75% multi-container tasks and 10% AI-tooling tasks — genuinely richer than prior CVE benchmarks, and the shipped task directories back the composition claims (language and CWE tags reproduce Tables 9 and 10; multi-container counts match Table 2 within one).
• Executable artifacts hold up under a spot re-run. On a seed-0 stratified sample of 30 shipped task packages, the repo's own build → vulnerable → apply-solution → fixed loop verified 24 of 29 packages (excluding one build-timeout), confirming that the task packages themselves are runnable and internally coherent even if the aggregate 66.2% cannot be reconstructed.
• Honest analysis of the raw test pass rate. The paper does not hide the low 54.88% raw pass rate of $C_t$ on $P_e$ and works through two benign sub-causes (16.03% stricter, 29.09% end-to-end). That is more transparent than the 96.13% headline alone would suggest.
• Substantial training gains. Fine-tuned Qwen3-32B goes from 5.3% to 35.8% on LiveCVEBench with cross-domain transfer to Terminal Bench — if independently reproducible, this is a strong argument for the utility of large-scale executable security environments.
• Useful ablations and behavioural analysis. The merge-Stages-1–3 (71.0%) and merge-Stages-4–6 (58.1%) ablations, the 15.88 → 57.54 step-count shift under fine-tuning, and the pre-vs-post-release CVE decomposition are all interesting; the deployment analysis of OpenHands / Terminus-2 / Mini-SWE-Agent scaffolds is a useful side-contribution.
• Discussed limitations are honest. Appendix G openly notes the missing model-scaling analysis, the absence of RL in interactive CVE environments, and the containerization limits.

Weaknesses

• Discrepant real-world CVE counts (454 vs 554). The abstract and introduction quote 454 CVEs, while §4.2 sets the candidate set at 554 (with 499 successes + 55 failures). One of the two is a typo, and it propagates into the paper's second headline number.
• Headline 66.2% verified success is not reconstructible in-text. With 499 successes and 187 manual-verification failures, verified successes = 312; none of the natural denominators (554, 499, 554 − 42, 499 − 42) yield 66.2%. The denominator that fits (≈471) appears only in the released README (312/471), never in the manuscript, so the paper's headline number is not auditable from the paper alone.
• Solver is permitted to modify the very tests that judge it. This directly conflicts with the architecture section's blinding constraint and the emphasis on "objective script-based verification." No safeguard against goalpost-shifting is described.
• Central quality claim depends on an LLM-as-judge Comparer with no human calibration. The 73.91% "Equal-or-Better" number is decided by an LLM comparison agent; given the §5 finding that the V1 Judger made 16/70 false positives, human calibration on a sample and inter-run stability should be reported for the Comparer too.
• Verification is one-sided. Manual verification samples only the 499 CVE-Factory-reported successes, not the 55 pipeline failures — the false-negative rate is never estimated (particularly consequential given the 3-retry cap).
• Empirical training scope is a single 32B model. The third headline contribution ("first large-scale scaling of agentic security tasks") is only demonstrated on Qwen3-32B; there is no model-size scaling curve, no cross-family check, and no RL — as §G acknowledges. As presented, the claim reaches beyond the evidence.
• Ablation and baseline sample is small (n = 31) with no CIs. The 90.3% / 71.0% / 58.1% split spans ~4-CVE swings; the "complementary bottlenecks" narrative should not be pushed without confidence intervals, and only CVE-Genie is exercised as a multi-agent baseline (SWE-Factory / SWE-smith / R2E-Gym / SWA are cited but never run).
• Internal inconsistency in Table 3 vs the LiveCVEBench evaluation text. Opus 4.5 / Sonnet 4.5 are 41.27 / 34.39 in Table 3 vs 42.33 / 38.10 in §4.2.3, with no caption tying rows to a scaffold; a reader cannot tell which number is authoritative. Similarly the abstract's "surpassing Claude 4.5 Sonnet" is only true on LCB and by ≈1.4 points; §4.3 says "approaching."
• Mock-failure count is 94 in the main text and 52 in Table 7. The paper reads as if the same population is being counted; if two populations, they must be labelled with denominators.
• Headline 95%/96% relies on undisclosed post-hoc exclusion. The 96.13% is measured after excluding cases where $C_t$ is "demonstrably stricter"; the raw pass rate ($C_t$ on $P_e$) is only 54.88%. The abstract and intro should carry both numbers or hedge.
• Case-study log inconsistency. The Appendix F "0 failed, 24 passed" summary is preceded by FAILED lines and AssertionError strings on tests marked PASSED. As displayed, the log cannot have come from a single clean run, which undermines the "significantly better than PatchEval" conclusion built on it.
• "20 workers, <5 hours" arithmetic disagrees with the 48-min mean. 215 × 48/20 = 8.6 h, well above 5 h. Either the count, the mean, or the wall-clock is misstated.
• Terminal Bench multiplier (2.3×) is inconsistent with the paper's own endpoints (12.5→31.3 = 2.50×). The released README uses 28.75 as the endpoint (the CVE-4k row in Table 3), suggesting the abstract's 31.3 came from the CVE-3k row — please align.
• Missing composite score formula. Appendix B.2 announces $S_{final}$ and defines $S_{div}$ / $S_{nov}$ but never writes the equation; $S_{base}$ is never anchored to Table 5's Reproduce Score. This blocks reconstructing the diversity-sampling pipeline.
• Ambiguous units. XSS n = 850 for a 190-task benchmark reads like a task count but is probably an (evaluation-pair) count; Table 9's Count column sums to 294, not 190, without labelling that a task can be tagged with multiple languages.
• Simpson's-paradox / repository-source causal claim is under-supported. The reversal is shown for one CWE (SQL injection); base rates, subgroup sizes, and mixing proportions are not reported, and "unlikely to be a causal driver" is drawn from a single controlled category.
• Small arithmetic issues. 27.75% multi-container vs 53/190 = 27.89% (Table 2); range "19.05%–42.33%" excludes lower OpenHands cells (12.70%); "16.03%/29.09% of failures" are arithmetically fractions of the total sample; AI aggregate reported as both 15.50% and 19.3%.

Reproducibility & code

The released repository ships the benchmark artifact in a usable form: 190 task directories under LiveCVEBench/, each with task.yaml, Dockerfile, docker-compose.yaml, tests/, run-tests.sh, and solution.sh; per-task language and CWE tags reproduce Tables 9 and 10, and multi-container counts reproduce Table 2 within one. The Analyzer / Generator / Builder / Validator / Solver / Checker / Comparer / Judger prompts are shipped as agent markdown files, and the sampling pipeline (cve-sampler/) is available. Executing the shipped verification loop on a seed-0 stratified sample of 30 packages yielded 24/29 verified end-to-end (82.76% excluding one build timeout), which confirms that individual task packages are runnable but does not reconstruct the paper's 66.2% (different population, different denominator). Concretely:

• Table 1 cross-execution is not runnable from the release. The PatchEval runner is external, and of 8 CVEs staged for cross-execution only cve-2018-18074 reached a verdict; Solution Correctness and Environment Fidelity produced no denominator at all, and Test Quality yielded a one-sample observation (0 of 1 rated equal or better) that cannot be interpreted as evidence about the 73.91% figure. The shortfall is scale plus a missing external harness, not a broken code path.
• Training and multi-model evaluation are entirely un-runnable from the release. A gap audit found no SFT / DeepSpeed / torchrun entrypoint, no trajectory-collection driver, and none of the LiveCVEBench / PatchEval / Terminal-Bench evaluation harnesses. The 35.79 / 23.58 / 28.75 / 5.29 numbers appear only in README.md and model_card.md restating the paper — not as measurements produced anywhere in the release. Combined with the missing eval harness, the 6.8× / 4.2× / 2.3× training-gain headline cannot be reproduced from this artifact.
• The 66.2% headline is un-audited by the release. The 554-candidate list and the per-CVE manual-verification annotations are not shipped. The README's "312 of 471 reviewed (66.2%)" line is documentation restating the paper (as an audit script confirmed), not a reproduced measurement. This also blocks resolving the 94-vs-52 mock-count gap.
• The failure classifier in the verification loop is fragile. When run under rootless podman, the classifier initially reported 100% success because a benign cgroup warning was absorbed as an infrastructure failure and discounted; after auditing, 5 of 6 failures on the 30-package sample were reclassified as genuine reproduction defects (2 no-vuln-tests-found, 2 vuln-test-unexpected-pass, 1 vuln-stage-fail). Downstream aggregates are quietly runtime-dependent.
• Frontier-LLM configuration is not pinned. No model version-string, seed, temperature, max_tokens, or rate-limit envelope is fixed anywhere in the shipped configuration. Since the pipeline and evaluation harness both run through frontier LLMs, every aggregate number is un-pinnable within several points across re-runs.
• Per-CVE evaluation outcome logs are not shipped. Tables 3, 8 and Figures 3, 4, together with the AI-vs-Web, pre-vs-post-release, per-CWE-per-language, and OpenHands-vs-Terminus-2 analyses in Appendix C, all depend on per-CVE, per-(model, scaffold) outcomes not in the artifact.
• Timing/cost data behind Figures 5 and 6 are not shipped. The 2,000-run per-agent time/cost dataset is absent; the paper itself carries small internal drift between Fig 5 and Fig 6 means (Builder 12.0 vs 11.8, Validator 15.0 vs 14.7, etc.), only detectable because both figures are printed.
• Ablation setup and Judger V1 are absent. The 31-CVE sample list drawn from CVE-Genie, the merged-agent runtime entry points for Table 4's 71.0% / 58.1% cells, and the V1 Judger prompt behind "16/70 false positives" are not in the release, so §5's key comparisons are un-checkable from code.

Recommended Changes

Essential

• Reconcile the CVE count (454 vs 554). Pick one and use it consistently through the abstract, introduction, and §4.2; state the arithmetic path from 554 → 499 → 187 → 66.2% so the headline is auditable (denominator, "objectively irreproducible" count) — the released README's 312/471 chain should also appear in the manuscript.
• Reconcile mock-failure counts (94 in prose vs 52 in Table 7). Label the two populations and give explicit denominators (187 vs 145); otherwise the reader cannot tell whether these are the same set counted differently.
• Restrict Solver test edits, or add a safeguard. Either forbid Solver from modifying test_vuln.py / test_func.py, or add an explicit Checker/Judger pass that flags Solver-modified tests as suspect; this addresses the blinding-constraint contradiction raised in Weaknesses.
• Ship (or clearly reference) the training driver and LCB / PE / TB evaluation harness. The gap audit found no SFT / trajectory-collection / evaluation entry point in the release, so every row of Table 3 is un-runnable from this artifact. Either add the fine-tuning driver, trajectory-collection scripts, and per-benchmark eval adapters, or scope the training claim explicitly to "data + checkpoints released, harness external" and name each external harness.
• Release (or clearly reference) the PatchEval cross-execution driver and its per-CVE outcomes. Without them, Table 1's 95.35 / 96.13 / 73.91 cells are un-checkable end-to-end; only 1 of 8 staged CVEs even reached a verdict in a re-execution attempt.
• Release the manual-verification annotations and the failure-taxonomy rubric. A prose-graded (CVE-ID → verdict + category) table lets readers audit 66.2%, Table 7, and the 94-vs-52 mock gap in one step.
• Report human calibration for the Comparer and Judger. Given the §5 admission that V1 Judger made 16/70 false positives, report human-annotator agreement (Cohen's $\kappa$ or plain accuracy) with the Comparer on a sample of test pairs and with V2 Judger beyond the "500 CVEs full agreement" line — including the labeling protocol.
• Fix or hedge the Terminal Bench multiplier. 12.5% → 31.3% is 2.50×, not 2.3×; correct either the multiplier or the endpoints, and use the ratio convention consistently.
• Align Table 3 with the LiveCVEBench evaluation text and Table 8. State which scaffold each Table 3 row uses; pick a single value per (model, benchmark) throughout; and either soften the abstract's "surpassing Claude 4.5 Sonnet" to "matching on LiveCVEBench (35.79 vs 34.39)" or scope it explicitly to LCB.
• Regenerate the CVE-2021-21384 case-study log. The "0 failed, 24 passed" summary above FAILED lines is not a valid pytest output; produce a clean single-run log, and add an explicit four-way head-to-head (PatchEval × PatchEval tests, PatchEval × CVE-Factory tests, CVE-Factory × both) that supports the "significantly better" verdict.
• Pin frontier-LLM configuration. Add a models.yaml (or equivalent) fixing exact model version-strings, temperatures, seeds, and max_tokens for every reported experiment; otherwise every aggregate number floats several points across re-runs even with unchanged code.

Suggested

• Ship per-CVE, per-cell evaluation outcomes. A CSV mapping (CVE-ID, model, scaffold) → (pass/fail, turns, tokens, wall-clock) would let readers verify Tables 3 / 8 and Figures 3 / 4 and rerun the AI/Web, pre/post-release, and per-CWE analyses in Appendix C.
• Report per-seed / per-run variance for at least the headline LiveCVEBench and PatchEval numbers; the current 3–4 point differences between "same-model, different-scaffold" cells cannot be interpreted without noise estimates.
• Broaden the training scaling story. Even one additional model size (e.g., 7B and 70B, or a non-Qwen family) would separate "CVE-Factory data helps a small model close the gap" from "CVE-Factory data helps agents in general," addressing the empirical-scope Weakness.
• Add confidence intervals to Table 4. With n = 31, report Wilson/binomial intervals for CVE-Factory / merge-1–3 / merge-4–6 / CVE-Genie; consider expanding to n ≥ 100.
• Audit CVE-Factory-reported failures. Spot-check a sample of the 55 pipeline failures against a manual reproduction attempt to estimate false negatives, so the paper reports both precision and recall of the pipeline's success signal.
• Harden the verification-loop failure classifier. Make it return-code aware, tolerant of benign cgroup warnings under rootless podman, and explicit about "no vuln tests found" being a failure — otherwise downstream aggregates depend silently on the container runtime.
• Write out the composite scoring formula. Give $S_{final}$ as an equation (e.g., $S_{final} = S_{base} + S_{div} + S_{nov}$) and anchor $S_{base}$ to Table 5's Reproduce Score.
• Clarify units. State that Table 9's Count is task appearances (base 294), that Appendix C's XSS n = 850 is (task × configuration) pairs, and that the "16.03% of failures" phrasing means "16.03% of all runs."
• Strengthen the Simpson's-paradox / repository-source analysis. Report base rates, subgroup sizes, and mixing proportions for GitHub vs non-GitHub; extend the controlled comparison to more than SQL injection, or soften "unlikely to be a causal driver."
• Fix small arithmetic issues. Align 27.75% with 53/190 = 27.89% (Table 2); state the actual observed floor (12.70%) or scope the "19.05%–42.33%" range; reconcile AI aggregate rates (15.50% vs 19.3%).
• Add more multi-agent baselines (a SWE-Factory or SWE-smith run on a matched subset) to strengthen the "multi-agent decomposition plus flexible agent autonomy" comparison beyond CVE-Genie alone.
• Ship the ablation 31-CVE list and merged-agent runtime scripts, plus the V1 Judger prompt and 70/500-case evaluation sets so Table 4 and the Judger-calibration claim are checkable from code.