daVinci-Dev: Agent-native Mid-training for Software Engineering

Summary

The paper argues that agentic software engineering benefits from a new intermediate training stage — agent-native mid-training — that sits between raw pre-training and the SFT/RL post-training now standard for code agents. The conceptual move is to redesign mid-training data so it mirrors the information flow an agent actually sees at deployment. Two complementary corpora are proposed: contextually-native trajectories (D_ctx, 68.6B tokens), which bundle issue text, relevant file contents, and edits from GitHub PRs into a single sample so the localize → read → edit sequence is preserved rather than factorized into disjoint tasks; and environmentally-native trajectories (D_env, 3.1B raw tokens), which are unfiltered SWE-Agent rollouts of GLM-4.6 inside SWE-Rebench-style Docker environments with real unit tests. Starting from the non-coder Qwen2.5 base at 32B and 72B, the recipe reports 56.1% and 58.5% Pass@1 on SWE-Bench Verified under SWE-Agent, competitive with or exceeding the Kimi-Dev mid-training recipe using roughly half the mid-training tokens, with behavioral evidence that gains come primarily from better file localization. The contribution matters because it reframes what mid-training data for agents should look like — replacing factorized subtask supervision with trajectory-shaped supervision is a clean, testable hypothesis, and the sizable open corpora and pipeline code are a real service to the community. The main conceptual limitation is that the empirical story leans on a single evaluation family (SWE-Bench Verified and its Pro/Multilingual siblings, which the authors themselves flag as contaminated) and a single scaffold (SWE-Agent), so scaffold- and benchmark-invariance is not established, and the central bundled-vs-factorized hypothesis is never isolated by a controlled same-data ablation. Several headline comparisons (the Kimi-Dev gap, "less than half tokens," scientific-benchmark transfer) rest on inconsistent baselines that inflate the apparent effect. A code-execution reproduction confirmed the released repository is a data-construction pipeline only: the D_env corpus statistics reproduce cleanly at full scale, but no training or evaluation code exists, so the headline resolution rates could not be checked.

Strengths

• Conceptual contribution. The framing of mid-training data as agent-native — supervision that preserves the full action-observation trajectory rather than factorized subtasks — is a clear, testable design principle. The distinction between contextually-native (broad, template-organized PR bundles) and environmentally-native (executed rollouts with real test feedback) trajectories is intuitive and organizes the paper cleanly.
• Scale and openness. 68.6B contextually-native tokens plus 3.1B environmentally-native tokens is a substantial corpus, and releasing the data-construction pipeline, trajectory conversion/tokenization utilities, checkpoints, and — where licenses permit — the datasets is a meaningful step beyond the closed reports (KAT-Coder, GLM-4.5) the paper cites.
• Robustness sweep. The 32B/72B × weak/strong SFT × ±MT design in Table 1 is a reasonable stress test, and the paper reports per-entry standard errors up to 1.05%, more calibration than most comparable papers include.
• Data statistics hold up under execution. Re-running the released D_env pipeline at full scale over all 74,313 trajectories reproduced the ~3.1B-token total, the 128k discard behavior, and the passing/non-passing split within ~1% (see Reproducibility & code) — direct confirmation that the corpus-statistics claims are accurate.
• Behavioral analysis. Section 5.3 goes beyond aggregate resolution rate to trace the improvement to localization (edit-level recall/precision, View→Edit ratio), and the "more headroom at 32B" story is coherent with the reported numbers.
• Honesty about contamination. SWE-Bench Pro and Multilingual results are explicitly flagged as non-comparable due to repository overlap with the mid-training corpus — a norm that should be more common in this literature.

Weaknesses

• Central mechanism is not isolated. The paper's core thesis — that bundling context beats factorized subtask supervision — is supported only by the cross-recipe comparison to Kimi-Dev, which differs simultaneously in data source, PR selection, tokenizer, pipeline, and token budget. There is no head-to-head ablation training on the same PRs rendered factorized vs. bundled, so the gains cannot be attributed to the agent-native format specifically.
• Scaffold/benchmark invariance unestablished. Every headline number uses SWE-Agent on SWE-Bench Verified; resolution rate is highly sensitive to scaffold prompt, tool schema, and step budget, so the benefit is confounded with SWE-Agent-specific biases. This is the largest missing empirical check.
• Kimi-Dev headline gap uses the least favorable number. §4.4 pairs "same base model and agentic scaffold" with the 48.6% quoted figure, while Table 1 also reports an in-house Kimi-Dev reproduction at 56.2% under the authors' own harness. Under the fairer same-infrastructure comparison the gap shrinks from ~9.9 pp to ~2.3 pp.
• MT effect confounded with SFT-data choice. The strong-SFT configurations use D_env_pass, and the matched no-MT baseline already reaches 56.6% at 72B, so the true MT effect at 72B strong SFT is 1.9 pp, not the ~10 pp implied by the 58.5-vs-48.6 pairing. A "MT effect at matched SFT" summary is needed.
• Generalization claims overreach. The scientific-benchmark deltas the paper calls "more notable" (GPQA-Main +0.67 at 32B, SciBench +0.44 at 72B) are within plausible run-to-run noise and reported without error bars, yet framed as evidence that SE logic "fosters fundamental reasoning skills." The large code-benchmark gains are confounded with plain code exposure and non-coder-base format effects, and DS-1000 is not decontaminated at all.
• Token-count inconsistency. The corpus is 71.7B (Conclusion; raw) and 73.1B (abstract/§5.1; effective, after 3× upsampling), and the "less than half" claim is barely satisfied and only under the effective convention. The basis should be single and explicit.
• Selection bias in behavioral indicators. §5.3 discloses that error-recovery, planning, and debugging-cycle indicators were dropped for low SNR, leaving only high-SNR indicators that favor MT — a garden-of-forking-paths setup around a headline narrative.
• Under-specified protocols. The zero-shot MT-only evaluation (Table 4) runs un-fine-tuned checkpoints under a tool-calling scaffold without documenting how valid tool calls are produced; the 32k D_ctx length cap (vs. 128k deployment context) likely excludes the hardest multi-file cases; and the scaling-law extrapolation rests on ~2.7 orders of magnitude at a single mixture with an unstated checkpoint count.
• Minor issues. "No filter strategy" contradicts the 128k discard; the 32B marginal gain is reported as both +3.1 and +2.0 pp; Table 4's caption says D_ctx where the data is D_ctx_py; "the other 7 settings" does not match the View→Edit indicator; the Appendix-E leakage-ratio formula is missing; and the D_env task count / rollout distribution is unspecified.

Reproducibility & code

• Headline resolution rates — unverifiable from the artifact. An end-to-end inventory of the released repository confirms it is a data-construction pipeline only (Go + Rust + Python): no training loop, no SFT/RL code, no SWE-Agent/SWE-Bench harness, and no model weights (the only SWE-Bench string is a benchmark-repo exclusion filter). All model-accuracy claims — 58.5%/56.1%, the Kimi-Dev comparison, Tables 1/3/4/6, the scaling curve, and the behavioral indicators — are therefore out of scope for reproduction. This is a repository-completeness limitation, not evidence the numbers are wrong, but it leaves the central claims only textually attested; the overall replication score is correspondingly low.
• D_env corpus statistics — reproduced at full scale. Running the released env_traj_utils convert+tokenize pipeline over all 74,313 trajectories with the Qwen2.5-72B tokenizer and the 131,072-token discard rule produced 3.09B tokens (paper ~3.1B), 73,993 retained (320 discarded), and an 18,288 / 56,025 passing/non-passing split (paper ~18,500 / ~55,500) — all within ~1%. The pass/fail split, however, is not emitted by the shipped scripts and had to be computed downstream from a boolean success field.
• D_ctx construction — partial and Python-only. The tokenizer ran only on D_ctx_py, over ~0.07% of the input (~555 MB of ~771 GB), yielding ~57.1M tokens vs. the paper's 41.9B — a data-availability gap, not a code defect. There is no code path that builds D_ctx_gen (the 26.7B general subset), so the 68.6B combined total cannot be regenerated even in principle.
• HumanEval decontamination — method reproduces, count does not. The decontam_humaneval tool faithfully implements the 13-gram leakage-ratio procedure with τ=0.10 and runs over all 164 items, but on the ~0.07% corpus subset the max leakage ratio (0.0757) stayed below threshold, flagging 0 vs. the paper's 24. The algorithm is auditable; the count needs the full corpus. The EvalPlus count is never reported at all.
• Rollout driver and D_env_pass filter — absent. The utilities operate on already-collected rollouts; the rollout driver (Docker construction, GLM-4.6 wiring, per-task policy) is not shipped, and the test-passing selection filter that produces D_env_pass (the SFT-consumed subset) is not implemented — only the 128k length-discard rule is. D_env/D_env_pass cannot be regenerated from the release.
• Evaluation-harness fixes and training configs — not shipped. The ~7 Appendix-B harness adjustments are described in prose only, with no patch/fork; and no per-row training configs or launch commands for Tables 1/4 ship. Both materially affect the reported numbers. Running the data utilities also surfaced an undeclared jinja2 dependency, indicating the install instructions were not tested from a clean environment.

Recommended Changes

Essential

• Add the controlled bundled-vs-factorized ablation. Train on the same set of PRs rendered in factorized form vs. the agent-native bundled form (matched tokens/compute) to isolate the paper's central mechanism, rather than inferring it from the Kimi-Dev comparison.
• Add a scaffold-invariance experiment. Report daVinci-Dev under at least one non-SWE-Agent scaffold (OpenHands, R2E-Gym, or MOpenHands) so the benefit is not confounded with scaffold-specific biases.
• Fix the Kimi-Dev headline and separate MT from SFT-data. In §4.4, supplement "surpassing 48.6%" with the same-infrastructure 56.2% reproduction and its ~2.3 pp margin, and add an explicit "MT effect at matched-SFT no-MT" summary (e.g., 58.5% vs. 56.6% at 72B).
• Reconcile the token counts. Use one convention (raw or effective) across abstract, §5.1, and Conclusion, note when the ~150B Kimi-Dev figure includes upsampled tokens, and restate or drop "less than half" against a matched basis.
• Ship the evaluation runner and Appendix-B fixes as code. Release a pinned SWE-Agent config, the evaluation entry-point (ideally with a metrics JSON attesting to the numbers), and the ~7 harness adjustments as a patch set or fork.
• Release the D_env rollout driver and D_env_pass selection filter, plus per-row training configs/launch commands for Tables 1/4 and the D_ctx_gen builder, so the corpora and ablations are regenerable; add a per-subset corpus manifest (token/PR/repo counts as JSON).

Suggested

• Add error bars to Table 3 and moderate the "transfer learning"/"fundamental reasoning" language for entries within noise; decontaminate DS-1000 or flag it, and report the EvalPlus decontamination count separately.
• Report the discarded behavioral indicators. Publish signs and unadjusted deltas of the error-recovery, planning, and debugging-cycle indicators, or preregister the SNR threshold.
• Document the zero-shot MT-only protocol (how un-fine-tuned checkpoints emit valid SWE-Agent tool calls) and state whether it matches the Table 1 setup.
• Justify or ablate the 32k D_ctx cap at 64k/128k on D_ctx_py, and state the number of evaluation checkpoints and confidence bands behind Figure 4's R² values.
• Clarify D_env task counts and rollout distribution, and state explicitly that decontamination applies to both D_ctx_gen and D_ctx_py.
• Fix minor items: insert the Appendix-E leakage-ratio formula, qualify "no filter strategy" as "no success/quality filter," reconcile the +3.1/+2.0 pp 32B figures, correct Table 4's caption to D_ctx_py, fix the "other 7 settings" wording, correct Table 5's duplicated column header, and add the missing jinja2 dependency to the install instructions.