Reinforcement Learning with Evolving Rubrics for Deep Research

Summary

The paper introduces DR Tulu-8B, presented as the first fully-open model trained end-to-end for long-form, citation-backed deep-research tasks, and its training method Reinforcement Learning with Evolving Rubrics (RLER). The conceptual move is to close a well-known gap in RL-with-verifiable-rewards for open-ended generation: rather than training against a static verifier that is either a general LM-judge (prone to hacking) or a closed-book instance-wise rubric (bounded by the generator's parametric knowledge), RLER maintains a per-prompt rubric buffer that (i) is initialised from search-grounded rubrics generated with retrieved documents as privileged information, (ii) evolves at every step by contrasting current rollouts and folding newly-searched evidence into positive/negative rubric items, and (iii) is pruned by rubric-level reward standard deviation, retaining only the most discriminative items. The framework is packaged with an SFT-cold-start recipe on GPT-5 teacher trajectories, an agent library dr-agent-lib, and a specialised medical-domain benchmark (GeneticDiseasesQA). Empirically DR Tulu-8B is competitive across four long-form benchmarks (SQAv2, HealthBench, ResearchQA, DRB), meaningfully ahead of open baselines and roughly on par with OpenAI Deep Research on average, at a dramatically lower per-query cost — a strong result for an 8B open model.

The contribution has real force: privileged-information rubrics and adaptive verifiers together plug a meaningful hole in RLVR-style training for tasks whose desiderata are ill-specified, and the executed reproduction confirms that the released checkpoint delivers the headline aggregate (65.9 reproduced vs 65.6 in the paper, three of four benchmarks). However, the isolated marginal effect of the evolving component over a strong initial-search-based rubric baseline is characterised only as 'up to a 2-point drop' on a 3-benchmark average (Figure 6), which is close to the observed evaluation variance and leaves the paper's central methodological claim less sharp than the framing implies — and the executed reproduction was not able to touch any of the rubric-ablation runs (Tables 3–4 and Figure 6) because no RL training was replayed and no intermediate checkpoints are released, so the mechanism the title advertises is presently unverified end-to-end outside the authors' own runs. Several supporting analyses (the open-judge ablation with truncated context, the anecdotal 'robust to tool errors' section, the mechanism-conflating CACNA1A exemplar) also carry confounds. The release is genuinely open, but marquee axes such as cost/query, ablation branching, evaluation subsets, and figure aggregation are not scripted end-to-end in the released code.

Strengths

• Conceptual contribution. The idea of a per-prompt rubric buffer whose items are grounded in retrieved evidence and continuously refreshed against fresh rollouts is a clean generalisation of both static rubric-as-reward (RaR) and closed-book online rubrics, and it directly addresses the 'ungrounded verifier' failure mode that has hampered RL on long-form tasks.
• Empirical breadth. Four independent long-form benchmarks plus short-form QA generalisation (Table 2), plus a real expert-curated genetics evaluation, is a substantially broader empirical footprint than most open DR work.
• Genuine openness. The stack is unusually complete: data, code, model checkpoints, agent library, evaluation suite, and training-time infrastructure for asynchronous tool calls are all publicly released, which is the actual bottleneck the paper claims to address for the community. The shipped inference stack executes as described: in a code-executed re-run, the vLLM policy server, browse agent, and MCP tool backend all came up, and the pipeline produced coherent, cited long-form reports.
• Headline aggregate reproduces. The paper's average of 65.6 across four long-form benchmarks was checked at 65.9 on three reproducible benchmarks (SQAv2 excluded due to a gated HF dataset), with rel-err ~0.5%, and the RL-over-SFT direction held on every benchmark measured.
• Well-designed ablations. The rubric-type ladder (general → closed-book → initial search-based → evolving), the SFT-mixture ablations, the browser-tool swap, the tool-budget sweep, and the on-policy-SFT comparison are the right controls to make the design story sharp.
• Transparency about limitations. The paper openly reports search-engine variance (Table 14), judge variance (Table 16), the train/test reward mismatch (Appendix I.8), and tool-error episodes during training. Few DR papers surface these caveats.
• Cost-efficiency framing. The Pareto-frontier presentation (Figure 1, Table 12) is a useful reframing for a community that has focused on raw quality; the executed reproduction independently recomputed per-query cost at $0.00135 against the paper's $0.0019, preserving the ~1000× cheaper-than-OpenAI-DR framing.

Weaknesses

• Central claim of the evolving component is under-quantified. The gain of evolving-over-initial rubrics is only 'up to 2 points' on a three-benchmark average (Figure 6). Given search variance (Table 14) and judge variance (Table 16 shows a 17.1-point absolute HealthBench difference between two GPT-4.1 judge runs), this margin is close to noise. Per-benchmark deltas with seeds are not reported, so it is hard to say whether the evolving mechanism is reliably above the floor.
• Open-judge ablation is confounded. Table 4 is used to conclude that RLER 'does not rely on a strong proprietary judge', but the Qwen3-8B rubric generator sees only final answers (context-length disclosure in Appendix I.4), not the full trajectory with search context — i.e., the privileged-information mechanism that defines RLER is partially disabled in the open-judge arm. The 1.3-point gap therefore mixes judge quality with a truncation effect.
• Baseline framing on SQAv2 hides a near-tie. DR Tulu-8B's SQAv2 rubric score (88.3) is 0.6 above Ai2 ScholarQA (87.7); Table 11 further shows Ai2 ScholarQA leads on both citation precision (92.4 vs 90.5) and citation recall (81.2 vs 71.6). The 'outperforms all open deep research models' framing works only because Ai2 ScholarQA is categorised as fixed-pipeline (proprietary backbone). That is a defensible choice but should be surfaced rather than left to a footnote.
• Off-label evaluation of short-form baselines. Search-R1, ASearcher, and similar systems are asked to produce long-form report-style outputs and score in the single digits on DRB and HealthBench. This is a design mismatch on top of a capability gap, and the aggregated 15.6-point lead over Tongyi DR partly reflects task-format incompatibility, which should be qualified.
• Ill-specified components in the reward system. The rubric-based score aggregation is announced ('we evaluate a final response $y$ using the rubric-based score') but never printed; the buffer management is described as 'filters, merges, and ranks' but no merge criterion is given; the citation Recall and Precision definitions each print both cases as 'If $I(c) = \emptyset$', making the citation reward ill-defined as written; capital $R$ is overloaded for both individual rubrics and rubric sets; the paper's own text says '5 total tool calls' while Table 9 and Section 5.1 say 10.
• Buffer pruning could produce a moving-goalpost verifier. Keeping only top-$K_{max}=5$ rubrics by reward-std across steps favours short-horizon distinguishability. Stable-but-correct criteria whose variance transiently collapses when the current batch satisfies them uniformly may be pruned even though they still matter. A short discussion (buffer churn rate; sensitivity to $K_{max}$) would justify the design.
• Genetic-domain exemplar is internally inconsistent. The CACNA1A example (Figures 22–25), the sample explicitly held up as a successful GeneticDiseasesQA response, stacks GoF, haploinsufficiency, and dominant-negative as a single 'most parsimonious pathomechanism' despite acknowledging no direct functional data, and cites the same snippet (7a079b1d-6) for opposing mechanistic claims elsewhere in the report. As the marquee expert-domain demo, this undercuts the 'reliable multi-source reasoning' framing. In addition, at $n=47$ questions and 3-10 trials per system, the head-to-head Figure 3 comparisons cannot resolve small gaps.
• Anecdotal tool-error robustness. The 'somewhat robust to tool errors' claim is drawn from an unplanned Serper credit-outage during training and is confounded with 900 continuing training steps; it is presented as a discovery but is not a controlled experiment.
• Overstated verifier-scaling claim. The framing that RLER can 'extend the verifier's capabilities far beyond what infinite reasoning tokens alone can achieve' is not supported by an experiment that varies verifier reasoning budget at matched compute.
• Prompt-filter cutoffs are unmotivated. The GPT-5-based query grader uses >3 for OpenScholar and >2 for SearchArena (yielding 20% vs 10% retention). Given how much of the SFT and RL data flows from these two sources, the asymmetric thresholds warrant justification or a sensitivity check.
• Reference and typographical hygiene. Duplicate Mallen et al. entries (2022 arXiv and 2023 ACL); the Table 5 acronym 'SFR-EDR' is expanded as 'SFT-Enterprise Deep Research'; cost footnotes state pricing per single token where per-million is the only unit that fits Table 12.
• Terminology precision. The abstract and main-results text use '%' for what are absolute point differences on rubric metrics ('15.6% over Tongyi DR', '0.7% over OpenAI DR'), which invites misinterpretation as relative improvement.

Reproducibility & code

The release is unusually complete for a DR paper, and the shipped inference stack executed as described in a full code-executed replication: the vLLM policy server, the browse agent, and the MCP tool backend all came up, and the pipeline emitted 15–22K-character reports with inline citations. Every fix required for the run to complete came from missing environment credentials (Serper, Jina, OpenAI, Gemini, OpenRouter) rather than broken code. That said, several central numbers of the paper are not reproducible end-to-end from the released artefacts.

• Headline aggregates. DR Tulu-8B (RL) on the three non-gated long-form benchmarks reproduced at 65.9 vs 65.6 in the paper (rel err ~0.5%), and RL cleanly beat SFT on every benchmark measured (+22.7 HealthBench, +20.7 DRB, +17.4 ResearchQA). The per-benchmark RL-vs-SFT deltas overshoot the paper's stated '6.4–16.0' band, most plausibly due to the reproducer's substituted judge (Mistral in place of GPT-4.1/4.1-mini) and degraded retrieval — not a paper defect — but the paper's own numbers are therefore unverifiable in this environment without the exact judge and retrieval stack. Cost per query recomputed at $0.00135 vs $0.0019, the right order of magnitude.
• RLER mechanism is presently untested outside the authors' runs. The evolving-rubric ablation (Figure 6), the general/closed-book/search-based static-rubric ladder (Table 3), and the GPT-vs-Qwen judge ablation (Table 4) all require RL training that was not replayed here — no RL training was executed at all in the reproduction, and no SFTv0.1 or intermediate RL step-650/1000/1900 checkpoints are released. The mechanism is present in the code (apply_adaptive_rubric_reward, no_citation_reward toggles in grpo_fast.py) but the branching protocol is prose-only, so the paper's title claim is not currently verifiable without a full 27,000-GPU-hour training campaign.
• Baseline rows are not reproducible. The 15.6-point lead over Tongyi DR and analogous comparisons against Search-R1, ASearcher, WebExplorer, and WebThinker required baseline runs that the paper describes but does not fully script; the reproducer evaluated only DR Tulu-8B (RL) and (SFT), so nothing in Table 1's comparative headline number is checkable end-to-end.
• Short-form results diverge. DR Tulu-8B (RL) short-form averages dropped to 16.7% (SimpleQA 20, 2Wiki 30, WebWalker 0 at $n=10$), against the paper's 61.3. The reproducer attributes this to the keyless DuckDuckGo fallback replacing Serper, which specifically hurts exact-match factoid retrieval; the paper's own Section I.7.1 quantifies real drift in search returns. Because none of the eval-time search returns are cached and no fixed sample IDs are versioned, this is a first-order source of drift that reproducers will hit.
• SQAv2 is gated and cannot be evaluated as documented. The allenai/asta-bench dataset needs an approved HF token; the environment's token returned HTTP 401, and the entire SQAv2 column of every headline table was skipped. The dataset gate is out of the authors' control, but combined with the fact that SQAv2 is the strongest single result (88.3 vs 87.7 for Ai2 ScholarQA), an alternative distribution/replication path (e.g., a snapshot of the eval set + a released Gemini judge script) would let more reviewers verify the marquee number.
• Cost-per-query calculator missing. The 'three orders of magnitude cheaper' story depends on OpenRouter pricing plus tool-API costs, but no script computes cost per query. The reproducer had to write one from scratch to independently reach $0.00135; the missing script recreates exactly the drift the paper wants to eliminate.
• Ablation subsets and eval manifests are not versioned. Figure 4's 5%/10% SFT subsets have no seeds; the ~1000-example HealthBench, 2Wiki, and ResearchQA subsets are re-sampled at eval time; row-level reproduction of Tables 1, 3, or 4 is therefore not possible.
• Aggregation and plotting utilities are minimal. Tool-distribution (Figure 30), training-curve (Figure 26), and cost/perf Pareto (Figure 1) figures require log aggregation but no plotting scripts are shipped; the reproducer added ad-hoc analysis scripts. In addition, scripts/evaluate.py never emits a top-level deep_research_bench_eval_results.json, forcing manual aggregation of DRB's RACE and FACT metrics.
• Documentation gap on required paid keys. The top-level README lists Serper/S2/Jina/Gemini as 'free keys', but faithful reproduction also requires an OpenAI key (GPT-4.1-mini hardcoded in the ResearchQA evaluator and used for HealthBench grading) and Serper is not free at the evaluation volume. A user following the documented setup will configure four services and then fail at scoring.
• Compute barrier. The full RL run is 27,000 H100-hours over 70 days; without intermediate SFTv0.1 and step-650/1000/1900 RL checkpoints, most ablations in Section 6 cannot be replayed at reasonable cost even in principle.

Recommended Changes

Essential

• Report per-benchmark evolving-rubric deltas with variance. Replace 'up to a 2-point drop' with per-benchmark, multi-seed effect sizes so the central RLER claim is clearly above the search / judge variance quantified in Tables 14 and 16. Address the corresponding weakness bullet.
• De-confound the open-judge ablation. Either (a) match the input format across arms — pass full trajectories to both GPT-4.1 and Qwen3-8B (chunk if needed) — or (b) explicitly separate 'judge model' effect from 'trajectory truncation' effect, e.g., by running GPT-4.1 with truncated inputs as a third row in Table 4. Ties directly to the confounded-ablation weakness.
• Fix the ill-specified reward components. (i) Restore the rubric-based-score aggregation equation promised at 'we evaluate a final response $y$ using the rubric-based score'; (ii) correct the two branches of the Recall and Precision definitions (first branch should read $I(c) \ne \emptyset$); (iii) specify the 'merges' step of buffer management or drop the word; (iv) unify capital-$R$ notation and make weight semantics explicit.
• Fix the cost pricing footnotes. Change 'per input token' / 'per output token' to 'per million input tokens' / 'per million output tokens' in Appendix I.5 footnotes 12–14, or the cost claims cannot be reconciled with Table 12.
• Ship the cost-per-query calculator. Release the pricing + tool-cost accounting script that produced Figure 1 and Table 12 so the 'three orders of magnitude cheaper' claim is reproducible from the released code without reproducers reimplementing it (as the replication had to). Addresses the Reproducibility cost-calculator bullet.
• Ship the RLER ablation branching scripts and intermediate checkpoints. The RLER on/off (Figure 6), citation-reward on/off (Table 13), rubric-type ladder (Table 3), and GPT-vs-Qwen judge (Table 4) results are all currently unverifiable outside the authors' own runs. Release SFTv0.1 and the RL step-650/1000/1900 checkpoints plus the launch/resume commands so Section 6 ablations are replayable without a 27,000-GPU-hour re-run. Ties directly to the 'RLER mechanism is untested outside the authors' runs' bullet.
• Correct the '5 tool calls' inconsistency. Update the sentence 'stay strictly within … 16384 output tokens and 5 total tool calls' to match the 10-tool-call training cap in Table 9 / Section 5.1.
• List all paid API keys in one setup section. Add OPENAI_API_KEY (needed by ResearchQA and HealthBench scoring) to the top-level README and remove 'free keys' framing where it does not apply (Serper). Addresses the documentation-gap bullet.

Suggested

• Version evaluation subsets and cache search returns. Publish the specific 1000-example HealthBench / 2Wiki / ResearchQA sample IDs and the 5% / 10% SFT subset seeds. Publish (or point to) a cached Serper/Jina snapshot for the headline evaluation runs to bound day-to-day search drift. Addresses the eval-subsets and search-drift bullets.
• Release baseline reproduction commands. For each Table 1 baseline (Tongyi DR, WebThinker, WebExplorer, ASearcher, Search-R1), publish the exact dr-agent-lib config + eval seeds used, so the aggregated 15.6-point lead over Tongyi DR is independently checkable. Addresses the 'baseline rows not reproducible' bullet.
• Emit a top-level DRB summary from scripts/evaluate.py. Aggregate RACE and FACT into a single deep_research_bench_eval_results.json so reproducers do not have to reassemble the headline DRB number by hand.
• Soften the abstract's proprietary comparison and switch '%' to 'points'. Rephrase 'matches or exceeds proprietary deep research agents (by 0.7% over OpenAI DR on average)' to reflect that DR Tulu-8B is competitive with proprietary agents on the four benchmarks where both are evaluated, and consistently use 'points' rather than '%' for absolute score differences (also '15.6% over Tongyi DR').
• Surface the Ai2 ScholarQA comparison. Add a sentence to Section 5.2 explicitly discussing SQAv2 (88.3 vs 87.7, and citation-precision/recall trade-offs from Table 11) and the categorisation of fixed-pipeline Ai2 ScholarQA as non-open, so the 'outperforms all open DR models' framing is grounded rather than reliant on category boundaries.
• Qualify or replace the CACNA1A exemplar and add per-question variance to GeneticDiseasesQA. The showcase response conflates GoF, haploinsufficiency, and dominant-negative and re-uses the same citation for opposing claims; also report bootstrap CIs over questions given $n=47$ and 3-10 trials. Ties to the GeneticDiseasesQA weakness.
• Move the tool-error 'robustness' finding out of the main narrative. Reframe as a suggestive observation (or add a controlled degradation study), addressing the anecdotal-robustness weakness.
• Justify the query-selection thresholds. Add a one-line rationale for the >3 vs >2 asymmetry between OpenScholar and SearchArena (or run a sensitivity ablation), addressing the prompt-filter-cutoffs weakness.
• Soften the 'far beyond infinite reasoning tokens' framing in the discussion section, or add the missing verifier-compute vs. verifier-information comparison at matched budget.
• Add a $K_{max}$/buffer-stability ablation. Report buffer churn across steps and sensitivity to $K_{max}$, addressing the buffer-pruning weakness.
• Fix Table 5 / references / prompt-figure hygiene. Reconcile 'SFR-EDR' with its expansion; deduplicate the two Mallen et al. entries; correct the mislabeled 'Trajectory Generation Prompt' box (Figure 17) that in fact contains a rubric-generation prompt.