How much science is verifiable? We replicated ICML 2026 oral papers, and only seven mostly held up.

As AI agents become capable of increasingly complex, long-horizon tasks, we have an exciting opportunity to rethink the research ecosystem. One possibility is to verify the science rather than only reviewing the narrative in a paper, a clear blind spot in human peer review. To explore this possibility, when ICML wrapped up, we used AI agents to review and replicate every oral paper to see what we can learn from actually verifying the science. We address three questions:

• What does it cost to produce a top machine learning paper?
• What can replication reveal that narrative-only reviewing misses?
• How well does our review system align with existing human reviews?

Background

To answer these questions, we looked at all 168 ICML 2026 oral papers, which are generally considered papers of the highest quality in CS conferences. Looking closer at the categories in Figure 1, we see that most of these papers include experiments. This means that to actually verify the papers, one needs to run the reported experiments and see if the papers are reproducible.

To our best knowledge, no existing peer review system does more than reviewing the manuscript alone. This is why we built SAI review, extending our open source systems including openaireview and veritas. Given a paper, it reads and extracts all the essential claims from the paper, fetches all provided resources, and replicates the paper with diligence. Then it compares the replication results with the original claims to verify the paper. SAI review also supports reviewing the paper without running them, just like what peer review provides.

We ran SAI review on all ICML 2026 oral papers, with 105 full replications.

Producing a full ICML oral paper costs $$$$

Let us start with the cost estimate. As expected, producing a top paper in a top CS conference is not cheap. With our replication runs, we recorded all of our model calls and resource usages, and performed estimates for cost to reproduce everything, including contents in appendices. Figure 2 shows that the median cost for an oral paper is about $8,900 to reproduce, 17 of the 105 cost more than $100,000 each, and the most expensive one comes to nearly $2.2M.

More details on the cost estimation: For the subset of claims that our own runs covered, we have the measured numbers directly. For everything else, we read the full manuscript including the appendix, and computed estimates based on existing runs.

To extract every experiment in the paper, we include the main results, every ablation, every hyperparameter sweep, and every run repeated across seeds, datasets, and model sizes. For hardware cost, some papers state this directly, and one of them reports 38,400 GPU hours on 80 A100 GPUs for a single pretraining stage alone. When a paper does not specify usages, we used our run logs and derived the hours from the model size and the reported hyperparameters. Specifically, We computed GPU costs using Google Cloud's public on-demand rates.

Our cost estimations are still conservative, since the reported costs only include the work done in the manuscript. However, research is rarely one-shot, and the cost for retrying different directions can easily make the numbers 2x or 3x. Additionally, we also left out salaries and all other infrastructure costs.

Papers are rarely replicable

So, does the cost lead to verifiable science? Figure 3 unpacks the 168 papers: 104 of them shipped code that we could actually run, and we completed full replications for 105. We selected the 105 based on an initial resource estimate, and then we started from the low-resource to extensive-resource papers. Among those 105, only 27 reproduced more than 40% of their claims that we attempted to replicate, and only 7 reproduced more than 80%.

So what exactly goes wrong when we run them? Figure 5 shows the common causes. The two largest are code that does not run as shipped, usually from missing files, underspecification, broken dependencies that our agents cannot fix, or producing different results from the paper. Note that four papers depend on models that have retired, which means those results can no longer be reproduced by anyone.

To be clear, a low replication score does not mean a paper is intentionally deceptive. It's possible that code goes stale, proprietary data cannot be made available, and a script that ran on the authors' machine may not run the same on other machines. However, the practical consequence is the same, i.e., the science in the paper cannot be validated.

So, how do SAI reviews compare to human reviews?

The final question we address is whether SAI review catches what human reviewers caught? For each of the 168 papers, we collected their peer reviews on OpenReview and compared them with SAI's reports. Figure 6 unpacks the comparison: 78% of the comments that two or more human reviewers agreed on are also covered by SAI review, indicating that SAI review can be a very useful tool for authors to get high-quality feedback and improve their work.

The difference is largest on code and reproducibility. There SAI raised 903 issues that no human reviewer mentioned, while human reviewers raised only 22 that SAI missed. Human reviewers still lead on novelty and positioning, which requires more experience and good taste in the field.

Two examples from our runs: One paper claims its method trains only 0.77% of the base model's parameters, which is one of its headline selling points. The checkpoint that it actually released trains 6.31%, about 8x more, and the 0.77% number only works out for an adapter one eighth the size of the one shipped. Another paper reports a reliability table scored against a judge model that appears nowhere in its released code, and the repo ships nothing that computes those numbers. Note that neither issue is visible from reading the manuscript alone.

Check out the whole list of results here.

Where we go from here

None of this is specific to ICML papers. The same risk applies whenever you replicate someone else's result and prepare your own work. This analysis suggests possible shifts in our knowledge infrastructure. Authors could use such tools to improve their work and publishers can use AI tools to verify the science beyond the story.

We are working on making this service generally available (paper reviewing is already available). What you get back is a referee report that puts the most serious issues first and gives a concrete suggestion for each. Inline comments point to the exact lines and numbers they refer to. Give it a try!

A note on scope. We report only aggregate numbers here. Our goal is to imagine what the research ecosystem could look like, not to single out any authors. SAI review is still in beta and it can make mistakes, so verify its findings against your own work.