Does It Reproduce?Project concluded
Archive · concluded October 2026

Does It Reproduce?
The project has concluded.

Between June and August 2026, an autonomous agent fleet re-ran the analyses behind 1,286 published computational papers from their deposited data and code, and scored how far each result reproduced. The live service has been shut down. This page preserves the results; the complete dataset is free to download.

This project was designed to showcase the ability of autonomous agents in the framework of Brainbox, enabling them to independently reproduce scientific datasets - from download to computing.

1,286papers assessed
1,187scored (0-100)
648reproduced (score ≥ 75)
73.9mean score · median 76

Score distribution

1,187 scored papers, in classes of ten points. The 70-79 class straddles the 75 threshold.

0-9: 0 papers0010-19: 6 papers61020-29: 13 papers132030-39: 27 papers273040-49: 50 papers504050-59: 160 papers1605060-69: 182 papers1826070-79: 235 papers2357080-89: 268 papers2688090-100: 246 papers24690

Outcome

Share of scored papers per band.

≥ 75 reproduced · 648 (55 %) 50-74 partial · 443 (37 %) < 50 discrepant · 96 (8 %)
Nothing to recompute (wet-lab only)49
Data not accessible (restricted / unavailable)34
Papers with a public data accession1,208
Papers naming a code repository1,209
Journals covered286
Compute jobs · CPU hours1,607 · 5,437

By publication year

Scored papers per year · mean score (right).

≤20121978.9
20133363.9
20144764.6
20151377.0
20161872.5
20172078.3
20182975.2
20195873.7
20207776.1
202118775.0
202221973.7
202313676.1
20249974.2
202515974.4
20267372.0

Most frequent journals

Journals with at least 20 scored papers.

JournalPapersMean
Nat Commun9180.0
PLoS One5069.4
Sci Rep4470.2
Nucleic Acids Res4273.6
iScience3077.5
Elife2976.1
Genet Sel Evol2977.0
Front Immunol2967.7
BMC Genomics2675.3
BMC Bioinformatics2676.3
Genome Biol2473.0
Front Microbiol2476.8
F1000Res2169.2
Bioinformatics2174.1

How it worked

  1. An agent read each paper, fetched its deposited data and code, and re-ran the analysis autonomously on the Brainbox compute.
  2. Every reported claim was graded against the re-run; the grades rolled up into one 0-100 reproducibility score.
  3. Each record carries a level: L1 (AI only) → L2 (a scientist plausibility-checked the grades) → L3/L4 (a scientist re-ran it).
  4. Every verdict was version-sealed and could be contested. Bands: ≥ 75 reproduced · 50-74 partial · < 50 discrepant.

Not an accusation

A failed reproduction is not an integrity finding and not peer review. It measures what could be recomputed from public data at the time. Failures most often come from missing files, undocumented steps or access restrictions, not from the authors. The records below are therefore listed alphabetically, never ranked.

All 1,286 records

Search by title, journal, DOI or PMID. Each record keeps its original summary.

PaperYearScoreLinks

Download the data

All 1,286 records with scores, verdicts, data accessions and summaries. Free for non-commercial scientific use with attribution (“Does It Reproduce?, Schlein Lab, UKE, 2026”). Bibliographic metadata comes from open sources (PubMed, OpenAlex) under their respective licences.

records.csv · 2.2 MB records.json · 2.5 MB

Questions or corrections: c.schlein@uke.de