Can we reproduce the numbers a published paper reports, from the data it shares?
Each cycle we take published research with an open replication package and recompute its reported results. We publish what reproduced, what did not, and what we could not test. A cycle where everything reproduces is not a test: it is a shop window.
Tasks come from REPRO-Bench, an open benchmark of 112 social-science reproduction tasks (ACL 2025 Findings). Each task ships the paper PDF, a list of findings to check, and the replication package.
We recompute from the package only, in Python, and compare against the published values transcribed by hand from the paper, not read out of it by a model. Published rounding is accepted (±0.005). Confidence intervals are recomputed, not paraphrased. The benchmark's own labels are not consulted: the verdicts below are ours.
| Paper | Findings checked | Verdict |
|---|---|---|
| Predatory publishing in Scopus Macháček & Srholec (2022) · Quantitative Science Studies · doi:10.1162/qss_a_00213 |
Tables 2, 3, 4 of 5 listed findings |
Reproduced 158 of 168 published values exact (94%) |
| Environmental impact labels and food purchases Potter et al. (2022) · PLOS ONE · doi:10.1371/journal.pone.0272800 (retracted 22 Aug 2024) |
Primary outcome + Tables 1, 2 Study 1 only |
Reproduced with one basal deviation we could not resolve |
Predatory publishing (Table 2). 40 of 40 published values exact: top and bottom 20 countries by share of articles in suspected predatory journals. Kazakhstan 17.00, Indonesia 16.73, Iraq 12.94 … Turkmenistan 0.00.
Predatory publishing (Table 4). 78 of 80 published values exact across the four fields of research; all four "all countries" averages exact (1.98 · 3.39 · 1.96 · 3.99).
Label trial, Study 1 (primary outcome). Control mean 61.86 against a published 61.9. Label effects −3.94 (Petal), −3.85 (A–E) and −3.19 (Combined) against published −3.9, −3.9 and −3.2. All three published confidence intervals overlap ours.
Label trial, Study 1 (Table 2). All four environmental indicators land inside the published ranges: greenhouse gases −15.2 to −14.0%, biodiversity −15.2 to −12.2%, eutrophication −14.2 to −11.4%, water −25.7 to −19.2%.
Group definitions (Table 3, predatory publishing). 18 of 22 rows exact. Four deviate: "French spoken" 2.42 vs 2.41; "other language spoken" n=84 vs 86 (2.39 vs 2.42); "other natural resources" n=37 vs 39 (1.84 vs 1.77); "upper middle income" n=46 vs 44. The deviations are group boundaries, not arithmetic: the exact rules for language sets, resource-rent thresholds and income classification are not fully recoverable from the shared package.
Standard deviation of basket size (Study 1). Published 4.2 items; the shared data gives 2.49, because 88.5% of baskets contain exactly 10 items. Either the file is truncated or the published figure comes from another variable. We did not resolve it.
Not tested this cycle. The regression table and the marginal-effects figure of the predatory-publishing paper; Study 2 of the label trial (that study was invalidated and the article retracted, and its data is not in the package).
The label trial is retracted. We report that before the result, because it is the most important thing about the paper. One of our two verdicts carries an unexplained deviation. Our descriptive coverage is 3 of 5 findings in one paper and a single study in the other: the hard part of reproduction (the statistical models) is mostly still untested.
If a result is weak, it stays up.