FORRT AI in Metascience · hands-on session
What the agents produced
Every coding agent got the same prompt for each case. Here are their reports, with the scripts and figures behind them. Open two reports for the same case side by side, and look for where they disagree.
The agents ran unattended on 28–29 September 2026, from the prompts on the session page. Afterwards, each agent turned its own report.md into the HTML report linked here. Each folder also holds the agent's scripts and outputs. For the runs from 28 September (cases A and B with Claude Fable, Codex and DeepSeek), rerunning the agent's own script reproduced every number in its Markdown report. The strengths and weaknesses are my reading of each report.
Case A · "Female hurricanes are deadlier than male hurricanes"
Jung et al. (2014), PNAS. Task: reproduce the main result, then test how robust it is. The data's documentation (from the DHARMa R package) describes one variable wrongly: ZMinPressure_A is scaled from the original pressure values, not the updated ones.
| Agent | Time | Strengths | Weaknesses | Files |
|---|---|---|---|---|
| Claude Opus 5.5 high effort | 13 min | Reproduced every number it found in the main text, including the published standard errors, and explained why R's default gives smaller ones. Found the documentation error. Reproduced the paper's 15 vs 42 predicted deaths. Flagged a likely sign typo in the paper. | The slowest of the four. Its headline ratio (×5.2) uses a different damage level from the paper's (×2.8); it explains the difference. | report · files |
| Claude Fable 5.1 | 2.5 min | Checked the data and found the documentation error. A clear table of nine robustness checks. | Standard errors smaller than published (0.150 vs 0.184); it guessed, correctly, that the paper used different software. Did not reproduce the predicted deaths. | report · files |
| Codex GPT-6 Sol high effort | 6 min | Matched the published standard errors exactly. Its script writes the whole report, so every number comes from code. | Read the paper from a PDF on a cycling blog. Trusted the wrong documentation, so its "original pressure" check re-ran the published model. | report · files |
| DeepSeek Flash via OpenCode | 4 min | Reproduced the paper's predicted-deaths model with a median split (×2.82 vs published ×2.76). | Followed the wrong documentation, so the baseline for its robustness checks is not the published model. | report · files |
All four concluded that the claim does not hold up. All four chose the check that uses only storms from 1979 on, when male names came in, which is the best-known critique of the paper.
Case B · Do referees give more red cards to dark-skin-toned players?
Silberzahn et al. (2018), AMPPS. Task: answer the question your own way, test robustness, then compare with the 29 teams (median odds ratio 1.31, range 0.89–2.93).
| Agent | Time | Strengths | Weaknesses | Files |
|---|---|---|---|---|
| Claude Opus 5.5 high effort | 37 min | Odds ratio 1.38 [1.12, 1.70] from a mixed model with player and referee random effects. Explained, with numbers, why its dark/light split gives a smaller estimate. A careful comparison with the teams that used similar models. | Slow: the mixed models take minutes each. States the paper's numbers from memory, labels them as such, and checks them against the teams' file. | report · files |
| Claude Fable 5.1 | about 1 h | The same model and the same estimate (1.38). Flagged its dark/light split as a narrower contrast. Said openly that it had not rerun its script after a final cosmetic edit. | Running unattended, it started its long R job in the background and stopped; it needed one nudge to finish. | report · files |
| Codex GPT-6 Sol high effort | 8.5 min | Fast. Ten checks, all consistent. Its script writes the whole report. | Reported the effect for half the skin-tone scale (0.25 → 0.75), then placed its 1.12 below 24 of the 29 teams, although it noted that their contrasts differ. Over the full scale its estimate is about 1.26. It counted second yellows as red cards. | report · files |
| DeepSeek Flash via OpenCode | 51 min | 1.34 [1.10, 1.62] over the full scale, with ten checks. | Concluded that a dark/light split "removes the association", without noting that the split is a narrower contrast. Its script only runs in its original folder. | report · files |
Once the outcome and the skin-tone contrast are lined up, the four agents agree: a modest association that holds across their own checks, near the middle of the 29 teams.
Case C · Does researchers' ideology shape their findings?
Borjas & Breznau (2026), Science Advances. Published in January 2026, so the models do not know it well. The agents got the authors' replication package and were told not to search for comments on the paper.
| Agent | Time | Strengths | Weaknesses | Files |
|---|---|---|---|---|
| Claude Opus 5.5 high effort | 11 min | Reproduced Table 1 exactly. Noticed that four discipline codes each contain a single team, and that the paper does not discuss this. Found a model whose missing test statistic is counted as significant. Ran a grid of 24 specifications. | Did not test what happens when those four codes are merged. | report · files |
| Codex GPT-6 Astra high effort | 10.5 min | Saved the article and parsed the published numbers from it, with built-in checks against the authors' log. Also found the missing test statistic. | Found the same four single-team codes, but treated them only as a technical problem for one type of standard error. | report · files |
| Codex GPT-6 Sol high effort | 8 min | Reproduced all nine columns of Table 1 quickly. Found that dropping the discipline controls makes the result non-significant. | Did not look into why the discipline controls matter. | report · files |
| DeepSeek Flash via OpenCode | 6.5 min | An exact reproduction, including R² values and a mismatch between the paper's significance stars and its p values. | Concluded that the claim "holds up". It read the result's dependence on the discipline controls as support for the paper. | report · files |
What the replicators found (spoiler: read after looking at the reports)
- A replication report from the Institute for Replication (Auspurg & Brüderl, 2026, I4R DP 303) points to the four single-team discipline codes: they absorb 80 estimates from four teams that go against the hypothesis.
- Merge those four codes, and the per-point ideology effect falls from 0.011 (p = .06) to 0.001 (p = .94). The pro–anti contrast that all four agents chose as the main result falls from 0.085 (p = .008) to 0.044 (p = .23) (my calculation).
- Dropping the regression weights, or using the original study's standardised estimates, also removes the per-point effect.
- The authors reply that their discipline coding is a theory-driven choice (Borjas & Breznau, 2026, I4R DP 304).
Two agents saw the single-team codes. None tested whether they change the result.
Lukas Wallrich, Birkbeck, University of London. Session page · Slides