FORRT AI in Metascience · Tuesday 29 September, 13:20 CEST
Getting our hands dirty: agentic reproductions and robustness checks
Give a coding agent a published analysis and its data. Ask it to reproduce the result, then to test how robust it is. Then look closely at what it hands back.
Slides · What the agents produced: all backup reports, compared
What you need to follow along
- A coding agent you can already use, for example Claude Code, Codex or OpenCode. A standard paid plan (around $20 a month) is enough for one case.
- R or Python on your machine, so the agent can run the code it writes.
- An empty folder for each case. Put the case files in it and start the agent from inside that folder.
Setting up an agent for the first time takes longer than this session allows. If you don't have one yet, watch the live runs and try a case afterwards.
Pick a case
Case A · small, agents finish in a few minutes
"Female hurricanes are deadlier than male hurricanes"
Jung et al. (2014), PNAS, doi:10.1073/pnas.1402786111. 92 US hurricanes, 1950–2012.
Download case A (data, README and prompt)
Case B · larger, slow: agents took 10–60 minutes in our tests
Do referees give more red cards to dark-skin-toned players?
Silberzahn et al. (2018), AMPPS, doi:10.1177/2515245917747646. 146,028 player–referee pairs; 29 teams analysed the same data.
Download case B (README and prompt) Then add the data from OSF: osf.io/download/fv8c3 (5 MB zip), unzipped into the same folder.
Case C · recent, agents finish in about 10 minutes
Does researchers' ideology shape their findings?
Borjas & Breznau (2026), Science Advances, doi:10.1126/sciadv.adz7173. 1,253 estimates from 71 teams in a many-analysts study of immigration and support for social policy.
Download case C (the authors' replication package, as published; 9 MB) Unzip it and paste the case C prompt below.
Your own paper, or a relevant one
A paper you know well, with open data
You know the right answer and where the analysis is fragile, so you can judge the agent best here. Use published or de-identified data only; don't give identifiable participant data to a commercial agent.
The prompts
Paste the prompt for your case into the agent, unchanged, so you can compare your results with ours. If you like, post your agent's main estimate in the Meet chat.
Case A: hurricanes
Case B: red cards
Case C: ideology and research findings
Your own paper, or a relevant one (fill in the brackets): the prompt from the live run
While it runs
- Watch the trace, not only the answer. Where did it find the published numbers?
- Note each choice it makes that the prompt did not ask for.
- When it finishes, rerun its script yourself. Do the numbers in report.md come out of the script?
Lukas Wallrich, Birkbeck, University of London. Materials: github.com/LukasWallrich/ai-metascience-hands-on. The hurricane data is the hurricanes dataset from the DHARMa R package (GPL-3). The red-card data is hosted on the project's OSF page.