Agentic reproductions and robustness checks

The next 70 minutes

Plan

StartA live run on a preprint of mine: we start it now.

AgentsHow coding agents work, and how to work with them.

CasesThree papers, four agents, one prompt each: what came back, and where the agents disagree.

LiveBack to the live run: what it found, and one choice tested.

LastWhat still needs a human, then discussion.

Try it out with your own or a relevant paper, or just watch.

Live · started now, back at the end

A live run on a preprint of mine

Not in any training data, and I know where it is fragile: osf.io/preprints/psyarxiv/jsbgu

Consider the preprint at https://osf.io/preprints/psyarxiv/jsbgu and its linked data and code. Reproduce all numbers reported in it, with one script that runs from a clean start. Then identify the 3–5 analysis choices that are most fragile, list them, and predict what changing each might do to the results. Do not run those alternatives yet.

It finds the data and code and reproduces the numbers while we talk.
It lists the fragile choices, with predictions.
At the end, I pick one, and it tests it.

Follow along

Try it yourself

lukaswallrich.github.io/
ai-metascience-hands-on

Pick your own paper or a relevant one with open data and code, or one of our three cases.

You need an agent you already use (Claude Code, Codex, OpenCode) and R or Python.

QR code for lukaswallrich.github.io/ai-metascience-hands-on

How an agent works

Getting AI to run in circles

your
goal ›

LLM

proposes the
next command

——› command "Rscript model.R"

‹—— result "coefficient table"

Your machine

runs it, sends
back the result

› final
answer

↻ repeat until the goal is met. The LLM only ever writes text; your machine does the doing.

Tools it can call

read part of a file search the web search your files edit a file run code ask you a question

The interface today

One window, several agents

T3 Code is a free, open-source desktop (and web) app from Ping.

One interface for coding agents such as Claude Code, Codex and OpenCode.
You sign in with your own accounts, so you pay the agent providers as usual.
Each run is a thread, with a diff viewer for the files it changed.

The agents work fine in a terminal or in their own apps. T3 Code is a convenience. t3.codes

Choosing

Which agent?

Claude Code

Anthropic's agent. Comes with a Claude subscription. Paid plans start at about $20/month.

Codex

OpenAI's agent. Comes with a ChatGPT subscription. Paid plans start at about $20/month.

OpenCode

Open-source agent. Works with many model providers (paid per use) or with local models. Today with DeepSeek Flash.

The models change every month. The habits on the next slide carry over.

Behind the prompt

Four habits, in this prompt

Context

The paper, with its linked data and code.

"Consider the preprint at [URL] and the linked supplementary data and code"

Grounding

One script that runs from a clean start.

"one script … produces every number you report"

Plan/questions

List the alternatives and predictions before running them.

"Before running anything, list 6–10 alternative choices"

Verify

Published vs. reproduced, with the source of each number, and a list of decisions.

"say where you got it"

Before you try it on your own data

Can I use AI with my data?

Easy

Published open data, like today's two cases.

With care

De-identified data you are allowed to share.

Never*

Identifiable participant data.

* without a data processing agreement (and ethics approval), e.g. an enterprise contract or an approved secure environment, or a fully local model.

Case A · small, agents finish in a few minutes

"Female hurricanes are deadlier than male hurricanes"

92 US hurricanes, 1950–2012. Storms with more feminine names killed more people.

The authors' explanation: people take a storm with a feminine name less seriously.

The data are open. use them as a worked example for specification curve analysis.

, PNAS, 111(24), 8782–8787.

Case B · larger, many analysts

Do referees give more red cards to dark-skin-toned players?

29 teams analysed the same data: 146,028 player–referee pairs.

0.89 – 2.93

range of odds ratios (median 1.31)

20 of 29

teams had a confidence interval that excluded 1

Too slow to run live. In last night's runs, the agents saw this spread only after finishing their own analysis.

, AMPPS, 1(3), 337–356. Numbers from the teams' estimates file on OSF.

Case C · recent: published January 2026

Does researchers' ideology shape their findings?

71 teams in a many-analysts study of immigration and support for social policy produced 1,253 estimates.

+0.085

pro- minus anti-immigration teams' estimates (p = .008)

+0.011

per point of team pro-immigration sentiment (p = .06, tested at the 10% level)

The agents get the authors' replication package as published, and are told not to search for comments on the paper.

, Science Advances, 12(1), eadz7173. Data from .

The prompts in short (full text on the page)

What each agent gets

The data with its documentation (for case C, the whole replication package), and the paper's reference and DOI. Then:

Cases A and C · hurricanes, ideology

Reproduce the main result, with one script that runs from a clean start.
Compare with the published numbers, and say where each came from.
List the decisions the paper left open.
Stress-test: predict, then run 6–10 alternatives.
Judge whether the claim holds up.

Case C adds: do not search for comments on the paper.

Case B · red cards

Answer the question your way, with one script and an odds ratio.
List the decisions the data left open.
Stress-test: predict, then run 6–10 alternatives.
Only then compare with the 29 teams.
Conclude.

Before we look at results

A caveat: famous papers are in the training data

Asked from memory: no files, no web

All three agents recalled Silberzahn's numbers: 29 teams, odds ratios 0.89–2.93, 20 significant.

All three knew the main critique of the hurricane paper: before 1979, every Atlantic hurricane had a female name. Given the data, all three then tested exactly that.

What to do

The prompt asks for the source of each published number.

For real checks, use material the model cannot have memorised: your own work, or recent work like case C. Asked from memory, no agent could give its estimates.

13:51 (4 min). We asked each agent, with no files and no web, what Jung et al. found, whether it held up, and what Silberzahn et al. reported. We checked the answers against the sources. Claude and Codex declined to give coefficients ("I will not give numbers that could be wrong"). All three named all four authors of the hurricane paper (the prompt said only "Jung et al."). DeepSeek also gave volume and pages: the volume (111) was right, the pages (10185–10190; actual 8782–8787) wrong. Before 1979 all Atlantic hurricanes had female names, so name gender is confounded with era; the case A robustness slide shows all three chose that check. Case C memory probe: Opus 5.5 recalled the study design (the many-analysts immigration data, direction of the claim) but no estimates and no critique; Codex (Sol) recalled only the rough idea; DeepSeek did not know the paper. So if the published numbers come from memory, "reproduced" can mean "recalled", and the robustness checks may follow a critique the agent has read. Other options: rerun the script yourself, rename the variables, or use synthetic data with a known effect.

Case A · results from last night's runs

Same coefficients, different headlines

	Published	Opus 5.5	Fable 5.1	Codex	DeepSeek
Name × damage	0.705	0.705	0.705	0.705	0.705
Its SE	0.184	0.184	0.150	0.184	–
Paper read from		PubMed Central	PubMed Central	a PDF on a cycling blog	PubMed Central
Deaths, feminine vs masculine name, severe storm	×2.8 high vs low damage split	×2.8 median split, as the paper	×5.1 damage +1 SD	×1.5 75th percentile	×2.8 median split, as the paper

Published

Opus 5.5

Fable 5.1

Codex

DeepSeek

Name × damage

0.705

Its SE

0.184

0.150

0.184

–

Paper read from

PubMed Central

a PDF on a cycling blog

PubMed Central

Deaths, feminine vs masculine name, severe storm

×2.8
high vs low damage split

×2.8
median split, as the paper

×5.1
damage +1 SD

×1.5
75th percentile

×2.8
median split, as the paper

The coefficients reproduce exactly. The paper does not say where "severe" starts, so each agent chose its own point.

Claude Opus 5.5 (high, 29 Sep) and Claude Fable 5.1 (Claude Code) · GPT-6 Sol, high effort (Codex) · DeepSeek Flash (OpenCode), 28 Sep.

Matching T3 threads: "A · hurricanes · …"; reports on the materials page. SEs: Claude used glm.nb, which treats the dispersion parameter as fixed (0.150); Codex's statsmodels fit estimates it jointly and matches the paper (0.184). Claude said the paper "probably used different software", which is right. All numbers were rerun from each agent's own script in a clean folder: they regenerate exactly. Codex found the paper through a copy on cyclingandchill.com: ask where your agent got the paper from. Published: 15.15 vs 41.84 predicted deaths (MFI 3 vs 9), from a model with damage split into high and low; the paper does not give the cut. DeepSeek reproduced that model with a median split: 17.85 vs 50.35 (×2.82). Claude: 10.09 vs 51.54 at damage +1 SD, pressure at the mean. Codex: 9.74 vs 14.69 at the 75th percentile of damage. DeepSeek's own robustness baseline used a different model (updated pressure; ×4.37 at the 90th percentile): see the next slide. Rerun this morning with Claude Opus 5.5 (high effort), T3 thread "A · hurricanes · Claude Opus 5.5", 13 min: reproduced every number it found in the main text, including SE 0.184 (it identified the observed-information SEs), found the documentation error on the pressure variable, reproduced 15.15 vs 41.84 with a median split (ties coded low), and flagged a likely sign typo in the paper's binary-gender model. Verdict: "These data do not support the headline claim as a robust finding."

Case A · results from last night's runs

Which robustness checks did they choose?

Does the effect survive…	Opus 5.5	Fable 5.1	Codex	DeepSeek
only storms from 1979 on	no	no	no	no
log damage	no	no	no	–
no interaction terms	no	no	–	no
binary name gender	yes	yes	yes	yes
dropping the deadliest storms	–	yes	yes	–
the other pressure variable	yes	yes	yes*	yes*

Does the effect survive…

Opus 5.5

Fable 5.1

Codex

DeepSeek

only storms from 1979 on

no

log damage

no

–

no interaction terms

no

–

no

binary name gender

yes

dropping the deadliest storms

–

yes

–

the other pressure variable

yes

yes*

Each ran 9–10 checks and concluded that the claim is fragile. All four chose the 1979 split, the best-known critique of the paper.

Each agent judged "survives" by its own yardstick. * Both followed an error in the data's documentation.

Before 1979 all Atlantic hurricanes had female names, so name gender is confounded with era. Each agent predicted before running which checks would matter; they predicted the 1979 split correctly. Is that insight, or memory of the critique? The data's documentation (from the DHARMa R package, which distributes the data) says ZMinPressure_A is scaled from the updated pressure values. It is scaled from the original ones. The READMEs for last night's runs repeated this; today's README is the DHARMa text itself, so the live agents face the same error. Claude checked the data and noticed. Codex and DeepSeek trusted the README: Codex's "original pressure" check re-ran the published model; DeepSeek's "paper replication" baseline used the updated pressure, so its "original pressure" check (×5.18 at the 90th percentile) is the published model. Yardsticks: Claude, CI of the femininity effect at damage +1 SD; DeepSeek, CI of the death ratio at the 90th percentile; Codex, interaction p < .05. Codex's own summary: "only 1 of 10 high-damage ratio intervals exclude 1".

Case B · results from last night's runs

Three analysts, three answers?

	Claude	Codex	DeepSeek
Model	mixed model, player & referee random effects	logistic, clustered SEs	logistic, clustered SEs
Outcome	red cards	red cards + second yellows	red cards
Skin-tone contrast	very light → very dark	0.25 → 0.75	very light → very dark
Odds ratio [95% CI]	1.38 [1.12, 1.70]	1.12 [1.03, 1.22]	1.34 [1.10, 1.62]
Time to finish	about 1 hour	8.5 min	51 min

Claude

Codex

DeepSeek

Model

mixed model, player & referee random effects

logistic, clustered SEs

Outcome

red cards

red cards + second yellows

red cards

Skin-tone contrast

very light → very dark

0.25 → 0.75

very light → very dark

Odds ratio [95% CI]

1.38 [1.12, 1.70]

1.12 [1.03, 1.22]

1.34 [1.10, 1.62]

Time to finish

about 1 hour

8.5 min

51 min

Codex's estimate covers half the scale. Over the full scale it is about 1.26, close to the others.

Case B is not run live. Last night Claude's R script alone took 18 minutes and DeepSeek took 51 minutes in total. Codex rescaled: 1.121 squared = 1.26 (CI 1.06–1.50). Codex's unadjusted all-dismissals estimate (1.115 per half scale, about 1.24 full scale) matches DeepSeek's all-dismissals estimate of 1.24. Claude's headless run backgrounded its R job and ended its turn; it was resumed once with "run it in the foreground". In T3 Code, Claude is woken when a background job finishes (the session that prepared these slides ran in T3 Code and was woken this way several times). All three agree once outcome and scale are lined up: a modest association that is robust across their own checks. Rerun this morning with Claude Opus 5.5 (high), T3 thread "B · red cards · Claude Opus 5.5", 37 min, no nudge needed: OR 1.38 [1.12, 1.70], same model as Claude Fable; it explained that its binary split (OR 1.20) is a narrower contrast (expected 1.38^0.7 ≈ 1.25).

Case B · results from last night's runs

Where the agents fall among the 29 teams

Odds ratios with 95% confidence intervals for the 29 teams in grey and the three agents in red, sorted by estimate. Claude and DeepSeek sit just above the teams' median of 1.31; Codex's half-scale estimate sits low, and rescaled to the full scale it is near 1.26.

Codex placed its 1.12 below 24 of the 29 teams, while noting that their contrasts differ.

Case C · results from this morning's runs

Four agents, the same replication package

	Opus 5.5	GPT-6 Astra	GPT-6 Sol	DeepSeek Flash
Reproduced Table 1	all 9 columns	all 9 columns	all 9 columns	all 9 columns
Noticed 4 discipline codes with one team each	yes: "the paper does not discuss this"	yes, as a technical nuisance	no	no
Verdict	"weaker than Table 1 suggests"	"depends on defensible analysis choices"	"depends on reasonable analysis choices"	"holds up"

Opus 5.5

GPT-6 Astra

GPT-6 Sol

DeepSeek Flash

Reproduced Table 1

all 9 columns

Noticed 4 discipline codes with one team each

yes: "the paper does not discuss this"

yes, as a technical nuisance

no

Verdict

"weaker than Table 1 suggests"

"depends on defensible analysis choices"

"depends on reasonable analysis choices"

"holds up"

All four chose the pro–anti contrast (p = .008) as the main result. In every run, removing the discipline controls (alone or with all others) made it non-significant.

Claude Code, Codex (both on high effort) and OpenCode, 6.5–11 minutes each.

Matching T3 threads: "C · ideology · …". Opus 5.5: "Four of the 13 mdegree cells hold one team each: team 33 (pro), teams 46 and 47 (anti) and team 59 (moderate). Their fixed effect absorbs them completely, so the anti-immigration coefficient is identified from 7 of the 9 anti teams. The paper does not discuss this." Astra: found the same "4 singleton discipline cells with unit leverage", but only because they made one standard-error method undefined; it excluded them for that one fit. DeepSeek read the dependence on discipline controls as support: "consistent with the paper's argument that discipline composition is the mechanism's load-bearing control". Opus and Astra also found a model with a missing z statistic that Stata counts as significant.

Case C · what the replicators found

Two agents saw the problem. None ran the check.

Four discipline codes each hold one team. Their fixed effects absorb 80 estimates that go against the hypothesis .
Merge those four codes, and the per-point effect falls from 0.011 (p = .06) to 0.001 (p = .94). The agents' pro–anti contrast falls from 0.085 (p = .008) to 0.044 (p = .23).
Dropping the weights, or using the original study's standardised estimates, also removes the per-point effect.
The authors reply that their discipline coding is a theory-driven choice .

Auspurg & Brüderl, I4R Discussion Paper 303 (June 2026): "Due to idiosyncratic variable coding, 80 data points from four research teams that contradict the ideology hypothesis were effectively excluded from the analysis. Correcting this error renders the ideology effect statistically insignificant." Their main text tests the per-point effect (Table 1, column 1), which they call B&B's preferred specification, and argues that extreme-group contrasts lose information and inflate effects. Their supplement (Section E, Table A3) reruns all nine Table 1 models with their three changes: the pro–anti contrast falls to 0.012 (p = .29), and no model stays significant even at 10%. The paper itself presents columns 1–3 as three alternative specifications and highlights the 0.085 contrast in the text, so the agents' choice of main result is defensible. The single-change numbers on the slide (merge only: pro–anti 0.085 → 0.044, p = .23; per-point 0.011 → 0.0006) are my own calculation. Borjas & Breznau's reply (DP 304) also says researcher-level models without the single-team cells stay significant in most specifications. The teaching point: seeing an oddity and testing whether it matters are different steps. The prompt asked for alternatives chosen in advance; none of the agents went back after noticing the cells.

All three cases

What each agent decided without being asked

Trusting the documentation. The data's documentation describes one variable wrongly. Claude checked the data and noticed; Codex and DeepSeek built on the error.
Where "severe" starts. A median split, 1 SD above average, or the 75th percentile of damage: death ratios from ×1.5 to ×5.1 from one model.
What counts as a red card. Codex added second yellows; the others did not.
How far apart to compare. DeepSeek concluded that a dark/light split "removes the association". Codex ran the same split and flagged that it measures a different contrast.
What the main result is. In case C, all four agents chose the contrast with p = .008 over the per-point effect with p = .06.

Live · back to the preprint

What did the live run find?

Which numbers did it reproduce, and which not?
Which fragile choices did it list, and what did it predict?
I pick one choice. It tests it, and we compare with its prediction.

What still needs a human

Deciding what counts as the "main result" before the agent does.
Checking where each published number came from: the paper, or memory.
Judging whether an alternative analysis is defensible, or only different.
Following up an oddity the agent reports: does it change the result?
Rerunning the script and reading the code, especially the data cleaning.

At this conference · all times CEST

Later today and tomorrow

Tue 14:40Predicting replicability

Tue 16:05Machines that build and test the replication record (incl. ReplicatorBench; at 17:15: "Can an agentic LLM run computational reproducibility checks at human quality?")

Wed 15:00Does the finding survive another analyst? (incl. PaperRobust and "Multiverse Analysis in the Age of Large Language Models")

Getting our hands dirty

Plan

A live run on a preprint of mine

Try it yourself

Working with coding agents

Getting AI to run in circles

One window, several agents

Which agent?

Four habits, in this prompt

Context

Grounding

Plan/questions

Verify

Can I use AI with my data?

"Female hurricanes are deadlier than male hurricanes"

Do referees give more red cards to dark-skin-toned players?

Does researchers' ideology shape their findings?

What each agent gets

Four agents, same prompt

A caveat: famous papers are in the training data

What came back

Same coefficients, different headlines

Which robustness checks did they choose?

Three analysts, three answers?

Where the agents fall among the 29 teams

Four agents, the same replication package

Two agents saw the problem. None ran the check.

What each agent decided without being asked

What did the live run find?

What still needs a human

Later today and tomorrow

Discussion

References