Two reasons replications don't fix science (yet)

A study you probably know

"Students who believe intelligence can grow achieve more over time."

Growth mindset, taught in schools, in textbooks, on CPD days. (Blackwell et al., 2007)

Why this should bother us

Replication is how science corrects itself, and findings often don't hold.

The argument FOR TODAY

Replication can only fix science if replications are both done and found.

A vicious cycle? nothing to find → it feels pointless → fewer get done → even less to find.

Why finding them is hard

Even the replications that exist hide from you.

So "has this been replicated — and how did it turn out?" has been almost impossible to answer at the point you need it.

A foundation for discovery

FLoRA: original → replication links

• FORRT Library of Replication (and Reproduction) Attempts:
  ~2,200 original–replication pairs and growing.
• Community-built and continuously updated, with some AI support.
• Each entry links an original study to the replication that tested it, with the outcome.

FReD (the FORRT Replication Database) is the subset with fully coded effect sizes.

My Current FOCUS

"Making Replications Count"

UKRI Metascience grant (2025), with Josefina Weinerova & Lukas Röseler.

Goal: surface replications throughout the research cycle — not in a separate database you have to remember to check, but where you already work.

The toolset

Replications, delivered to where you work

Zotero plug-in

Surfaces replications inside your reference manager.

Metacheck

Flags replications during peer review.

Browser plug-in (WIP)

Augments Google Scholar & publisher pages with details on replications (and more).

FLoRA-Notify

A bot that flags relevant replications on new preprints (currently running as an RCT).

Why so few get done

Not a motivation problem, an incentive problem.

• Novelty is rewarded; verification isn't.
• No obvious venue; fears about citations and reputation.
• And researchers' own curiosity pulls toward new questions.

Result: <0.1% of articles in top-50 economics journals ; ~1.5% of top-100 psychology articles.

However: who has started an analysis by first checking the original effect still shows up, before testing something new? More than 1.5% of the time? Hold that thought.

The rest of the talk

Two routes to more replication

Route 1 · the problem

"No room at the inn" for replications

• The Pottery Barn rule — a journal should publish replications of what it published — has almost no uptake (RSOS the notable exception).
• Publication is concentrated: 4 economics journals ≈ ⅓ of econ replications;
  6 psychology journals ≈ 60% of direct replications (2010–2021). (Reed et al., 2025)
• And staying a preprint roughly halves your citations vs. a journal article.

Route 1

A different vision of publishing

• Scholar-owned: public constitution, participatory governance, no commercial control, a large team.
• Diamond OA — a commitment to openness and independence. Going against the open-access industry: ~$8.97B in APCs to six publishers (2019–2023). (Haustein et al., 2024)
• A credible commitment: pledged to run at least to early 2028, then evaluated against pre-defined, public criteria.

Route 1 · take part

Submit, or review for R2.

Route 1 · practical aside

Want to replicate? Four lenses for a good target

Full version: How to find a replication target.

Route 2

Not every replication should be a standalone paper.

Many of us want to ask new questions — and a stand-alone replication can be a hard sell against career incentives and our own curiosity.

So there's a second route that, done right, produces replications anyway.

The reframe

Replicate and extend

Replication as a prospective design tool, not a retrospective audit.

• First test whether a prior effect / method / paradigm / measure holds…
• …then build on that base to probe a new boundary condition, mechanism, or outcome.

We often do this anyway. We just rarely report the first step as a replication. Worth changing, to make science more cumulative (and disciplined).

Why?

It embeds replication in "normal" research

• It rides the incentive most researchers already have (novelty) — so the replication actually gets done …
• … and reaches a normal journal as part of a novel paper.

One concrete way to deliver our broader ambition to mainstream replications.

Doing cumulative research right

Keep the replication interpretable

"If I change the population, the measure, and the context all at once, a null can't tell me whether the original was false or my extension changed the conditions ... and my contribution to cumulative knowledge is unclear even if I succeed."

• Keep the replication arm close enough to update confidence in the original; preregister the replication target.
• Strive to reduce publication bias — purely "conceptual" replications, selectively published, can keep a zombie field alive (cf. social priming).

Haven't I heard that before?

Not an entirely new idea, but closer to motivations?

• Builds on existing taxonomy: constructive replication and the Systematic Replications Framework.
• But those still rely on researchers volunteering to do the "more boring" first steps.
• Normalising replication as the foundation within a paper may be more realistic, it aligns with what people already want to do.
• The flip side: sometimes the extension is just a vehicle to publish a replication you wanted for its own sake ... then maybe drop the extension and send it to R2?

(I'm writing this up for Social Psychology & Personality Compass,
so tell me where it breaks.)

Pulling it together

Two routes, one goal

Both need reviewers, standards, and discoverability. (Klein et al., 2018)

Ways to take part

Three ways in

References