A human hand and a machine hand over shared research files

Birkbeck Business School · Research seminar

Using AI (agents)
for research



AI for research so far

The slop is real

+42% submissions to Organization Science after ChatGPT

Heavily-AI papers: 3.2% reach revise-&-resubmit vs 11.9% for minimal-AI ones.

AI use correlates with worse readability, more jargon.

Driven by incentives (publish-or-perish, rankings), not bad faith.

Gartenberg et al. (2026)

Reconsidering AI for research

AI for research so far

And fabrications are reaching print

Hallucinated citations, references to papers that don't exist,
are turning up in peer-reviewed conference papers, and rising fast:

0.28% → 2.59%

of papers carry a fabricated citation in a single year (ACL venues, 2024→2025).

154 papers

at EMNLP 2025 (3.7%) cited work that isn't real, 100+ accepted as main papers.

They look plausible and slip past review — and this doesn't count misattribution.

Sakai et al. (2026)

An agent working in a loop over files

A different bet

AI agents can do better — when grounded & verified.

Instead of asking AI for an answer, you get it to do the work — against your own files, with every step on show.


HOW AN AGENT WORKS

Getting AI to run in circles

your
goal ›
LLM
proposes the
next command
——› command  "read draft.md"
‹—— result  "5 DOIs found"
Your machine
runs it, sends
back the result
› final
answer

↻ repeat until the goal is met — the LLM only ever writes text; your machine does the doing.

Tools it can call

read part of a file search the web search your files edit a file run code ask you a question

The loop, in one task

"Are the DOIs in my draft correct?"

your goalCheck that the DOIs in draft.md are correct.
read
part of fileopens draft.md's reference section only — finds 5 DOIs.
search
the webchecks each against Crossref — #1–#4 match; #5 points to a different paper.
ask you"DOI #5 doesn't match its title — replace it with the correct one?"
edit a filewrites the corrected DOI into draft.md.
✓ done4 correct, 1 fixed. Not from LLM "memory" but a reproducible source.



Adoption is taking off

81% have tried AI chatbots for research, but only 20% use coding agents regularly.

More output early in the pipeline (~75% more working papers), not in journal submissions yet.

Adoption is uneven: economists 39% vs public health 6%; male-named researchers adopt at 2× the rate of female-named.

Researchers are more optimistic about their own productivity than about the field.

Survey of 1,260 quantitative social scientists, self-selected for AI interest. Lyttelton et al. (2026)

Reconsidering AI for research
A centaur contemplating a chess position

The bet

A competent researcher + verified AI contributions can beat either alone


In "centaur" chess, human + machine teams used to beat strong grandmasters
and engines playing on their own.

SO FAR WE ARE OK

On its own, agentic AI still lags competent researchers

Elo ratings: AER 1817, AEJ Policy 1717, APE (AI-generated) 1264

In head-to-head tournaments of economics papers, AI-generated papers (APE) score well below journal-level work — a 450–550 Elo gap behind AER and AEJ: Policy.


Project APE — Autonomous Policy Evaluation (non-peer-reviewed automated benchmark),
Social Catalyst Lab, Univ. of Zurich. ape.socialcatalystlab.org (Jun 2026).


Four habits for using it well

Context

Tell it what you actually want: the goal, the constraints, the audience.

Grounding

Point it at your own sources, not its memory.

Plan/questions

Ask for the plan, or its questions, before you ask for the output.

Verify

Check what it hands back - ideally both manually and automatically.

These apply to any agent you use: Codex (OpenAI) and Claude Code (Anthropic) are in the lead.

Reconsidering AI for research
Three tasks launched in parallel

Let's GET THIS StARTED

Three demos

  1. Data extraction: power-analysis reporting across recent management papers.
  2. Data analysis: a pre-registration + dataset turned into a clean, reproducible report.
  3. Teaching material creation: build a seminar resource site, and fact-check it.

Then we talk while they run.

Demo 1 · the prompt

Find & extract: power-analysis reporting

  • Goal: how recent management research justifies the effect sizes it powers its studies for.
  • Ground: I provide 2026 articles for 3 ABS 4 journals; agent reads the full text.
  • Extract per paper: type, assumed effect size, how it was justified, α, power, N, software, each with a verbatim source quote.
  • Prepare to verify: "Quote the source for every value; write 'not reported' rather than guessing."

Demo 2 · the prompt

Pre-registration + data → a reproducible report

  • Ground: a pre-registration, data + codebook (ALLBUS 2016)
  • Task: reproduce the pre-registered analysis as a clean Quarto report that renders end to end.
  • Plan: ask me any questions, and document areas to check.
  • Verify: ensure all output is generated by a single commented code file.

Demo 3 · the prompts

Build a resource site for today, then fact-check it

Builder

Build the site from these slides and my email course; follow its style guide and match an existing site's structure.
Every claim cites a real, verified source.

Checker — a fresh agent

Check every claim (file:line) and every DOI; flag invented or stale AI-tool claims, or areas that lack clarity.
Report only, don't fix.

The FOUR habits

Made concrete

  • Context & grounding: full-text retrieval for Demo 1, pre-registration that constrains Demo 2, personal style guide for Demo 3
  • Plan: under-emphasized here, can be developed with AI in chat.
  • Verify: a validated source quote per cell (Demo 1); the agent's results checked against a human analysis (Demo 2); an independent AI fact-checker (Demo 3).


Reconsidering AI for research

Grounding & division of labour

Ground it in data and deterministic code


The LLM

Reads your sources, extracts and narrowly annotates, drafts the analysis code.

Deterministic code

Runs the numbers reproducibly: tests and outputs you can inspect and re-run.

Demo 1 quotes its source for every value;
Demo 2's code can be validated by a researcher (or with simulated test data).

Reconsidering AI for research

Review the outputs

Let's see what they did — and how

Look at the output, the agents response, and the trace.

  • Demo 2: the "knitted" report. Ask for comparison to the paper.
  • Demo 1: the extracted table. Ask for deterministic validation.
  • Demo 3: the site. Ask for an independent fact-check.


Reconsidering AI for research

Holding on to our own thinking

Cognitive offloading,
scaffolding and
ownership


A genie rising from a lamp — a power you summon but can't simply put back
Reconsidering AI for research


Can I even use AI with my data?

Easy

Published (OA) papers, public datasets: Demos 1 & 3.

With care

Publishable / de-identified data, especially used agentically: ALLBUS in Demo 2.

Never*

Identifiable participant data; (others' sensitive unpublished work).

* without a data processing agreement (and ethics approval), e.g. an enterprise contract or an approved secure environment.

Reconsidering AI for research

Keeping up

The durability problem

Specific claims about these tools go stale fast — the best model this month
isn't the one from three months ago (Sonnet 4.5 → Opus 4.8 → Mythos/Fable …).

The durable thing is the mental model: context, grounding,
plan, verify. Within that, can experiment with pushing the boundaries.

Reconsidering AI for research

Demo 4 · "peer" review

A review agent that dives deep

coarse.ink is an open source review agent. Can use it online, or in your own agent. Long-running background skill (15–25 min/run).

Let's look at the report it produced on a recent preprint of mine.
Earlier reviews caught important issues that we addressed.


Reconsidering AI for research

Getting started

How to get started

Step 1 · Pick an agent

Claude Code, Codex (ChatGPT), or OpenCode (open models, cheaper & rougher).

Step 2 · Subscribe

A paid plan at ~£20/month gets you started, ChatGPT likely best value.

Step 3 · Experiment

Boring file management tasks, verifiable analysis code, second-coding, self-review, webpages ...

Choose deliberately

Decide where you want to learn something and do it your way,
and where you just want to get things done (deskilling vs. outsourcing).

Reconsidering AI for research

Lower-stakes starting points

Start by building comfort with safer tasks

  • Like Demo 1: rename files consistently; build a codebook prefilled with your file list and Crossref metadata.
  • Like Demo 2: run exploratory analyses on a dataset, or turn an existing analysis (e.g. from SPSS) into one reproducible report.
  • Like Demo 3: turn your own content, verbatim, into a clean website, and get help deciding where to host it.

Low stakes, easy to verify ... a good place to build the habits.

Going further

If you want to go deeper

The resource site Demo 3 just built was made from my GenAI email course.
Just about to launch, more thorough than this session (and less focused on agents).


Reconsidering AI for research
Two speech bubbles in conversation

Over to you

Questions & comments

Slides made with Claude Code · illustrations generated with gpt‑image‑2, via Claude Code.

Reconsidering AI for research

References

References