Ideological bias in the production of research findings: reproducing and stress-testing Borjas & Breznau (2026)

How to rerun. From this folder: Rscript analysis.R. The script reads only data/df.dta. It writes the tables below (output/*.md, output/*.csv), the figure (output/robustness.png) and a console log (output_log.txt when run as Rscript analysis.R > output_log.txt 2>&1). Run time is about 10 seconds. The seed is fixed. Packages: haven, dplyr, sandwich, clubSandwich, lme4, ggplot2, patchwork.

1. Main result and reproduction

Headline claim (abstract). "Teams composed of pro-immigration researchers estimated more positive impacts of immigration on public support for social programs, while anti-immigration teams estimated more negative impacts."

Main number. Table 1, column 3: in a regression of each model's average marginal effect (AME) on team type, pro-immigration teams report AMEs 0.085 (SE 0.031, p = 0.008) higher than anti-immigration teams. The regression is at model level (1,253 models, 71 teams), weighted by 1/(models per team), with controls for statistical skill, topic experience, team size and a 13-category discipline-composition code (mdegree), and SEs clustered by team.

Result. All nine columns of Table 1 reproduce exactly. The largest absolute difference from the Stata log is 5 × 10⁻⁸ for coefficients and for SEs. The script also rebuilds the team groups (group1, group3, p12, p56) and the tail outcomes (neg10s, pos10s) from their components and confirms that they equal the stored variables.

2. Published and reproduced estimates

Sources of the published numbers:

• "Published est (SE)" and R² pub: Table 1 of the article, read from the Europe PMC full text opened in this session: https://www.ebi.ac.uk/europepmc/webservices/rest/PMC12757037/fullTextXML
• "Stata log est (SE)": code/Log Files/02_Main_Regs.log in this folder.
• Published p = 0.008 for the column 3 difference: article text, same Europe PMC URL ("statistically significant (with a P value of 0.008)"). Reproduced p = 0.008.
• No number is taken from memory.

Terms: proindex = team mean immigration sentiment (1–6); p12/p56 = share of team answering 1–2 (anti) / 5–6 (pro); group1 = anti-immigration team; group3 = pro-immigration team (reference = moderate). Outcomes: ame; neg10s = AME below the 10th percentile and significant; pos10s = AME above the 90th percentile and significant (linear probability models).

Two differences from the published table, both in the paper's presentation and not in the estimates:

• R², column 8. The log gives 0.0909; the paper prints 0.090 (rounding).
• Significance stars. The table note defines * as P < 0.1 and ** as P < 0.05. Five published estimates have P < 0.05 but carry one star: column 5 difference (p = 0.028), column 6 difference (p = 0.032), column 8 p12 (p = 0.025), column 8 difference (p = 0.015), column 9 difference (p = 0.015). The stars understate the significance.

3. Analysis decisions the paper did not settle

The reproduction starts from df.dta, so decisions already made in 01_Data_Prep.do are inherited. Items 1–6 are decisions I had to make; items 7–10 are inherited choices that the paper does not state and that I checked or kept.

1. Which result is "the main result". I took the pro-minus-anti difference in Table 1, column 3 (0.085), because the abstract states the claim in terms of pro- and anti-immigration teams and the text reports its p-value. Columns 1–2 (continuous measures) and 4–9 (tail outcomes) are reproduced but not used as the robustness target.
2. Sample. df.dta has 1,254 rows. One row has no team, no AME and no covariates; it comes from the merge with sem_p.csv. I dropped it, giving 1,253 models and 71 teams.
3. Type of clustered SE. The paper says only "clustered at the team level". I used CR1 with Stata's correction G/(G−1)·(N−1)/(N−K) and p-values from a t distribution with G − 1 = 70 df. This matches the log to 7 decimals.
4. Weight type in the tail regressions. The Stata code uses analytic weights for the AME columns and importance weights (iw) for the tail columns. Both give the same coefficients. With sandwich::vcovCL and ordinary weights I reproduce the iw SEs exactly, so the weight type does not matter here.
5. Missing test statistic. One model of team 75 has AME −0.073 and a missing z. Stata treats a missing value as larger than any number, so this model counts as "significant" in neg10s. I kept this to match the published sample. The paper does not mention it.
6. Significance threshold for the tail outcomes. The Table 1 note says "P < 0.05"; the text says "significant at the 5% level in a one-tailed test (|t| > 1.645)"; the code uses z > 1.645. I followed the code. A two-sided 5% test would be |t| > 1.96.
7. Tail cut-offs. The 10th/90th percentile cut-offs (−0.071, 0.052) are unweighted percentiles over all 1,253 models, typed into the code as constants.
8. Teams in both extreme groups. Teams 61 and 64 have one anti member and a pro majority. The code classifies them as pro; the paper's definitions would put them in both groups and it does not mention the overlap.
9. Imputations and fixes in df.dta. Team 27's gender, topic experience, statistical skill and prior belief are hot-deck imputed; team 8's statistical skill is set by hand; team sizes of teams 93 and 94 are corrected. I did not change these.
10. Discipline fixed effects with single-team cells. Four of the 13 mdegree cells hold one team each: team 33 (pro), teams 46 and 47 (anti) and team 59 (moderate). Their fixed effect absorbs them completely, so the anti-immigration coefficient is identified from 7 of the 9 anti teams. The paper does not discuss this.

4. Robustness

4a. Robustness plan (written before any model was run)

Fixed research question. Do teams whose members are pro-immigration report different estimates of the effect of immigration on welfare support (AME) than teams with at least one anti-immigration member, given the teams' skills, size and disciplinary background?

Target quantity. Difference between pro-immigration and anti-immigration teams in the AME (Table 1, column 3; published value 0.085, SE 0.031). The published regression is weighted by 1/(models per team), has discipline-composition fixed effects (mdegree), and uses team-clustered SEs.

Facts about the data that shape the plan (from inspecting data/df.dta):

• The anti-immigration group has 9 teams. Two of them (teams 46 and 47) sit alone in their mdegree cell, so their fixed effect absorbs them. The anti coefficient is identified from 7 teams.
• Team 42 (anti) submitted 2 models with mean AME −0.258. With 1/nmodel weights it has the same weight as a 48-model team.
• 195 models use a scale outcome (scale == 1); their AMEs are not in probability units but are pooled with the binary-outcome AMEs.

Diagnostic (not one of the alternatives): drop each team in turn and re-estimate the baseline.

4b. Robustness results

Target: pro-minus-anti difference in AME. Each row changes one choice from the baseline. CIs use the t distribution with 70 df unless the note says otherwise (A2: residual df; A8: normal; A9a: Satterthwaite df). "Anti / pro teams" counts teams in the estimation sample.

Figure. A: each alternative choice on its own. B: the baseline re-estimated 71 times, each time without one team. C: all 24 combinations of weights (1/nmodel or none), outcome (raw or winsorised AME), discipline control (composition FE, lead-author FE, none) and group rule (published or majority).

Comparison with the expectations in section 4a.

Leave one team out. The estimate ranges from 0.052 (without team 42) to 0.101 (without team 28). All 71 estimates have p < 0.05. Team 42 (anti, 2 models, mean AME −0.258) is the most influential team, but the result does not depend on it.

All 24 combinations (panel C). Every estimate with a discipline control is positive (0.024 to 0.085), and 13 of these 16 have p < 0.05. None of the 8 estimates without a discipline control has p < 0.05 (range −0.020 to 0.041). The published specification gives the largest estimate of all 24.

Summary of the robustness checks.

• Direction. All 8 alternatives that change the estimate (A1–A8) and all 16 grid specifications with a discipline control give a positive difference.
• Size. Under the alternatives the difference is 0.041–0.069, against 0.085 published. The published combination of choices gives the largest estimate.
• Significance. The result stays significant at 5% under 5 of the 8 alternatives that change the estimate (A1, A3, A4, A5, A7). It is not significant at 5% with team-median outcomes (A2, p = 0.176), without discipline fixed effects (A6, p = 0.164) or with team random intercepts (A8, p = 0.052). For the baseline estimate, CR2 SEs give p = 0.065 and the wild cluster bootstrap gives p = 0.007.
• Dependence on the discipline control. Without any discipline control the difference is 0.041 or smaller and never significant. The paper names this control as the one that produces the effect.

5. Does the headline claim hold up?

Under every alternative with a discipline control, pro-immigration teams report more positive AMEs than anti-immigration teams, so the claim holds in direction. The published 0.085 is the largest estimate in the set; most alternatives give about half to four-fifths of it. Significance at 5% depends on the discipline fixed effects, the outcome summary and the small-sample correction, so the evidence for the size and precision of the effect is weaker than Table 1 suggests.