Lecture 5 Critiquing social psychology and understanding the replication crisis

Fueled by concerns about the reliability of psychological research, a large scale project in 2015 (Open 2015) attempted to replicate some major findings in the field. The results were a shock for many people - especially in social psychology, only a minority of findings were confirmed in replication studies. Since then, there has been a debate about why so many findings are not replicated, and what can be done about that. We will be discussing how social psychology research can be made more robust. In addition, we will consider more fundamental critiques of the way most research in social psychology is done and consider the case for using qualitative research methods. Finally, we will look at the attitude-behaviour links and consider how a focus on studying only attitudes might impoverish our conclusions.

5.1 Video lecture for this week

You can download the slides for this week’s lectures here - maybe download them now to follow along, there is quite a bit of text on some of them.

5.1.1 Qualitative critiques of social psychology

Before you move on, take a moment to reflect on the content of the video and answer the following questions:

• Which of the following is not a critical challenge to quantitative social psychology? There is no objective reality, instead we construct it collectively through language There is no place for a neutral observer - we are always embedded in relationships We should follow the scientific method and use accurate measures We need to check our assumption that problems are located in the individual
• What are WEIRD samples in psychology? When do you think we should worry about them? When do you think they are sufficient?

5.1.2 Replication crisis

NOTE: The sound quality issue with this video should now be fixed. If you tried to watch it before Tuesday afternoon, please give it another go.

Before you move on, take a moment to reflect on the content of the video and answer the following questions:

• The Reproducibility Project repeated 100 important studies that had reported significant results. Out of the social psychology studies, what share of replications yielded significant results? 10% 25% 50% 80%
• What are Type I errors? False positives False negatives Are they always a sign of mistakes/problems with the research process? TRUE FALSE

Don’t worry about remembering what Type I and Type II errors are (at least for this module), but make sure that you understand the difference between false positives and false negatives. Also, these errors are not always a sign of problems with the research process. Statistics is about quantifying the probability that a trend we observe in a sample tells us something real about a broader population - we then need to decide which belief to adopt, and will inevitable be wrong at times. However, having many more errors in the literature than expected is a problem, and too many failed replications are a definitive sign of that.

• Have a look at this tool for p-hacking. How might something similar happen in social psychology? Do you understand why this invalidates p-values? Otherwise, make sure to raise this in class.

5.1.3 Attitude-behaviour links

Before you move on, take a moment to reflect on the content of the video and answer the following questions:

• How would you explain the relationship between attitudes and behaviours, having watched the video? What does that mean for interpreting survey studies that measure attitudes?
• What was the main finding in LaPiere’s 1934 study about attitudes and behaviours? Discriminatory attitudes were more widespread than discriminatory behaviours Discriminatory behaviours were more widespread than discriminatory attitudes Discriminatory attitudes predicted discriminatory behaviours well How would you explain that finding?
• Have a look at the Parable of Polygons. Do you find any of it surprising? Can you think of any sphere in your personal life where a similar effect is visible?

5.2 Preparation before class

I’d encourage you to give the multiple choice exam - available from Tuesday on Moodle - a first go and bring your questions to class.

5.3 Recommended readings

• The seminal 2015 article by the Open Science Collaboration. It can be freely downloaded here and has a 1-page summary that you should read at the very least.

• The article in the BPS Research Digest on 10 famous psychology findings that its been difficult to replicate

• Wikipedia entry on the replication crisis. It gives a good overview over the issue in various disciplines and the various responses that have been proposed.

• A very new (short) article on “What is replication?” (Nosek and Errington 2020) - it shows quite nicely how replications can be not just a clean-up exercise but an important contribution to scientific progress.

• If you are curious about the start of the replication debate, you might want to read the article that kicked it all off: “Feeling the future” (Bem 2011) that provided experimental evidence for our ability to, well, feel the future before it happened. It’s not essential reading; if you read it, think about what had to go wrong for this to get published in an influential journal.