Never totally trust this phrase or any of its variants – “a study shows,” “scientists claim, ” etc.
Who did the research? How much experience did they have, and what biases – from cognitive to selection to sampling to publication – may have been involved?
Who funded the research, and how much influence did they have? What were their potential biases?
How many people were asked, examined, tested, and according to what methods?
Do any claimed findings show causation, or just correlation?
If they claim causation, what are the other possible explanations? There are always other possible explanations and often a lot of them. Were they considered? How convincingly were they ruled out?
Most importantly, has the research been validated by others and, where applicable, replicated? How many times? (By whom, with how many people. You get the idea.)
A by-product of the push-button publishing world we live in these days is that “research” is everywhere, and there is a lot of pressure to publish it. Certainly, that applies for traditional academics, but the thirst for research has multiplied exponentially with the growth of social media and content marketing. There is a continual and growing need to feed the beast.
And, the phenomenon extends well beyond social media and marketing. We live in an age in which one of the most popular forms of non-fiction is essentially to weave a tapestry from small studies – usually not replicated – that serve the larger story the author wants to tell. (Also known as confirmation bias.) Whenever you see “The Surprising Truth About …” or something similar in a title, proceed with critical caution.
This doesn’t mean we should never trust anything or start subscribing to conspiracy theories, but it does mean that, as serious lifelong learners, we need always to maintain a healthy skepticism, embrace epistemological modesty, and cultivate our critical thinking skills (including on this site!).
This, of course, is purely my opinion. I don’t know what the research says on this one.
And I probably wouldn’t trust it anyway.
Additional Resources
Here are some other resources to help you apply a healthy level of skepticism to any research you encounter:
Studying Studies
A great series from Dr. Peter Attia, a committed scientific thinker and one of my favorite sources on health and medicine.
- Studying Studies: Part I – relative risk vs. absolute risk
- Studying Studies: Part II – observational epidemiology
- Studying Studies: Part III – the motivation for observational studies
- Studying Studies: Part IV – randomization and confounding
- Studying Studies: Part V – power and significance
Thanks for commenting, Adeyemi. Love that story about the workshop. Lot’s of that kind of stuff out there! – Jeff
This is so relatable and also hilarious at the same time. I once attended a “workshop” where one of their sales strategy is to use the words “research says”. In their opinion, it offers credibility to whatever they have to say and will help them with more sales.
The world we live in today is so obsessed with getting results that most people don’t care about the means so far it justifies the end. Just as you pointed out earlier, this is purely my opinion based on personal experience. I don’t know what the research says about this.