Is that Good Science? A Guide to Cutting through BS
Part of getting an education isn’t about learning what to think, it is about learning how to find good information. When it comes to scientific literature [especially on-line], it’s hard to separate the good from the bad if you don’t know what to look for. With the emergence of pseudoscience in the mainstream I think it’s important to go over a few red flags when it comes to claims being made.
Here is 6 things to keep in mind when you are reading any sort of article, to help you sort through all the BS:
1. The burden of proof lies with the person making the claim
This happens a lot in scientific debates against a pseudoscience claim, lets use the climate change issue as an example. There is plenty of research showing that climate change is manmade [link from this post], the burden of proof that climate change is not man made is with the group making such claims.
We know that climate change is caused by human activity and have more than enough data to back that up. However, if someone is going to make a claim contrary to that, they need to have some sort of peer reviewed generally accepted proof. There is no magic involved, have proof for your claim and the science will follow.
2. Control groups are important in any study
Anytime you see some homeopathic, naturopathic, or alternative medical claim that it works as well [or better] than the mainstream medical option, first of all you should walk away, but more importantly you need to see the studies. Most studies done show that alternative medicine does not work. But some “groups” do their own studies and they work, this is because most pro alternative medicine studies done have poorly designed control groups which don’t allow for bias by the researchers [not to mention the conflict of interest aspect when funding your own study].
You want to ideally find a double blind study– meaning the neither the researcher,s nor the participants know who is or is not taking the actual medicine vs. the placebo — which is important since everyone is bias and this way you can do your best eliminate this bias.
3. Don’t confuse correlation with causation
We are wired with a cause and effect sort of mentality, I put my hand on the stove and burned it– the cause was I put my hand on the stove and the effect was I burned it. This does not always work, especially in science.
A good example is vaccinations, the anti vaccination movement argues that because autism has been on the rise since the introduction of vaccines then the vaccines are the cause of autism.
This is like trying to argue that everyone who robs banks wear black, so anyone who wears black robs a bank. Another good example would be looking at the graph below, showing the relationship between organic food sales and autism, which would be crazy to think that organic food causes autism, even though you can still draw a correlation between the two.
4. All natural isn’t always better
When reading an article be weary of anything that touts the benefits of “all natural”, organic or claims that synthetic, man-made, or “artificial” things are bad.
A good example, mercury and arsenic can kill you, despite being all natural and organic. Water in high enough amounts can kill you, in fact pretty much anything in high enough amounts can kill you natural or otherwise. A good rule of thumb is that there are plenty of man made things that are bioidentical, you would never be able to tell the difference between the natural and the man made. Just because something is “man made” does not make it bad, evil, or less than anything “natural”, see number one if anything you read disagrees.
5. Do other peer-reviewed studies agree? If not, why?
There are plenty of “oddball” studies out there suggesting something against mainstream science– the key to any good study is that the results are repeatable. Any study that offers you the “truth” about something is not even worth the paper it is printed on if you can’t repeat the study.
A cautionary tale and a great example would be the new[ish] stem cell paper out offering a new and cheap way to turn adult skin cells into stem cells. When it came out that no one could repeat the studies findings, not only was it pulled, the head author is under review because of it.
6. There are no magical forces
In science if it cannot be measured, it doesn’t exist. There are people in the world who believe in mystical energies, magnetic, crystal, the earth, or in the case of acupuncture releasing some magical energy block in the body [which, yes acupuncture does nothing], none of these things can be proven and more importantly they don’t do anything — because they don’t exist [except for magnetism, which also doesn’t do anything for health, again see the first point].
So any sort of non-established mechanism or mysticism involved in a study or article should be avoided if you are actually looking for something scientifically relevant.
There are too many things that could be added to this list to actually list, so I just picked the first few things that I think are important to keep in mind when sifting through peer reviewed research [or articles on peer reviewed research like this blog features].
The hope is to pass on some knowledge to people who are serious about learning, but don’t know what to trust. Want help with researching? Try Google Scholar for peer reviewed articles and other resources.
Pseudoscience is everywhere, chiropractic medicine, naturopathic, homeopathic, acupuncture, all BS. Antivaccination movement, global warming natural is movement, young earth and creationist movements all BS. Don’t agree? Prove it.
So how about you guys, what do you think is important to look for in research?
White P. (2012). Practice, practitioner, or placebo? A multifactorial, mixed-methods randomized controlled trial of acupuncture, Journal of the International Association for the Study of Pain, 153 (2) 455-462. DOI: 10.1016/j.pain.2011.11.007
Lovejoy S. (2014). Scaling fluctuation analysis and statistical hypothesis testing of anthropogenic warming, Climate Dynamics, 42 (9-10) 2339-2351. DOI: 10.1007/s00382-014-2128-2
Ventola C.L. Current Issues Regarding Complementary and Alternative Medicine (CAM) in the United States: Part 1: The Widespread Use of CAM and the Need for Better-Informed Health Care Professionals to Provide Patient Counseling., P & T : a peer-reviewed journal for formulary management, PMID: 20844696
Ventola C.L. Current Issues Regarding Complementary and Alternative Medicine (CAM) in the United States: Part 2: Regulatory and Safety Concerns and Proposed Governmental Policy Changes with Respect to Dietary Supplements., P & T : a peer-reviewed journal for formulary management, PMID: 20975811
Ventola C.L. Current Issues Regarding Complementary and Alternative Medicine (CAM) in the United States: Part 3: Policies and Practices Regarding Dietary Supplements In Health Care Facilities., P & T : a peer-reviewed journal for formulary management, PMID: 21037910