I will repeat my favorite example of this. Everyone understands that, when you flip a coin, it has an equal chance of coming up heads or tails. If you flip it twice, however, it could easily come up heads twice or tails twice, and if you didn’t understand the effect of sample size, you could easily be misled about the probabilities. If you flip it 1,000 times, you will get pretty close to 500 each heads and tails – and learn the truth.

Anyone who engages in scientific research should understand the effect of sample size on the value of the results and can do something biostatisticians call a “power calculation” to figure out how big the study has to be to yield meaningful results. Assuming these researchers were not clueless, and assuming they had access to someone who knew enough stats to do a power calculation, I find the only explanation for doing such a small study is that they intended it to be a very preliminary pilot study, a sort of “proof of concept” trial. But that isn’t how it was reported in the journal.

So the small sample size is major flaw #1. But there’s more.

When one does a study to test the effects of a treatment, the best design is a randomized, controlled trial (or RCT), and an RCT is best done in a way that is “placebo controlled” and “blinded.” Ideally it should be “double blind,” so that neither the subjects nor investigators know who got active treatment and who got placebo. Often, from the subjects’ perspective, placebo control and blinding are closely intertwined. It is easy to see the challenge of placebo control and blinding in this study.

Subjects randomly assigned to the active treatment group smoked marijuana cigarettes with a standardized amount of tetrahydrocannabinol (THC, the active ingredient). The control subjects smoked “sham” cigarettes made from cannabis flowers from which the THC had been extracted.

So how could blinding and the placebo effect work in this trial? Let us assume, giving the authors maximum benefit of the doubt, that they recruited only subjects who had never smoked marijuana and who would therefore be unfamiliar, at least from personal experience, with its effects. (The paper didn’t say that, and so it probably isn’t true, but I’m trying to help them out here.) Even then, subjects probably all had at least some idea what marijuana is supposed to do and could very easily tell when they smoked their cigarettes, real or sham, which they were. So there could not possibly have been any real placebo control or blinding effect. And that is major flaw #2.