In the post about the Beginning of History Effect, I used candidate gene research as a case in point to illustrate how unreplicable research can get lodged in the scientific literature and how it is difficult to then dislodge. A case in point emerged with perfect timing this weekend in the New York Times Magazine. In a horribly sourced story on “Why some kids handle pressure, while others fall apart”, the authors claim that “One particular gene, referred to as the COMT gene, could to a large degree explain why one child is more prone to be a worrier, while another may be unflappable, or in the memorable phrasing of David Goldman, a geneticist at the National Institutes of Health, more of a warrior.”
Just shoot me.
Keep in mind that there is not only a meta-analysis of existing studies showing that the relation between the COMT polymorphism and cognitive functioning is indistinguishable from zero (Barnett, Scoriels, & Munafo, 2009), but also a comprehensive review that shows that all the existing associations between gene polymorphisms and cognitive functioning are zero (Chabris et al 2012). No, the authors can’t let science stand in the way of selling their new book or the NY Times from actually checking the veracity of the claims made in the article–we need to sell ad space after all. No, it is far more important to misinform thousands, if not millions of students and their parents that their test anxiety can be attributed to one genetic polymorphism, even if it is not true.
This is one illustration of the way unreplicable research gets instantiated in our field and in the minds of the public and inevitably, granting agencies. The original article and/or idea is compelling. It fits a broad world view that some groups researchers/people want to believe. All subsequent disconfirmations of the effect are rationalized or ignored. Then, we spend 20 years or so waisting our time because the majority of follow up research fails to replicate the original effect and the original idea just fades away. Also keep in mind that the genetics literature is far better than most of our paradigms in psychology because the failed replications are often published–that is, if people are motivated to write them up. Think of the far more insidious situation where a finding is p-hacked to prominence in JPSP and few if any disconfirmations are published, because we “don’t publish replications.” Those fake results can live forever as they infiltrate text books and theoretical reviews in psychology.
At some point we have to decide whether we are doing science or propaganda. If we are going to do science, we need to care more about having robust ideas than robust careers.
Brent W. Roberts