I seem to replicate the same conversation on Twitter every time a different sliver of the psychological guild confronts open science and reproducibility issues. Each conversation starts and ends the same way as conversations I’ve had or seen 8 years ago, 4 years ago, 2 years ago, last year, or last month.
In some ways that’s a good sign. Awareness of the issue of reproducibility and efforts to improve our science are reaching beyond the subfields that have been at the center of the discussion.
Greater engagement with these issues is ideal. The problem is that each time a new group realizes that their own area is subject to criticism, they raise the same objections based on the same misconceptions, leading to the same mistaken attack on the messengers: They claim that scholars pursuing reproducibility or meta-science issues are a highly organized phalanx of intransigent, inflexible, authoritarians who are insensitive to important differences among subfields and who to impose a monolithic and arbitrary set of requirements to all research.
In these “conversations,” scholars recommending changes to the way science is conducted have been unflatteringly described as sanctimonious, despotic, authoritarian, doctrinaire, and militant, and creatively labeled with names such as shameless little bullies, assholes, McCarthyites, second stringers, methodological terrorists, fascists, Nazis, Stasi, witch hunters, reproducibility bros, data parasites, destructo-critics, replication police, self-appointed data police, destructive iconoclasts, vigilantes, accuracy fetishists, and human scum. Yes, every one of those terms has been used in public discourse, typically by eminent (i.e., senior) psychologists.
Villainizing those calling for methodological reform is ingenious, particularly if you have no compelling argument against the proposed changes*. It is a surprisingly effective, if corrosive, strategy.
Unfortunately, the net effect of all of the name calling is that people develop biased, stereotypical views of anyone affiliated with promoting open and reproducible science**. Then, each time a new group wrestles with reproducibility, we hear the same “well those reproducibility/open science people are militant” objection, as if it is at all relevant to whether you pre-register your study or not***. And this is not to say that all who promote open and reproducible science are uniformly angelic. Far from it. There are really nasty people who are also proponents of open science and reproducibility, and some of them are quite outspoken.
Just. Like. Every. Other. Group. In. Psychology****.
And, just like every other group in psychology, the majority of those advocating for reform are modest and reasonable. But as seems to be the case in our social media world, the modest and reasonable ones are lost in the flurry of fury caused by the more noisy folk. More importantly, the existence of a handful of nasty people has no bearing on the value of the arguments themselves. Regardless of whether you hear it from a nasty person or a nice one, it would improve the quality of our scientific output if we aspired to more often pre-register, replicate, post our materials, and properly power our studies.
The other day on Twitter, I had the conversation again. My colleague Don Lynam (@drl54567) likened the sanctimonious of the reproducibility brigade to ex-smokers, which at first blush was a compelling analogy. Maybe we do get a bit zealous about reproducibility because we’ve committed ourselves to the task. Who hasn’t met a drug and alcohol counselor or ex-smoker who isn’t a tad bit too passionate about helping us to quit drinking and smoking?
But, as I told Don, a better analogy is water sanitation.
The job of a water sanitation engineer is to produce good, clean water. Some of us, circa 2011 or so*****, noticed a lot of E. coli in the scientific waters and concluded that our filtration system was broken. Some countered that a high amount of E. coli is normal in science and of no concern. Many of us disagreed. We pointed out how easily the filtration system could be improved to reduce the amount of E. coli–pre-registering our efforts, making our data and methods more open and transparent, directly replicating our own work, adequately powering our studies so that they actually can work as a filter–you get my point.
When you replace “scientific reform” with “water filtration” and “our subfield” with “our water source”, it reveals why having this same conversations over and over is so frustrating:
Them: “The water in our well is clean. There is no problem.”
Us: “Have you tested your water (e.g., registered replication report)?”
Them: “No.”
Us: “Then you can’t really be confident that your water is clean.”
Them: “Stop being so militant.”
Or,
Them: “I haven’t noticed any problems with our well, so there’s no reason to doubt the effectiveness of our filtration system.”
Us: “Has anyone else applied your filtration system to another well to make sure it works (direct replication)?”
Them: “No. Having other people do the same thing we do isn’t necessary (it’s a waste of time).”
Us: “But if you haven’t tested the effectiveness of your filtration system, how can you be sure that your filter works?”
Them: “Stop being so sanctimonious.”
Or,
Them: “Look at my shiny, innovative filtration system that I just created.”
Us: “Has it been tested in different wells (pre-registered study)?”
Them: “No. my job is to create new and shiny filters, not test whether they work for other people.”
Us: “But the water still has E. coli in it.”
Them: “Stop being such an asshole.”
Or,
Us: “Your well doesn’t give off enough water to even test (power your research better).”
Them: “What little water we have has always been perfectly clean”
Us: “How about if we dig your well deeper and bigger so we can get more water out of it to test?”
Them: “How dare you question the quality of my water you terrorist.”
Or,
Them: “We get pure, clean water from every well we dig”
Us: “Awesomesauce. Can you share your filtration system (open science)?”
Them: “With you? You’re not even an expert. You wouldn’t understand our system.”
Us: “If you post it in the town square we’ll try and figure it out with your help.”
Them: “Unqualified vigilante.”
Much of the frustration that I see on the part of those trying to clean the water, so to speak, is that the changes are benign and the arguments against the changes are weak, but people still attack the messenger rather than testing their water for E. coli. We have students getting sick (losing ground in graduate school) from drinking the polluted water (wasting time on bogus findings), and they blame themselves for drinking from non-potable water sources.
In the end, it would be lovely if everyone were kind and civil. It would be great if folks would stop using overwrought, historically problematic monikers for people they don’t like. But we know from experience that one person’s sober and objective criticism of a study is another person’s methodological terrorism. We know that being the target of replication efforts is intrinsically threatening. The emotions in science have been and will continue to run raw. When these conversations focus on the tone or the unsavory personal qualities of those suggesting change, it shows how powerfully people want to avoid cleaning up the water.
Of course, emotional reactions and name calling are immaterial to whether there is E. coli in the water. And, it is in every scientist’s long-term interest to fix our filtration system******. Because it is broken. Those promoting open science and the techniques of reproducibility are motivated to improve the drinking water of science. Tools, like pre-registration, posting materials, direct replication, increased power are not perfect and they merit ongoing discussion and improvement. Yet presently, if you happen to be sitting on what you believe to be an unspoiled well-spring of scientific ideas, there is no better way to prove it than to have another team of scientists test your ideas in a well-powered, pre-registered, direct replication. When the results of that effort come in, we will be happy to discuss the findings, preferable in civil tones with no name calling.
Brent W. Roberts
*I’m not sure it was a deliberate decision, but if you want to avoid changing your methods, making the people the issue, not the ideas, is a brilliant strategy.
**In one very awkward, tragic dinner conversation one of my most lovely, kind colleagues described another one of my lovely, kind colleagues as a bully based solely on second hand rumors based on the name calling.
***A pre-registered hypothesis in need of testing–anyone who tells you the open science cabal is a cabal or militant or nasty or any other bad things, are scholars who have not attempted the reforms themselves and are looking for reasons not to change.
****And in science as a whole. And in life for that matter.
*****Some way before that.
******It might not be in every scientist’s short-term interest to do things well….
pigee 
I recently read a new criticism of Hans Eysenck’s research on cancer-prone personality types. As far back as 1993, Eysenck described critics of his severely implausible results as “Pedantic to the last degree, any error, however slight, random, and unimportant from the point of view of the grand design, is a sin against the Holy Ghost, to be hunted down, exposed and eradicated.”
It is interesting to look back 25 years and see the same trope of “fast-and-loose visionaries vs. miserable bean counters”.
See https://journals.sagepub.com/doi/10.1177/1359105318822045