An interesting little discussion popped up in the wild and wooly new media world in science (e.g., podcasts and twitter) concerning the relative merits of “descriptive” vs “hypothesis” driven designs. All, mind you, indirectly caused by the paper that keeps on giving—Tal Yarkoni’s generalizability crisis paper.
Inspired by Tal’s paper, a small group of folks endorsed the merits of descriptive work and the fact that psychology would do well to conduct more of this type of research (Two Psychologist, Four Beers; Very Bad Wizards). In response, Paul Bloom argued/opined for hypothesis testing–more specifically, theoretically informed hypothesis testing of a counterintuitive hypothesis.
I was implicated in the discussion as someone who’s work exemplifies descriptive research. In fact, Tal Yarkoni himself has disparaged my work in just such a way.* And, I must confess, I’ve stated similar things in public, especially when I give my standard credibility crisis talk.
So, it might come as a surprise to hear that I completely agree with Bloom that a surgical hypothesis test using experimental methods that arrives at what is described as a “counterintuitive” finding can be the bee’s knees. It is, and probably should be, the ultimate scientific achievement. If it is true, of course.
That being said, I think there is some slippage in the verbiage being employed here. There are deeper meanings lurking under the surface of the conversation like sharks waiting to upend the fragile scientific dingy we float in.
First, let’s take on the term that Bloom uses, “counterintuitive,” which is laden with so much baggage it needs four porters to be brought to the room. It is both unnecessary and telling to use that exact phrase to describe the hypothetical ideal research paradigm. It is also, arguably, the reason why so many researchers are now clambering to the exits to get a breath of fresh, descriptive air.
Why is it unnecessary? There is a much less laden term, “insight” that could be used instead. Bloom partially justifies his argument for counterintuitive experiments with the classic discovery by Barry Marshall that ulcers are not caused by stress, as once was thought, but by a simple bacteria. Marshall famously gave himself an ulcer first, then successfully treated it with antibiotics. Bloom describes Marshall’s insight as counterintuitive. Was it? There was a fair amount of work by others pointing to the potential of antibiotics to treat peptic ulcers for several decades before Marshall’s work. An alternative take on the entire process of that discovery was that Barry Marshall had an insight that led to a “real” discovery that helped move the scientific edifice forward–as in, we acquired a better approximation of the truth and the truth works. As scientists, we all strive to have insights that move the dial closer to truth. Calling those insights counterintuitive is unnecessary.
It is also telling that Bloom uses the term counterintuitive because it has serious sentimental value. It is a term that reflects the heady, Edge-heavy decades pre-Bem 2011 where we could publish counterintuitive after counterintuitive finding in Psychological Science using the Deathly Hallows of Psychological Science (because that was what that journal was for after all) that in retrospect were simply not true. Why were they not true? Because our experimental methods were so weak and our criteria for evaluating findings so flawed that our research got unmoored from reality. With a little due diligence–a few QRPs, a series of poorly powered studies, and some convenient rationalizations–(e.g., some of my grad students don’t have the knack to find evidence for what is clearly a correct hypothesis….), one could cook up all sorts of counterintuitive findings. There was so much counterintuitive that counterintuitive became counterintuitive. And why did we do this? Because Bloom is right. The coolest findings in the history of science have that “aha” component demonstrated with a convincing experimental study.
This is not to say that p-hacking and counterintuitive experimental methods are synonymous, just that as a field we valued counterintuitive findings so much that we employed problematic methods to arrive at them. Because of this unfortunate cocktail, the “counterintuitive” camp still has serious, painful reckoning to face. We got away with methodological malpractice for several decades in the service of finding counterintuitive results. And, it was so cool. We were stars and pundits and public intellectuals riding a wave of confection that went poof. We ate ice cream for breakfast, lunch, and dinner. Even a small dose of methodological rigor dished up in the form of “eating your vegetables” is going to feel like punishment after that. But since the reproducibility rate of all of those counterintuitive findings is holding steady at less than 50%, I believe some vegetables are in order–or maybe an antibiotic. Having had an ulcer, I know first hand the robustness of Marshall’s work. The relief that occurred after the first dose of antibiotics was profound. Psychology does not currently produce robust findings like that. To opine for the old days when we could flog the data to produce counterintuitive results without first cleaning up our methods, while understandable, is also counterproductive.
Naturally, many folks have reacted to the credibility crisis, which the counterintuitive paradigm helped to foster, with something akin to revulsion and have gone in search of alternatives or fixes, some conceptual, some methodological. One consistent line of thinking is that we should prioritize a range of methods roughly described with terms like descriptive, observational, and exploratory. I’m going to go out on a slight nerdy, psychometrically-inspired interpretive limb here and say that these are all manifest indicators of the true latent factor behind these terms–reality-based research. A bunch of us would prefer that the work we do is grounded in reality–findings that are robust or, even more provocatively, findings that reflect the true nature of human nature.
Chris Fraley put it to me well. He said that the call for more descriptive and exploratory research is grounded in the concern that we don’t have a sound foundation for understanding how and why people behave the way they do. The theory of evolution by natural selection, for example, would have not come about but for a huge repository of direct observations of animal behavior and morphology. Why not psychology? It seems reasonable that psychology should have a well-documented picture of the important dimensions of human thought, feeling, and behavior that is descriptively rich, grounded in the lives that people lead, accurate, and repeatable. When I hear colleagues say that we should do more descriptive work, this is what I’m hearing them say.
Preferably, this real understanding of human behavior would then be the basis upon which insightful experiments would be tested.
Of course, Bloom is partially right that descriptive work can and should put people to sleep. Much of my work does.** Just ask my students. And just by being descriptive, it may not be any more useful than a counterproductive counterintuitively motivated experiment. What of all of that descriptive, observational work on ulcers before Barry Marshall’s work? It had come to the conclusion that stress caused ulcers. Would another observational study of stress and ulcer symptoms have brought insights to this situation? How about a fancy longitudinal, cross-lagged panel model? Ooooh, even better, a longitudinal, growth mixture model of stress and ulcer groups. I’m getting the vapors just thinking about it. No, sorry, given my experience with ulcers I prefer a keen insight into the mechanisms that allowed ulcers to be treated quickly and easily, thank you.
That said, the fetishizing of clever counterintuitiveness and demeaning of descriptive work as boring also smacks of elitism. After all, the truth doesn’t care if it is boring. I remember watching in bemused wonderment back in grad school when Oliver John would receive ream after ream of factor structures in the snail mail from Lew Goldberg who was at the time cranking out the incredibly boring analyses that arrived at the insight that most of how we describe each other can be organized into five domains. It was like watching an accountant get really excited about a spreadsheet. On the other hand, the significance of the Big Five and the revolutionizing effect it has had on the field of personality psychology cannot be overstated. If there was an aha moment it wasn’t the result of anything counterintuitive.
And the trope that observation and description of humans are intrinsically boring is possibly more of an indictment of psychologists’ lack of imagination and provincialism than anything else. After all, there is an entire field across the quad from most of us called Anthropology that has been in the practice of describing numerous cultures, countries, and tribes across the globe. Human ethology, cultural anthropology, and behavioral ecology are remarkably interesting fields with often surprising insights into the uniquenesses and commonalities of all peoples. One could argue that we could get a head start on the whole description thing by reading some of their work instead of cooking up our own stew of descriptive research.
If there is a little homily to end this essay I guess it would be not to lionize either description or counterintuitive methods. Neither method has the market cornered on providing insight.
* Just kidding. I think he meant it as a compliment.
** They say sleep is good for you. Therefore, my research can and does have a positive impact on society.
pigee 