How about a little blast from the past? In rooting around in an old hard drive searching for Pat Hill’s original CV [1], I came across a document that we wrote way back in 2006 on how to write more effectively. It was a compilation of the collective wisdom at that time of Roberts, Fraley, and Diener. It was interesting to read after 13 years. Fraley and I have updated our opinions a bit. We both thought it would be good to share if only for the documentation of our pre-blogging, pre-twitter thought processes.
Manuscript Acronyms from Hell: Lessons We’ve Learned on Writing the Empirical Research Article
By Brent Roberts (with substantial help from Ed Diener and Chris Fraley)
Originally written sometime in 2006
Updated 2019 thoughts in blue
Here are a set of subtle lessons that we’ve culled from our experience writing journal articles. They are intended as a short list of questions that you can ask yourself each time you complete an article. For example, before you submit your paper to a journal, ask yourself whether you have created a clear need for the study in the introduction, or whether everything is parallel, etc. This list is by no means complete, but we do hope that it is useful.
Create The Need (CTN). Have you created the need? Have you made it clear to the reader why your study needs to be done and why he or she should care? This is typically done in one of two ways. The first way is to show that previous research has failed to consider some connection or some methodological permutation or both. This means reviewing previous research in a positive way with a bite at the end in which you explain that, despite the excellent work, this research failed to consider several things. The second way is to point out that you are doing something completely unique. Even if you are taking this approach, you should review the “analogue” literature. The analogue literature is a line of research that is conceptually similar in content or method, but not exactly like your study.
Fraley 2019: I try to encourage my students to do this based on the ideas themselves. Specifically, the question should be so important, either for theoretical reasons or due to their natural appeal, that the “so what, who cares, why bother” (i.e., the “Caroline Trio”) is unambiguous.
I don’t like it when authors justify the need by saying something along the lines of “No one has addressed this yet.” Research doesn’t examine the association between coffee consumption and the use of two vs. one spaces after a period either. Thus, there is a gap in the literature. But the gap is appropriate: There is no need to address that specific question.
I mention this simply because, imo, the “need” should emerge not only from holes/limitations in the literature. The “need” should also be clear independently of what has or has not been done to date.
Always Be Parallel (APB). Every idea that is laid out in the introduction should be in the methods, results, and discussion. Moreover, the order of the ideas should be exactly the same in each section. Assume your reader is busy, tired, bored, or lazy, or some combination of these wonderful attributes. You don’t want to make your reader work too hard, otherwise they will quickly become someone who is not your reader. Parallelism also refers to emphasis. If you spend three pages discussing a topic in the introduction and two sentences in the results and discussion on the same topic, then you either have to 1) cut the introductory material, or 2) enhance the material in the results and discussion.
Correlate Ideas and Method (CIM). The methods that you choose to adopt in your study should be clearly linked to the concepts and ideas that inspire your research. Put another way, the method you are going to use (e.g., correlation, factor analysis, text analysis, path model, repeated measures experiment, between-subject experiment) should be clear to the readers before they get to the method section.
Eliminate All Tangents (EAT). If you introduce an idea that is not directly germane to your study, eliminate it. That is, if it is not part of your method or not tested in your results then eliminate it from your introduction. If it is important for future research, put it in your discussion. Remember Bem’s maxim: If you find a tangent in your manuscript make it a footnote. In the next revision of the paper, eliminate all footnotes[2].
Roberts 2019: It is interesting looking back now and seeing that I cited without issue, Bem’s chapter that everyone now excoriates for containing a recipe for p-hacking. Yes, I used to assign that chapter to my students. In retrospect, and even now, his p-hacking section did not bother me, largely because I’ve always been a fan of exploratory and abductive approaches to research—explore first, then validate. If you are going to explore, then it is typically good to report what your data show you. Of course, you should not then “HARK the herald angels sing” and make up your hypotheses after that fact.
Always Be Deductive (ABD). Papers that start with a strong thesis/research question read better than papers that have an inductive structure. The latter build to the study through reviewing the literature. After several pages the idea for the study emerges. The deductive structure starts with the goal of the paper and then often provides an outline or advance organizing section at the beginning of the article informing the reader of what is to come.
Fraley 2019: I don’t endorse this claim strongly (but I think it has its uses). I think this mindset puts the author in the position of “selling” an idea–When authors use a deductive structure, I start to question their biases and whether they are more committed to the idea that motivates the deduction or the facts/data that could potentially challenge that framework.
No Ad Hominen Attacks (NAHA). Don’t point out the failings or foibles of researchers, even if they are idiots. This will needlessly piss of the researcher, who is most likely going to be a reviewer. Or, it will piss off friends of the researcher, who are also likely to be reviewers. If you are going to attack anything, then attack ideas[3].
Fraley 2019: Only an idiot would make this recommendation.
Roberts 2019: Have you considered being more active on Twitter?
Contrast Two Real Hypotheses (CTRH). Although not attainable in every instance, we like to design studies and write papers that contrast two theoretical perspectives or hypotheses in which one of the hypotheses is not the null hypothesis. This accomplishes several goals at once. First, it helps to generate a deductive structure. Second, it tends to diminish the likelihood of ad hominin attacks, as you have to give both theoretical perspectives their due. In terms of analyses, it tends to force you into contrasting two models rather than throwing yourself against the shoals of the null hypothesis every time, which is relatively uninteresting.
Fraley 2019: This is the most important idea, in my mind. This is also a good place to call attention to Platt’s Strong Inference paper.
Writing Is Rewriting (WIR). There is no such thing as a “final” draft. There is simply the paper that you submit. This is not to say that you should be nihilistic about your writing and submit slipshod prose because there is no hope of attaining perfection. Rather, you should strive for perfection and learn to accept the fact that you will never achieve it.
Two-Heads-Are-Better-Than-One (THABTO). Have someone else read your paper before turning it in or submitting it. A second pair of eyes can detect flaws that you have simply habituated to after reading through the document for the 400th time. This subsumes the recommendation to always proofread your document. In general, we recommend collaborating with someone else. Often times, a second person possess skills that you lack. Working with that person leverages your combined skills. This inevitably leads to a better paper.
Grammarish rules:
Use Active Language (UAL). Where possible, eliminate the passive voice.
Define Your Terms (DYT). Make sure you define your concepts when they are introduced in the paper.
One Idea Per Sentence (OIPS).
Review Ideas Not People (RINP). When you have the choice of saying “Smith and Jones (1967) found that conscientiousness predicts smoking,” or “Conscientiousness is related to a higher likelihood of smoking (Smith & Jones, 1967),” choose the later.
Don’t Overuse Acronyms (DOA).
Ed Diener summarizes much of this more elegantly. When writing your paper make the introduction lead up to the questions you want to answer; don’t raise extra issues in the introduction that you don’t answer. Make it seem like what you are doing follows as the next direct and logical thing from what has already been done. Moreover, emphasize that what you are doing is not just a nice thing to do, but THE next thing that is essential to do.
Happy rewriting.
Fraley 2019: Another idea worth adding: Write in a way that would allow non-experts to understand what you’re doing and why. Also, many of your readers might not be Native-English speakers. As such, it is best to write directly and avoid turns of phrase or idioms. Focus on communicating ideas rather than showing off your vocabulary or your knowledge of obscure ideas.
Roberts 2019: I can’t help but think about this list of recommendations in light of the reproducibility crisis. The question I would ask now is whether these recommendations apply as well to a registered report as they would to the typical paper from 2006. I think the 2006 list implicitly accepted some of the norms of the time, especially that the null is never accepted, at least for publication, and HARKing was “good rhetoric.” Where the list might go astray now is not with registered reports, but in writing up exploratory research. I think we need some new acronyms and norms for exploratory studies. Of course, that would assume that the field actually decides to honor honestly depicted exploratory work, which it has yet to do. If we aren’t going to publish those types of papers, we don’t need norms do we?
Fraley 2019: Building off your latest comment, I don’t see anything in here that wouldn’t apply to registered reports or (explicitly) exploratory research. In each case, it is helpful to build a need for the work, to articulate alternative perspectives on what the answer might be (even if it is exploratory), to write clearly, eliminate tangents, not make personal attacks on other scholars, etc.
Having said that, I think most authors still operate under the assumption that, if they are testing a hypothesis (in a non-competing hypotheses scenario), they have to be “right” (“As predicted, …”) in order to get other people to value their contribution. I think we lack a framework for how to write about and discuss findings that are difficult to reconcile with existing models or which do not line up with expectations.
Do you have any 2019 recommendations on how to approach this issue?
My off-the-cuff initial suggestion is that we need to find a way, especially in our Discussion sections, to get comfortable with uncertainty. A study doesn’t need to provide “clean results” to make a contribution, and not every study needs to make a definitive contribution.
Roberts 2019. I think there are really deceptively simple ways to get comfortable with uncertainty. First, we could change the norms from valuing “The Show” which says publish clever, counterintuitive ideas that lead directly to the TED-Gladwellian Industrial Complex (e.g., a book contract and B-School job or funding from a morally questionable benefactor) to getting it right. And, by getting it right, I mean honestly portraying your attempts to test ideas and reporting on those attempts regardless of their “success.” Good luck with that one.
A second deceptively simple way to grow comfortable with uncertainty is to work on important ideas that matter to people and society rather than what your advisor says is important. Who cares whether attachment is a type or continuum or whether the structure of conscientiousness includes traditionalism? What matters is whether childhood attachment really has any consequential effects on outcomes we care about—likewise for conscientiousness. Instead of asking, “Does conscientiousness matter to _____”, we could ask “How do I help more adults avoid contracting Alzheimer’s disease.” When asked that way, finding out what doesn’t work (e.g., a null effect) is just as important as finding out what does.
By the way, I just found a typo in the original text….13 years and countless readings by multiple people and it was still not “perfect.”
[1] Pat claims he only had 2 publications when he applied for our post doc. I remember 7 to 9. Needless to say, he went on to infamy by publishing at a rate during the post doc that no-one to my knowledge has matched. I’d like to take credit for that but given the fact that he continues to publish at that rate, I’m beginning to think it was Pat….
[2] This, of course, is a bit of an overstatement. As Chris Fraley points out, the judicious use of footnotes can assuage the concerns of reviewers that you failed to consider their research. By eliminating tangents, I mean getting rid of entire paragraphs that are not directly relevant to your paper.
[3] This is not to say that the motivation for a line of research should not be inspired by a negative reaction to someone or someone’s ideas. It is okay to get your underwear in a bunch over someone’s aggressive ignorance and then do something about it in your research. Just don’t write it up that way.
pigee 