The Pre-Publication Transparency Checklist: A Small Step Toward Increasing the Believability of Psychological Science
We now know that some of the well-accepted practices of psychological science do not produce reliable knowledge. For example, widely accepted but questionable research practices contribute to the fact that many of our research findings are unbelievable (that is, that one is ill-advised to revise one’s beliefs based on the reported findings). Post-hoc analyses of seemingly convincing studies have shown that some findings are too good to be true. And, a string of seminal studies have failed to replicate. These factors have come together to create a believability crisis in psychological science.
Many solutions have been proffered to address the believability crisis. These solutions have come in four general forms. First, many individuals and organizations have listed recommendations about how to make things better. Second, other organizations have set up infrastructures so that individual researchers can pre-register their studies, to document hypotheses, methods, analyses, research materials, and data so that others can reproduce published research results (Open Science Framework). Third, specific journals, such as Psychological Science, have set up pre-review confessionals of sorts to indicate the conditions under which the data were collected and analyzed. Fourth, others have created vehicles so that researchers can confess to their methodological sins after their work has been published (psychdisclosure.org). In fact, psychology should be lauded for the reform efforts it has put forward to address the believability crisis, as it is only one of many scientific fields in which the crisis is currently raging, and it is arguably doing more than many other fields.
While we fully support many of these efforts at reform, it has become clear that they leave a gaping hole through which researchers can blithely walk. People can and do ignore recommendations. Researchers can avoid pre-registering their work. Researchers can also avoid publishing in journals that require confessing one’s QRPs before review. And, published authors can avoid admitting to their questionable research practices post hoc. What this means is that research continues to be published every month in our most prestigious journals that in design and method looks indistinguishable from the research that lead to the believability crisis in the first place.
In searching for solutions to this problem, we thought that instead of relying on the good graces of individual researchers to change their own behavior, or waiting for the slow pace of institutional change (e.g., journals to follow Psychological Science’s lead), that it might be productive to provide a tool that could be used by everyone, right now. So what are we proposing? We propose a set of questions that all researchers should be able to answer pre-publication in the review process—the Pre-Publication Transparency Checklist (PPTC). Who should use these questions? Reviewers. Reviewers are free to ask any question they want, as many of us can attest to. There is nothing stopping researchers from holding other researchers accountable. The goal of these questions is to get even those unwilling to pre- or post-register their research process to cough up background information on how they conducted their research and the extent to which their results are “fragile” or “robust”. The questions are inspired by the changes recommended by many different groups and would hopefully help to improve the believability of the research by making authors describe the conditions under which the research was conducted before their paper is accepted for publication.
The Pre-Publication Transparency Checklist
- How many studies were run in conceptualizing and preparing the set of studies reported in this paper?
- How many studies were run under the original IRB proposal?
- How many “pilot” studies were run in support of the reported studies?
- If an experiment was run, how many conditions were included in the original study and were all of these conditions included in the current manuscript?a
- Was any attempt made to directly replicate any of the studies reported in this paper?
- Would you be willing to report the direct replications as an on-line appendix?
- Note: In some fields it is common to replicate research but not report the efforts.
- Note: Some studies are difficult to replicate (e.g., longitudinal, costly, technologically intense).
- Approximately how many outcome measures were assessed in each study?a
- How many of these outcome measures were intended for this study?
- How many outcome measures were analyzed for each study?
- Do all of the conceptually and empirically related DVs show the same pattern of effects?
- In any of the studies presented, were the data analyzed as they were being collected (i.e., peeked at)?
- If it was “peeked” at, was there an effort to address the potential increase in the Type I error rate that results from peeking, such as conducting direct replications or using Bayesian estimation approaches?
- Note: The goal is not necessarily to eliminate p-hacking but to make sure our findings are replicable despite p-hacking (see Asendorpf, et al, 2012 for a discussion).
- What was the logic behind the sample sizes used for each study?a
- Was a power analysis performed in the absence of pilot data?
- Was an effect size estimate made on the initial work and used for power estimates of subsequent studies whether they were direct or conceptual replications?
- Were any participants assessed and their data not included in the final report of each study?a
- What was the rationale for not including these participants?
- Do all of the co-authors of the study have access to the data?
- Can all of the co-authors of the study reproduce the analyses?
- If not, why and who can?
- Note: It is common for statistical experts to lend a helping hand so it is not necessarily bad that all the authors cannot reproduce the analyses. But, it is important to know who can and cannot reproduce the analyses for future efforts to reproduce the results.
10. Were there any attempts to test whether the results were robust across analytical models and different sets of control variables?
- If the results do not replicate across models was this factored into the alpha level (multiple tests of significance?)?
11. Approximately how many different analytical approaches were used?
- Were alternative ways of analyzing the data considered and tested?
- Note: it is common to try different variants of the general linear model (ANOVA, ANCOVA, regression, HLM, SEM). It would be important to know whether the results replicate across the various ways of analyzing the data.
So, as we noted above, these questions could be asked of researchers when they present their work for review. Ideally, the answers to these questions would become part of the reported methods of every paper submitted, possibly as an on-line appendix. If reviewers asked these types of questions of every paper that was submitted, that in itself would change the publication incentive structure quite dramatically.
A second way that the Pre-Publication Transparency Checklist could be used is by editors of journals other than Psychological Science. Like reviewers, editors could ask authors to simply answer each of these questions along with their submission. There is no reason why Psychological Science should go it alone with this type of questioning. The effort to answer these questions is minor—far less, for example, than the time taken to complete the typical IRB form. Again, if editors used the PPTC, which they should be free to do today, we could be on our way to a better more substantial body of research on which to base our future scientific efforts.
Given the heterogeneity of reactions to the believability crisis in psychology, we do not foresee the answers to these questions being “right” or “wrong” so much as providing information that other researchers can use to determine whether they personally would want further follow up before concluding that the research was reliable. But, of course, like the traditional methods we use in psychological science which rely on transparency, accuracy, and honesty, the answers will only be as good as they are truthful.
We are also sympathetic to the point that many of the questions will be difficult to answer and that many questions will not apply to different types of research. That is okay. The goal is not an exacting account of every behavior and decision made on the way to a finished publication. The goal is to provide background information to help determine how robust or delicate the findings may be. For example, if dozens of studies were run looking at scores of outcomes and only a few of the studies and outcomes were reported, then other researchers may not want to attempt to build on the findings before directly replicating the results themselves. Similarly, if multiple analyses were conducted and only the statistically significant ones reported, then other researchers would likewise be cautious when following up on the findings.
As noted above, the PPTC would not be necessary if researchers pre-registered their studies, posted their materials and data on-line, and were transparent with the description of their methods. But, given the obvious fact that not every researcher is going to pre-register their materials, the Pre-Publication Transparency Checklist provides a means through which reviewers and editors can get these individuals to provide desperately needed information on which to judge the robustness or fragility of the reported findings in any submitted manuscript.
Brent W. Roberts, University of Illinois, Urbana-Champaign
Daniel Hart, Rutgers University
 These accords include those proposed for pre-review at Psychological Science, post-publication disclosure at Psychdislosure.org and as part of the 21 Word Solution. Those questions marked with an “a” are similar in content to the questions found on existing systems.