Sample Sizes in Personality and Social Psychology

R. Chris Fraley

Imagine that you’re a young graduate student who has just completed a research project. You think the results are exciting and that they have the potential to advance the field in a number of ways. You would like to submit your research to a journal that has a reputation for publishing the highest caliber research in your field.

How would you know which journals are regarded for publishing high-quality research?

Traditionally, scholars and promotion committees have answered this question by referencing the citation Impact Factor (IF) of journals. But as critics of the IF have noted, citation rates per se may not reflect anything informative about the quality of empirical research. A paper can receive a large number of citations in the short run because it reports surprising, debatable, or counter-intuitive findings regardless of whether the research was conducted in a rigorous manner. In other words, the citation rate of a journal may not be particularly informative concerning the quality of the research it reports.

What would be useful is a way of indexing journal quality that is based upon the strength of the research designs used in published articles rather than the citation rate of those articles alone.

In an article recently published in PLoS ONE, Simine Vazire and I attempted to do this by ranking major journals in social-personality psychology with respect to what we call their N-pact Factors (NF)–the statistical power of the studies they publish. Statistical power is defined as the probability of detecting an effect of interest when that effect actually exists. Statistical power is relevant for judging the quality of empirical research literatures because, compared to lower powered studies, studies that are highly powered are more likely to (a) detect valid effects, (b) buffer the literature against false positives, and (c) produce findings that other researchers can replicate. Although power is certainly not the only way to evaluate the quality of empirical research, the more power a study has, the better positioned it is to provide useful information and to make robust contributions to the empirical literature.

Our analyses demonstrate that, overall, the statistical power of studies published by major journals in our field tends to be inadequate, ranging from 40% to 77% for detecting the typical kinds of effect sizes reported in social-personality psychology. Moreover, we show that there is considerable variation among journals; some journals tend to consistently publish higher power studies and have lower estimated false positive rates than others. And, importantly, we show that some journals, despite their comparatively high impact factors, publish studies that are greatly underpowered for scientific research in psychology.

We hope these rankings will help researchers and promotion committees better evaluate various journals, allow the public and the press (i.e., consumers of scientific knowledge in psychology) to have a better appreciation of the credibility of published research, and perhaps even facilitate competition among journals in a way that would improve the net quality of published research. We realize that sample size and power are not and should not be the gold standard in evaluating research But we hope that this effort will be viewed as a constructive, if incomplete, contribution to improving psychological science.

Simine wrote a nice blog post about some of the issues relevant to this work. Please check it out.

 

Advertisement