This is a guest post by Lee Jussim. It was originally posted as a comment to the Beginning of History Effect, but it seemed too important to leave as a comment. It has been slightly edited to help it stand alone.
Effect sizes may matter in some but not all situations, and reasonable people may disagree.
This post is about one class of situations where: 1) They clearly do matter; and 2) They are largely ignored. That situation: When scientific articles, theories, writing makes explicit or implicit claims about the relative power of various phenomena (see also David F’s comments on ordinal effect sizes).
If you DO NOT care about effect sizes, that is fine. But, then, please do not make claims about the “unbearable automaticity of being.” I suppose automaticity could be an itsy bitsy teenie weenie effect size that is unbearable (like a splinter of glass in your foot), but that is not my reading of those claims. And it is not just about absolute effect sizes. It would be about the relative effects of conscious versus unconscious processes, something almost never compared empirically.
If you do not care about relative effect sizes, please do not declare that “social beliefs may create reality more than reality create social beliefs” (or the equivalent) as have lots of social psychologists.
If you do not care about at least relative effect sizes, please do not declare stereotypes to be some extraordinarily difficult-to-override “default” basis of person perception and argue that only under extraordinary conditions do people rely on individuating information (relative effect sizes of stereotypes versus individuating information in person perception are r’s=.10, .70, respectively).
If you do not care about at least relative effect sizes, please do not make claims about error and bias dominating social perception, without comparing such effects to accuracy, agreement, and rationality.
If one is making claims about the power and pervasiveness of some phenomenon — which social psychologists apparently often seem to want to do — one needs effect sizes.
Two concrete examples:
Rosenhan’s famous “being sane in insane places” study:
CLAIMED that the “insane were indistinguishable from the sane.” The diagnostic label was supposedly extraordinarily powerful. In fact, his own data showed that the psychiatrists and staff were over 90% accurate in their judgments.
Hastorf & Cantril’s famous “they saw a game” study:
This was interpreted both by the original authors and by pretty much everyone who has ever cited their study thereafter as demonstrating the power of subjective, “constructive” processes in social perception. It actually found far — and I do mean FAR — more evidence of agreement than of bias.
Both of these examples — and many more — can be found in my book (you can get the first chapter, and abstracts and excerpts here:http://www.rci.rutgers.edu/~jussim/TOC.html
(it is very expensive, so, if you are interested, I cannot in good faith recommend buying it, but there is always the library).
If (and I mean this metaphorically, to refer to all subsequent social psychological research, and not just these two studies) all Rosenhan and Hastorf & Cantril want to claim is “bias happens” then they do not need effect sizes. If they want to claim that labels and ingroup biases dominate perception and judgment — which they seemed very much to want to do — they need not only an effect size, but to compare effect sizes for bias to those for accuracy, agreement, rationality, and unbiased responding.