Science or law: Choose your career

I recently saw an article by an astute reporter that described one of our colleagues as a researcher who “…has made a career out of finding data….”

Finding data.

What a lush expression.  In this case, as it seems always to be the case, the researcher had a knack for finding data that supported his or her theory. On the positive side of the ledger, “finding data” denotes the intrepid explorer who discovers a hidden oasis or the wonder that comes with a NASA probe that unlocks long lost secrets on Mars.

On the negative side of the ledger, “finding data” alludes to researchers who will hunt down findings that confirm their theories and ignore data that do not. I remember coming across this phenomenon for the first time as a graduate student, when a faculty member asked whether any of us could “find some data to support X”.  I thought it was an odd request.  I thought in science one tested ideas rather than hunted down confirming data and ignored disconfirming data.

Of course, “finding data” is an all too common practice in psychology.  Given the fact that 92% of our published findings are statistically significant and that it is common practice to suppress null findings, it strikes me that the enterprise of psychological science has defaulted to the task of finding data.  One needs only to have an idea, say that ESP is real, and, given enough time and effort, the supporting data can be found.  Given our typical power (50%) and the Type 1 error rate (at least 50% according to some), the job is not too tough.  One only has to run a few underpowered studies, with a few questionable research practices thrown in and the data will be found.  Of course, you will have to ignore the null findings.  But, that apparently is easy to do because as one of our esteemed colleagues wrote recently “everyone does it”—“it” meaning throw null effects away.

There are other careers and jobs that call for a similar approach—pundits and lawyers.  The job of Fox or MSNBC pundits is not to report the data as it is, but to find the data that supports their preconceived notion of how the world works.  Similarly, good lawyers don’t necessarily seek the truth, but rather the data that benefits their client the most.  It appears that we have become a field of lawyers who diligently defend our clients, which happen to be our ideas.

To the extent that this portrait is true, it leads to some painful implications.  Are psychological researchers just poorly paid lawyers?  I mean, most of us didn’t get into this career for the money, but if we are going to do soulless lawyer-like work, why not make the same dough as the corporate lawyers do?  Of course, given our value system psychologists would most likely be public defenders so maybe asking for more money would be wrong.  But consider the fact that law school only lasts three years.  The current timeline for a psychology Ph.D. seems to be five years minimum, sometimes six, with post doc years to boot.  Do you mean to tell me that I could have simply gone to law school instead of a Ph.D. program and been done in half the time and compensated far better? Maybe it is not too late to switch.

What’s so bad about being a lawyer?

Nothing. Really. I have no prejudice against lawyers.  Practicing law can be noble and rewarding.  And, like many careers it can be a complete drag.  It is work after all.

And, there are similarities between science and law.  Both professions and the professionals therein pursue certain ideas, often relentlessly.  Many defendants are grateful for the relentless pursuit of justice practiced by their lawyers.  Similarly, many ideas in science would not have been discovered without herculean, single-minded focus, combined with dogmatic persistence.

Then again, there are the lawyers who defend mob bosses, tobacco firms, or Big Oil.  None of us would want to be like them, right?

In an ideal world, there is one, very large difference between practicing law and science.  At some point, scientists are supposed to use data as the arbiter of truth.  That is to say, at some point we must not only entertain the possibility that our all-consuming idea is wrong, but also firmly conclude that it is incorrect.  I had an economist friend who pursued the idea that affirmative action programs were economically detrimental to beneficiaries of those programs.  He eventually determined that his idea was wrong.  Admittedly, it took him ten years to come to that position, but he at least admitted it.  Changing one’s mind like this would be akin to the tobacco lawyer suddenly admitting in the middle of a trial that smoking cigarettes really is bad for you.  This doesn’t happen exactly because these lawyers are paid big money to ignore the truth and defend their clients despite these truths.

This means that the difference between being an advocate and a scientist lies almost solely on the integrity of our data and our response to that data.  If our data are flawed, then we can act like scientists and really be no better than a pundit or propagandist.  If we hide our “bad” data (e.g., non-significant findings), we are likewise practicing a less than noble form of law—we are ambulance chasers or tobacco lawyers.  If we don’t change our minds as a result of data that disconfirms our most closely held ideas, we are again, advocates not scientists.

The bottom line is that many of us are being lawyers/pundits with our research.  We drop studies, ignore problematic data, squeeze numbers out of analyses, and use a variety of techniques in order to present the best possible case for our idea.  This is the fundamental problem with the p-hacking craze going on in many sciences, including psychology.  We are not truly testing ideas but advocating for them, and often we are really advocating only for our careers when we do this.  Just because we defend seemingly noble ideas, such as social justice, doesn’t make the work any different.  If we only pay attention to the data that supports our client, then we aren’t doing science.

What should we do?

Many, many earnest recommendations have been made to date and I will not reiterate or contradict any of the missives describing optimal publishing practices and the like.  What I think has been missing from the dialogue is a clear case made for us to change our attitudes, not only our publishing practices and research behavior.  So, the recommendations below go to that effect.

First, and most ironically, I believe we need to be legalistic in our approach to our research.  That is, we need to be judge, jury, prosecutor, and defense council of our own ideas.  As noted elsewhere, psychology is a field that only confirms ideas (and only in data that reveals a statistically significant finding).  Alternatively, we need to do more to prosecute our own research before we hoist it on the world.  The economists call this testing the robustness of a finding.  Instead of just portraying the optimal finding (e.g., the statistically significant one), we need to present what happens when we put our own finding to the test. What happens to our finding when statistical assumptions are relaxed or restricted, when different control variables are included, or different dependent variables are predicted? If your finding falls apart when you conduct a slightly different statistical approach, use a new DV that correlates .75 with your preferred DV, or run the study in a sample of Maine versus Massachusetts undergraduates, do we really want to endorse that finding as true? No.

Second, we need value direct replication.  I get a lot of push back on this argument, and that push back deserves its own essay (later Chris).  But, given how prevalent p-hacking is in our field, we need an outbreak of direct replications and healthy skepticism of “conceptual” replications.  For example, those who argue that they value and would prefer a 4-study paper with 3 conceptual replications, have to assume that p-hacking is not prevalent.  Unfortunately, p-hacking is wide-spread (see quote about “everyone does it”).  At this juncture, a 4-study paper with 3 conceptual replications using some perversely nonsensical range of sample sizes for each study (from 30 to 300) screams out “I P-HACKED!”  Combining conceptual replications with simple, direct replications is not difficult and is really hard to argue against in light of how difficult it is to replicate our findings.

Third, we need to walk back our endorsement and valuing of brief journal formats found in journals like Science, Psychological Science, and Social Psychological and Personality Science.  This is not because short reports are evil per se, but because they promote a lax attitude toward research that exacerbates our problematic methodological peccadillos.  I must admit that I used to believe that we needed more outlets for our science and I loved the short report format.  I was wrong.  We made a huge mistake—and I was part of that mistake—in promoting quick reports with formats so brief and review processes so quick that we end up promoting bad research practices. At JPSP after all, you have to “find” 3 or 4 statistically significant effects to have a chance at publication.  At short report journal outlets, you only have to “find” one such study to get published, especially if the finding is “breathtaking.”  Thus, we promote even less reliable research in top journals in an effort to garner better press. In some ideal world, these formats would not be a problem.  In the context of pervasive p-hacking, short, often single-shot studies are a problem. We have inadvertently promoted a “quick-and-dirty” attitude toward our research efforts, making it even easier to infuse our field with unreliable findings.  Until we have our methodological house in order, we should reconsider our love of the short report and the short report outlet.

Fourth, we need to be less enamored with numbers and more impressed with quality.  Building a lengthy CV is not that difficult.  All one needs to do is put together a highly motivated team of graduate and undergraduate assistants to churn through dozens of studies per year.  Then, combine that type of cartel with a willingness to ignore the null effects or practice some basic QRPs and you will have at least 4 JPSP/Psych Science-like multiple study papers completed per year.  If you are willing to work the “messier” studies in lower-tier journals you are well on your way to an award.  Even better, publish unreplicable, provocative findings and get into a nasty argument with colleagues about your findings. Then, your CV explodes with the profusion of tit-for-tat publications that come with the controversy.  In contrast, if we evaluate researchers based on the ideas they have and how they go about testing them, rather than their ability to churn the system to discover statistical significance, we might actually do more to clean up our methodological mess than any pre-registration registry could ever achieve.

The obvious ramification of adopting a more skeptical attitude toward our own research would be to slow things down.  As Michael Kraus has argued, why not adopt a “slow research” movement akin to the slow food movement?  If rumors are true, over half of our research findings cannot be directly replicated.  That means we are wasting a lot of time and energy writing, reviewing, reading, and believing arguments that are, well, just that, arguments—arguments that look like they have supporting data, but are really fiction.  While I appreciate a good argument and impassioned punditry, science is not supposed to be an opinion echo chamber.  It is supposed to be a field dedicated to creating knowledge.  Unlike a baseless argument, knowledge stands up to cross-examination and direct replication.

Brent W. Roberts

Advertisements
This entry was posted in Uncategorized. Bookmark the permalink.

17 Responses to Science or law: Choose your career

  1. David Funder says:

    This is an interesting, important post but it rests on a dichotomy between studies that “work” and studies that “don’t work” that, I’m starting to realize, can lead us seriously astray. Science is not about verdicts (or at least it shouldn’t be). I became aware of this point point thanks to paper in the new PPS by the late Bill McGuire (rescued from his files by John Jost). McGuire says (nearly) all hypotheses and their opposites are true, sometimes. So one way to generate interesting research is to take a bit of common-sense knowledge and design a study to find the reverse. Of course, the study won’t work at first. Common sense is correct under most circumstances. So you tweak, rewrite, reframe, redesign, and rerun the study until it does work, meaning (in this case) that you have found a context in which the common-sense expectation is violated.

    McGuire would say: so far, so good. You have found an interesting example of how something surprising can happen. Reported as such, and in the context of the (many) circumstances where the surprising effect does not occur, the finding could be an useful addition to psychological knowledge.

    But this is where things typically go very wrong. Instead of saying “here’s an interesting context in which the common sense expectation is not fulfilled,” the paper trumpets the “counter-intuitive” finding that “common sense is wrong!” The exception to the rule is presented as if it were a general principle.

    The world then divides into two camps. In one camp, the author and his/her acolytes run more studies, tweaking away until they find more contexts in which the common sense expectation is incorrect. Conceptual replications! The other camp gets its members from two sources. The stereotype is that cynical individuals don’t believe the finding and so seek to undermine it. I suspect more common is that would-be acolytes, wishing to jump on the exciting bandwagon of a new counter-intuitive effect, try their own studies, but don’t succeed (or don’t try very hard) to tweak them into working, and then become disbelievers. (Remember, it took a lot of tweaking to get the effect to work in the first place.) Either way, the two camps now can do battle about whether the counter-intuitive effect is valid or not. The first camp continues to produce studies that show their effect; the second camp produces study after study that show nothing of the sort. The tone of the discussion gets nasty. Members of the first camp start to sound like Richard Nixon. Members of the second camp start to sound like Inspector Javert.

    All of this could have been avoided if we had started with McGuire’s recognition that just about every hypothesis and its opposite is true, under some circumstances. Therefore, rather than trying to adjudicate whether a hypothesis is “true” or “false,” we should seek to establish WHEN a hypothesis holds and does not hold. To phrase this point another way, the answer to ANY question in psychology is “it depends.” Our science should be about answering the further question, “depends on what?”

    ps:this comment paraphrases my own interpretation of McGuire’s main point, but read it for yourself at http://pps.sagepub.com/content/8/4/414.full.pdf+html.

    • Katie Corker says:

      I see your point, David, that “rather than trying to adjudicate whether a hypothesis is ‘true’ or ‘false,’ we should seek to establish WHEN a hypothesis holds and does not hold,” but what of Meehl’s crud factor? Meehl seems to agree with McGuire that all effects “exist” at some level, but Meehl insists that the question of *existence* is less important than the question of *size*. I agree with Meehl that the question of “Does it exist?” can be more fruitfully framed as “How big is it?” And, to me, the question of “How big is it?” is a much more important question than “On what does it depend?” The former is our first responsibility; the latter question can only be answered once the former is.

  2. pigee says:

    David, there are two fundamental issues with McGuire’s position. First, the robust alternative hypothesis is that “all hypotheses and their opposites are true, sometimes” because McGuire worked in a field that ran poorly designed research. As noted in the blog post and repeated many times in other places, psychology in general, and experimental psychology in particular, has a bad habit of running low powered studies. With the probability of finding an effect at 50% (the average power in psychological research) it is quite easy to find the “opposite” of a hypothesized effect simply through sampling error. In fact, it is more probable to find conflicting findings by running repeated low powered studies. Thus, McGuire’s position may be correct, but it has nothing to do with the veracity of one’s hypotheses and is simply an early indictment of conducting poorly designed research. Give us ten years of running well designed studies and let’s see if we come to the same conclusion.

    Second, if McGuire’s position is true that “the answer to any question in psychology is ‘it depends'”, then all effects in psychology are moderator effects. If this is true, then since the time of McGuire’s paper all experimental psychologists should have been designing their studies to detect moderator effects. Following from Cohen’s work, that would mean the average sample size in JPSP would be around 400. I suspect you don’t have to work very hard to find out that the 400 person experimental study in JPSP is the anomaly and not the norm.

  3. rcfraley says:

    David, I think the researcher in your story took a “turn for the worse” right from the start.

    The researcher made up his or her mind from the beginning to find data that support a specific (and counter-intuitive) idea. When one is trying to “find data” that corroborates a specific idea, one is behaving more like a pundit than a scientist.

    The problem, as Brent points out nicely, is that it is possible to find data to support any hypothesis in psychology–intuitive or counter-intuitive. And, if such data are not readily available, all you need to do is run a few small sample studies with a variety of QRPs to find what you’re looking for.

    Is this the way we want to do science in psychology? It isn’t my preference. But this is, in fact, the kind of science that we tend to reward in social and personality psychology.

    Even if we acknowledge that a preferable way to move forward is to acknowledge that the magnitude of various associations may depend on other factors, our work is still more akin to “idea advocacy” than science until we change some basic practices in the field.

    Looks like Brent chimed in before I did. Our comments might overlap since we tend to chat a lot.

  4. David Funder says:

    I accept all the above comments as “friendly amendments” to what I said. And to expand slightly:
    1. McGuire (or maybe just my summary) risks confusing stochastic variation (or just plan crud) with moderator variables. So the point about “it depends” is valid only in the case where a reliable “exception to the rule” is found.
    2. Despite the proliferation of underpowered studies with doubtful findings that Brent notes, I do think there are some examples of that. For example, the errors-and-biases literature has developed several robust demonstrations of how people reason erroneously. But most of these demonstrations don’t work if you change them even a little bit — people reason correctly unless you work very hard to fool them. Errors are exceptions to the rule. So while these studies do demonstrate specific circumstances where reasoning may go astray, and are useful to that (limited) extent, their frequently-advertised meta-message, that people are generally incompetent, is dangerously misleading.
    3. I completely agree with the principle that we need to assess the size rather than “existence” of effects. However, “existence” can be redefined as “large enough to matter.” Which raises the interesting question: how large of an effect is large enough to matter? While I don’t think we have full answer to this question, I’m pretty sure it begins with “it depends.”

  5. Brian Clark says:

    Great ideas and I agree with most of them, but something is bugging me and that bug has spurred me to comment for the first time since I stared reading your blog at the beginning of this past academic year.

    You write, “It appears that we have become a field of lawyers who diligently defend our clients, which happen to be our ideas” and much of the rest of the post also reads like it was different in the past and we’ve since lost our way, fallen from grace as it were. Well, was it different? Were there actually good ol’ days when psychologists were “real” scientists? Or, is this lament about a past that never existed? In other words, is your argument that we ought to return to being as we once were or is it that we ought to start being something we’ve never been? My reading of the history of psychology is that it’s pretty much the latter with a lot of lip service to and the occasional genuine effort at doing some idealized version of science, but I’m very curious about your take (and that of others) is on the matter.

    I’m curious a) because I’m a noob, not a vet, so maybe there’s something I’m not grasping about our field, and b) because if these good ol’ days did exist, then there’s at least a partial model of what we presumably ought to be doing buried somewhere in our scattered scientific record. If they don’t exist, then we’re stuck with trying to be something we’ve never been and don’t know how to be. And that’s a hard thing to think about doing, not to mention do.

    • Aaron Weidman says:

      Interesting question Brian! This reminds me of some old literature I recently reviewed which was formative in the development of theory regarding Need for Achievement. Many of these studies examined complex hypotheses that involved moderation (e.g., high achievers will persist more when tasks are easy or moderately difficult, whereas low achievers will persist more at extremely difficult, or unsolvable, tasks), and yet frequently employed small sample sizes such that the number of individuals per “cell” (e.g., high achievers getting a difficult task) was frequently less than 30, and sometimes even less than 10! Importantly, these papers were often published in top journals (JPSP, or prior to that, Journal of Abnormal and Social Psychology), and many of them still get cited today when authors discuss foundational principles of achievement motivation. So, that type of research seems to indicate, like Brent wrote in response to your question, that many fields have improved substantially in important areas related to replicability power, etc.

      If you’re interested, the following paper is an example of what I’m talking about, and David McClelland discusses studies with similar issues (studies by giants in the field such as Heckhausen and Atkinson) in Chapter 6 of his 1987 book.

      Feather, N. T., (1961). The relationship of persistence at a task to expectation of success and achievement-related motives. Journal of Abnormal and Social Psychology, 63, 552-561.

      McClelland, D. C. (1987). Human motivation. Cambridge, UK: Cambridge University Press.

  6. pigee says:

    Brian,
    Your point is well taken. I have portrayed things incorrectly. The past was not better. If anything, we are doing much better than before–well, some fields are. In personality psychology, we now seem to have more high powered studies than before. On the other hand, there remains an entrenched group unwilling to acknowledge the realities of running multiple group interaction studies and the power needs therein. I wouldn’t worry too much about the necessity of trying something “we’ve never been and don’t know how to be.” Running more participants in our research is well within our skill set. It just means working a little harder and publishing a little less. I think we can handle that.

    Brent

  7. Kevin says:

    This is basically a call for intellectual honesty. In my graduate school training, I noticed a more insidious challenge to intellectual honesty. In my science communication course, we were trained to critique published articles. Our professors taught us to find and to emphasize the most inconsequential flaws in published research and blow them up so large as to to discredit the research. At first I thought there were just being old and crotchety. But then I realized that tearing down your opponent’s research is a valuable skill to have when you are trying to discredit your competition and win grants. I thought that this kind of training was limited to law school. I was wrong. The most sickening part of the training was the insistence that this sort of criticism was “good for science” when it is only good for the researcher who is unfairly discrediting his/her opposition. What’s good for science is pointing out errors and ascribing the amount of harm proportionally to the research. At the end of the class I felt like a nihilist, and that everyone else’s research is crap. To rise to the top in this system you have to be ruthless, self-serving, and intellectually dishonest. Is it any surprise then that scientific fraud is more and more commonly being revealed among researchers at the top of their fields. What this article emphasizes is important,, but the problem goes much deeper than just cherry-picking of data points. Thought I do appreciate your noticing how scientists are adopting the nihilistic mentality of lawyers.

  8. Alexa says:

    I think it may be the case that several issues surrounding the robustness of an effect could be clarified by deciding which level of resolution is useful when we’re talking about an “effect” (or, in the spirit of moderation, which levels of resolution are useful in which contexts). It’s possible to talk about effects at a very low level of resolution (e.g. the relationship between agreeableness and prosocial behavior) or at a very high level of resolution (e.g. the relationship between the agreeableness subscale of the BFI and charitable donations amongst freshman college students at University X). This seems like it could be relevant to the question of whether it’s more worthwhile to establish the size of an effect or its moderators. Establishing the size of an effect seems important and answerable when we talk about effects in high-resolution terms, but when we move to lower levels of resolution this becomes more difficult. How do we come up with a precise estimate of effect size when alternative measures, populations, etc. could potentially produce widely varying results? (Perhaps the answer to this is that, at lower levels of resolution we need to tolerate effect size estimates with wider confidence intervals). On the other hand, the question about moderators seems very important and answerable at high levels of resolution, but starts to become meaningless when we define effects in very concrete terms (e.g. it becomes meaningless to ask “What would happen with another measure of agreeableness” if the specific measure is part of our definition of the effect). So, it may be the case that in contexts where we’re interested in the effect at an abstract level we would pursue the question of moderators, and when we’re interested in the effect at a concrete level we would pursue the question of effect size.

    The issue of resolution also comes up, I think, when discussing conceptual vs. direct replications. Although I am in complete agreement that more direct replications have several advantages over more conceptual replications, treating this as a dichotomy may obscure some of the relevant issues. If we’re being overly literal, a completely direct replication is never possible because it would be impossible to replicate all elements of a study. Put another way, there is always something that one can cling to as a moderator when attempting to determine the implications of a failed replication (e.g. it depends on the region of the country, it depends on the computers you use, etc.). Again, it may be the case that the degree of information provided by more or less direct replications could depend on whether or not we’re interested in the effect at a high or low level of resolution. To me it doesn’t seem clear that either one of these levels would be consistently more useful than the other, but it might resolve some apparent differences of opinion to consider the distinction.

    • Kevin says:

      Hoo boy. This is the classic response of the scientist who cannot possibly believe that something in science research is not a technical problem. The interpretation of data is subject to the mind of the scientists, and when that mind also desperately wants success and access to grants, corruption is inevitable. Grant money has decreased, competition has increased, and naturally, intellectual dishonesty and outright malfeasance have risen with these trends. This problem has nothing to do with the minutiae of statistics.

  9. pigee says:

    Alexa,

    Thanks for the thoughtful query about effect sizes. I have seen your thoughts echoed by many researchers and I’ll try and do them justice.

    First, let me try and decouple “effect size” from “high resolution” or “low resolution”. Similarly, there is a common perspective among my experimental colleagues that effect sizes are for “applied issues.” Instead, I would invite you and others to think of effect sizes like p-values. We all seem to love p-values (okay, I’ll be honest, I detest them), or at least accept that they are a widely shared index of something we agree to be important. Effect sizes are the same. Every study will have one or more even if they are not reported. Most importantly, like p-values we really, really need to start taking responsibility for our effect sizes.

    Let’s use a good example of a “high resolution” finding that is entirely artificial–Harlow’s studies of the cloth and wire monkeys. He reported a statistically significant finding in terms of the baby monkey’s preference for the cloth and/or wire “mother”. No-one would argue that this study is applicable to the real world and one could therefore argue that you were simply interested in whether there was an effect, not the size of the effect (out of curiosity, has anyone ever computed the effect size for that study?). That would be an unfortunate conclusion for two reasons.

    1. Every researcher really needs to take responsibility for their effect sizes, even in “high resolution” or “artificial” circumstances because of the fact that other scientists will come along behind them and try and replicate and extend their work. This is how science is supposed to proceed, isn’t it? And, if some intrepid researcher, say a graduate student or assistant professor, tries to both directly replicate and extend a given study then they will want and need to know the effect size so they can know to what extent they did replicate the study. Of course, because we p-hack so much these unfortunate souls who follow up on any given study often find nothing and waste a year or so of their precious research lives. At the very least, you should report and care about effect sizes because of future research that builds on your finding.

    2. The second reason one wants to know about any given effect size is that we know a lot about effect sizes and that information can be used to judge the likelihood of a finding. In fact, I know of three meta-analyses that show that the average effect in social, personality, and organizational sciences is a d of .4. That means if you run a study with 40 subjects and find a d of 1.2 you have done something remarkable. You’ve found an effect size that is 2 standard deviations larger than the average, which means you are probably wrong. Combine this with the fact that a large proportion of our studies are purportedly studying “subtle” effects or moderators effects, which by definition, should be modest in size, and you get a nice indirect indicator of p-hacking. Ignore effect sizes at your own risk.

    I hope that is clear. It is really important that psychological scientists get on the same page when it comes to effect sizes. The continued misuse of p-values and willing refusal to take responsibility for our effect sizes will be an impediment to psychology actually being a science. This is not to say that effect sizes are a panacea, but ignoring them comes at a serious cost.

    In terms of your last point, I will be brief and simple. If you can’t replicate an effect directly should you really try to conceptually replicate it? Why waste the time?

  10. Dietmar says:

    Very good read! Many thanks for sharing. It definitely reflects my own socialization as a psychologist in a (german) university. As a scientific aide at an international highly valued institution I happened to get acquainted to all those “dirty” tricks to force an experimental outcome to be “significant”.

    We applied logarithms to the raw data, kicked out runaways of both sides of the distribution to avoid the spoiling of mean scores and even performed reiterations of experimental studies until the sample of experimentees relievingly condescended to fit into the hypothesis. All the morality of undertaking a “clean” and sincere way of science we learned in the first terms fell by the wayside.

    All those frustrating and sobering experiences happened for the sake and support of scientific careers – the employment of our bosses at this institution was supposed to be a springboard for a university-chair. Any non-significant outcome would´ve spoiled the opportunity to get published in a prestigious scientific magazine and thus would´ve spoiled the attaining of a professional summit.

    Needless to say, that the experimentees mostly were picked from the student body, young people who gladly accepted to make some bucks, but hardly represented the basic population the study pretended to target.

    In sum and at the bottom line, I´m happy to be a psychotherapist and telling my patients once in a while, that a “non-significant” outcome can be a valuable one as well.

  11. Alexa Tullett says:

    Hoo boy. I may have accidentally conveyed a casualness about effect sizes, direct replications, and the need for change in our field in my last post. I think it may be worth clarifying, partly because I think there are people who would identify with things that I said but who also staunchly believe in the importance of effect sizes, direct replications, and change in our field, and partly because I’m using my full name this time.

    First, I should make it clear that I think effect sizes are always important, and agree that it’s desirable that our field comes to a consensus on that. To me this doesn’t seem inconsistent with the idea that effect size gets less precise (and maybe less useful) when we’re talking about an effect on a very abstract level, and that it’s in these cases where the question of moderators becomes interesting.

    Second, I should emphasize that I think direct replications are the most useful tool we have in evaluating the truth about an effect. If an effect can’t be directly replicated, I agree it would be a misguided waste of time to attempt a conceptual replication. But, again, this doesn’t seem inconsistent with the idea that when we are interested in an effect in a very general form, conceptual replications are helpful in telling us how general we can be.

    I think perhaps Kevin was getting at the idea that discussing these technical distinctions clouds the broader issue of problems with integrity in our field. I agree wholeheartedly with his comments about how the current system creates motives and incentives that encourage even well-intentioned scientists to engage in questionable practices, and that hold our field back in reaching its potential as a predictive science. Nevertheless, if we’re going to persuade people of the importance of these issues I think there needs to be room to consider the merits of moderators and conceptual replications without being a traitor to effect sizes and direct replications.

  12. Pingback: The Berkeley Science Review » The Psychology of the “Psychology Isn’t a Science” Argument

  13. Pingback: The psychology of the “psychology isn’t a science” argument | Improving Vocational Education and Training

  14. Pingback: Science or law: Choose your career | pigee | A Dreamy Sparrow

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s