When I was a full-time academic, the Dean was calling for “a culture of evidence” to underwrite the School of Psychology. A senior colleague (from Theology) responded by calling instead for some “evidence of culture.”
The poor Dean was in a terrible predicament. The University’s funding was contingent on publications in specific journals, and we were charged with finding a way to publish in those journals, (to save the farm), hopefully by wearing the badge of empirical evidence. The University was poor, and the poor are driven to do stupid things. Or, as the head of research output later put it explicitly, “The University needs more publications in Law. If you are a psychologist, and you have publishable research on psychology, find a way to make it about psychology and the law.” I didn’t last long.
This to say that the business of scientific research and its publication is never neutral. Agendas abound. The neutral perception on reality that Bacon made the very basis of science in the Early Modern period is today regarded as truly impossible, for epistemological reasons. But I would like to pause over the political reasons that serve only to dilute that neutrality further. Psychotherapists who claim to offer “evidence-based” treatments employ a rhetorical stance that both signals a virtue and implies a distinction (i.e., I don’t practice those evidence-free treatments). It has been over a decade since research began finding that “evidence-based psychotherapies” do not outperform non-evidence based psychotherapies (e.g., Wampold et al., 2011) and we would do well to distinguish between psychotherapies that have no evidence for efficacy and those that have evidence of no efficacy. The latter constitute a very small list. When it comes to depression, standard outcome measures show effectiveness for almost anything that constitutes an intervention. In other words, the many therapies are equally able to produce symptom-change, including (famously) the therapy of going to the gym regularly. If a professional says “I practice evidence based psychotherapy for depression” they are saying almost nothing to distinguish themselves from other professionals. Our response should be “…as opposed to what?”
Symptom-change versus …what?
It can be easy to forget to ask “what is the evidence” while everyone is shouting about whether they have the evidence behind them. Symptom-change is the most common definition of effectiveness that underwrites evidence-based treatments; so “evidence-based treatment for depression” really means “treatment that will change symptoms of depression.” Let’s think about the value of that measure of effectiveness. While I believe that depressed people are right to want their depressive symptoms to change, is it a terrible mistake to accept that one criterion (symptom-change) as the only benefit worth seeking from psychotherapy—a mistake for therapists to think of offering only that, and a mistake for patients to think of considering nothing else when selecting a therapy. I practice psychodynamic therapy, and I am explicit with all my patients from the beginning, that I practise the longest and most expensive therapy known to effect symptom change. If that’s all you want, my therapy is wasteful, but if you want other measures of change too, this might be one of very few therapy-options for you. My esteemed colleague Robert King, once put it beautifully: It’s like taking fuel-efficiency as the only worthy measure of automobiles, and then promulgating the notion that everybody drive (only) the most fuel-efficient car on the road, as if there were no other parameters of value or significance offered by the varieties of car. As though someone were wrong for wanting anything more than fuel-efficiency. Our patients are there to change their lives, not for some greater scientific purpose. I mean to emphasise that the near-unanimous definition of “evidence” in the current environment is symptom-change. That’s what most psychotherapists in the “evidence-based” camp are (actually) talking about: evidence of symptom-change. Let us look now at some other important considerations of the appellation “evidence-based”.
Overview
This essay looks at some of the fundamentals of scientific activity that underwrite (or should underwrite) claims like “I practice evidence-based psychotherapy,” or “I practice the Gold Standard psychotherapy.” I will argue that even a basic understanding of research methodology will render terms like “Gold Standard” meaningless—at least they are meaningless from the scientific point of view—even if they persist as marketing slogans. I argue also that the term “evidence-based” belies a complex set of considerations.
Sometimes I fear my colleagues are saying, “just tell me if there’s any evidence, so I can write that I’m ‘evidence-based’ on my website.” Psychologists have been guilty of running home waving the verdict of an evidence-base, sometimes in disturbing neglect of what that assertion is actually founded upon if we looked in detail at the science underlying it (see This Substack for a telling example).
What we need to remember about science.
Science is a religion. That is not an original assertion1. It qualifies as a religion because it is founded upon certain assumptions that cannot be demonstrably proven. On the other hand, it differs from religion in a number of ways, the most relevant here being that it expects to get things wrong and to make mistakes. (Science asserts itself as fallible, despite some people using it as a badge to imply their own infallibility through scientific allegiance.) Part of this expectable mistake-making is not just a consequence of inevitable shortcoming and a lack of perfection, it’s also inherent in what science is, in itself. Scientific progress not only extends through time to refute what was previously cherished, it is also much less of a progressive accumulation of wisdom and more like fashion. To understand this, we can go back to the 70s and to Thomas Kuhn (whom everybody cites and nobody reads).
The paradigm
Kuhn gifted us the idea that science proceeds by revolutions, not by evolution, as many wish it could. In other words, ‘science’ is not a single, unified activity, carried out over centuries, with a steadily mounting accumulation of wisdom and insight. Rather it runs on certain lines for a while, and then there is a revolution, and it leaps from one line or trajectory to another. To me, this series of categorically independent phases best resembles fashion: There is no sound argument for why neck ties went from thin to wide in a certain year; it was simply a shift from one to the other, not a progression; wide ties are not an improvement on thin ties, they belong to a different category. Only during the phase of wide ties are they thought to be “better”; and they will again be thought inferior, given enough time. Likewise, phases of scientific activity proceed for a while before popping into new phases, without a through-line of coherent reason. Part of what changes is the very definition of what’s reasonable.
Before you think that I’m describing a shortcoming of science at particular points in time, let me say clearly that it is correct of science to make these categorical (not progressive) leaps. Kuhn called these points from which (and to which) science leaps paradigms, and the paradigm defines both the problem to be investigated and the means by which a meaningful answer can be found. The paradigm pre-defines the solution, otherwise we would not know it when we saw it. Something fundamentally wrong or meaningless in one paradigm can be the greatest scientific breakthrough in the subsequent one. Likewise, comparing Cognitive Behaviour Therapy with Psychodynamic Therapy requires some negotiating on which therapy gets to choose what the outcome measures are going to be.
So when we are talking about “evidence-based practice” we must remember to ask about the assumptions underwriting the paradigm making that declaration. (The most untrustworthy researcher is the one who thinks s/he is not operating within any paradigm.)
The method
A fundamental premise of sound Methodology includes selecting a measurement tool compatible with the data being measured. The paradigm defines the relevant data (what we are looking for), and the method must select a matching way of measuring it. This applies to literal tools as well as research designs. At the tool level, we do not use a microscope to look at the moon, no matter how fervently people argue that the microscope is an excellent scientific instrument. Likewise, at the level of research design, the double-blind randomised controlled trial (RCT) is excellent at measuring some things and utterly useless at measuring others. So comparing a double-blind RCT with a case study and proclaiming the former ipso facto “more scientific” is to proclaim the author confused about the distinction between a tool and sound methodology. You can beat a screw with a hammer, but a screwdriver gets better results. Nobody would conclude that a hammer is a better tool than a screwdriver, much less the “Golden Standard” of tools. It’s only better for nails.
So when we are talking about “evidence-based practice” we must remember to ask about which method led to which conclusion about which type of data that proclaim the practice effective.
Fundamentals of empirical research
Here is a summary known to all first year social sciences students. Empirical research needs an Independent Variable (IV) a Dependent Variable (DV) and a statistical procedure to establish a relationship between them. The IV refers to our measurement of the thing our paradigm declares the causal agent, while the DV refers to our measurement of the thing our paradigm declares the affected agent. As a simple example, if we wanted to study whether psychotherapy works, our (predicted) causal agent is the psychotherapy, (the IV) and the (predicted) effect of that agent is the outcome of the therapy (the DV). Such a study might typically have an IV of 12 sessions of psychotherapy, and a DV of self-report ratings of symptoms from the patients. There is never a “pure” test of “psychotherapy” itself, because we have to decide how we intend to measure it. We have to choose an operationalisation of psychotherapy, such as a specific number of sessions, with a specific kind of therapist, with a specific amount of experience. For example, a great deal of psychotherapy research simply uses data taken from the Student Services Clinic of whichever University the researcher works for. Likewise, there is never a direct test of whether the psychotherapy “works”, because have to decide how we intend to measure whether it works and therefore what our definition of effectiveness will be. Again, measuring symptoms in near-ubiquitous, and much of that is self-reported.
So when we are talking about “evidence-based practice” we must remember to ask about how the psychotherapy was operationalised, how the efficacy was operationalised, because if Dr Smith says she practises evidence-based therapy, some or most studies demonstrating that Dr Smith’s therapy is effective, might have measured something that looks vastly different from the therapy you will receive from Dr Smith.
Statistical procedures
Why do we need any stats? One of the main reasons we use statistics in the behavioural sciences is to translate what we measured in a manageable number of research participants into what is likely to be the case for everybody in general. We call these statistics “inferential statistics”, because we use them to take (for example) the self-reported symptom change of 30 participants in a study and infer from those data what we expect will be similar of different for the whole population. We can never know whether the 30 we tested are typical of the whole population, so we need stats (to try) to answer that question.
I taught statistics for the social sciences at university for many years, and it took some time to realise that students found Statistical Null Hypothesis Testing so difficult to understand because they came (rightly) expecting something rational. It’s logically flawed. I’m going to cover the nature of that flaw in a subsequent Substack, because it takes (at least) a whole essay to explain it. Stay tuned.
Summary
So when someone says, “I practice the Gold Standard form of psychotherapy” or “I practice only evidence-based therapy” we will (hopefully) not take that statement at face value, without curiosity about exactly what undewrites it. Open-minded skepticism is the proper stance towards such assertions.
That is also not an attack, except to the parts of us who both love science and hate religion, and who also hope that science will rescue disappointed emotional investments in religion.