When I was studying evidence based programmes, I kept coming across an interesting phenomenon. It is summarised in the following graph.
Fidelity matters in evidence based programmes. If they are not delivered as planned, then they don’t get the results. That is evident from the above slide.
It was produced by Steve Aos and Robert Barnowski, better known as Barney, at the Washington State Institute of Public Policy. The Institute had advised the Government to invest in Functional Family Therapy, or FFT for shorthand. But the results were less than they hoped. So they set about finding out why.
The vertical ‘Y’ axis measures an outcome, the rate of re-offending. A column marked ‘C’ indicates the bar. C means ‘control’, a group of young people in trouble who didn’t get FFT.
The recidivism rates for 25 practitioners delivering FFT are then reported in a series of columns numbered 1 to 25. A score above the bar indicates the practitioner is doing worse than control, a score below the line indicates they are doing better than control.
The practitioners are grouped according to their fidelity to the model. Numbers 20 to 25 deliver FFT by the book, the way programme developers Jim and Tom would have them deliver it. Numbers 1 to 7 are way off the mark. The others are somewhere in between.
Those practitioners delivering FFT with fidelity get good results, much better than the control group. Those practitioners messing it up get worse results, worse indeed than had there been no therapy at all.
But look again within the groups. Practitioners 1, 2 and 3. They get great results despite the fact that they fail on the FFT fidelity measures. And practitioner 25, a devotee of the FFT process gets unfavourable results.
What is going on? The characteristics of the therapist could be one explanation. Maybe practitioners 1, 2 and 3 are super smart, so smart in fact they know how to cut corners on the FFT method and still get excellent results.
Another hypothesis is that the results are explained by the quality of relationship between therapist and young person.
These were the propositions we tested when we looked at Functional Family Therapy through the lens of a learning machine.