The shared ‘reading’ included the Talking Politics podcast with David Spiegelhalter about super-forecasting. This is the task of predicting very hard events.
The work generates three opportunities for thinking about what comes next in the relationship between evidence and policy, in the third era as Berwick puts it:
Can we predict?
What are the rules of predicting?
How do we know whether something is true?
In this piece, I reflect on the first question.
There follows a two by two table. It splits the world into four tidy segments. Across the top is a division between those who see the world as complex. On the side is the separation between those who think prediction is possible (if hard) and those who believe (how can they know) that the social world is unpredictable. Let’s look at each of the segments in turn, starting with the bottom left corner and going anti-clockwise to the top left.
1. Complex and unpredictable
Two leading thinkers stand out here. First Hayek, the champion of the free-market, the progenitor of the neo-liberalism, a Nobel prize winner whose acolytes included several prime ministers and presidents. He may be very unpopular in some circles, but his ideas have shaped the world. Hayek did not believe it was possible to predict the social world. Thinking that the world did follow predictable patterns led policy makers to act, and to get in the way. The true way was to leave the world to the market, to let it find its own equilibrium.
Nassim Taleb comes at this from a different angle. His Black Swan theory notes that the world pivots on unpredictable events, 9/11, the rise of the internet, World War 1. He goes further to say that our psychological biases blind us to these rare and powerful events. We don’t like to think the world is unpredictable, so we don’t think about it.
2. Not complex, not predictable
This is an empty box, something of an outlier. For a small group of people the anti-body reaction to the types of measurement that blossomed in the second era is to reject all measurement in all contexts (complex or not). This is a burden carried, for example, by Toby Lowe who has written extensively about how complexity changes what and how we learn. Too many people interpret Toby’s work as saying ‘we cannot measure’ or ‘we cannot know what is true’.
3. Not complex but predictable
This is where I would place the era 2 paradigm, the ordering of the world into a set of linear relationships between inputs, outputs, outcomes and impact; the focus on outcomes that matter for funders, for public systems and, to a lesser extent, foundations; the neutering of extraneous variables by the inclusion of control groups and, ideally, random allocation to intervention and control groups.
4. Complex and predictable
I have put several contrasting examples into this box. First up, some practical examples of complex social problems that are being address by robust learning. The five year survival rates for some cancers (the five year survival rate for prostate cancer had increased from 68% in the late 70s to 90 plus per cent in this decade). Fatalities in air crashes (that have fallen from 3,250 per trillion passenger kilometres in 1070 to less than 100 today). Progress has come not from one form of learning, but from many, and from applying important rules about how we learn -and to which we return in the next input.
Second up is the work of somebody like Andy Haldane, chief economist at the Bank of England. He accepts the complexity of the financial world, and for hidden, non-linear patterns to generate the conditions for the global financial crash of 2008. He builds that complexity into his models.
Third up are the super-forecasters described in the podcast with David Spiegelhalter. I will say less about them now since they are the focus of the next input.
Finally there are those of us in the network, you and me. We are all, it seems to me, inhabiting this segment, a place where each bit of new knowledge changes how we interpret and learn. We don’t use complex models like Andy Haldane, but we are dodging and weaving, using each bit of new data, each disproven prediction, to decide what we need to do, and learn next.
The Arrows
Four segment boxes look better with some arrows. I have inserted three.
- Hayek’s work led to Neo-liberalism, and to new public management and to its strange bed-fellow of era 2 learning. Hayek believed the world was unpredictable, that we should leave it all to the market. We have ended up with more prediction and data than could have ever be imagined. How we go there I am not quite sure.
- The naysayers are rejecting era 2 and rushing to the not complex, not predictable box. They are rejecting prediction, measurement and learning.
- The question we in the network have been asking is ‘where next?’. What comes after Era 2? I think it is to be found in the complex and predictable box. Let’s explore next some of the rules that the super-forecasters use when they operate in this space.