What we have been reading last week — flawed evidence & complex systems edition
We are trying to work more “in the open” and share what we are learning as we go along. In that spirit, I wanted to share a note on some of the interesting reports and articles we have come across last week.
Flawed evidence: houses built on sand 🚩
The point of public policy is to change the world from how it is today into some more desirable state. But how do we know whether our interventions are effective? One increasingly popular answer is to urge policymaker to use more and better evidence to find out “what works”.
“Scared Straight” is a prison visitation programme meant to deter teenagers from committing crimes. It is frequently held up as the poster case for why looking at the evidence is so important. A systematic review of the scientific evidence concluded that “Scared Straight” programmes not only did not work but in fact increased the chances that the teenagers included in the programme would go on to offend.
This satisfyingly counterintuitive result was like catnip for popular science writers as well as advocates of evidence-informed policymaking. They, quite reasonably, went on to admonish those who run such programmes for not bothering to look at the evidence.
The only problem? The argument is like a house built on sand. Once you start digging near the foundations, everything crumbles. Adrian took a deep look at the systematic review and the studies behind it:
“The ‘very strong’ evidence against Scared Straight is based on seven US studies from 1967–1982, only two of which are statistically significant, and both of these have potentially serious flaws.”
Should we conclude from this, as William MacAskill does in “Doing Good Better”, that “Scared Straight” programmes — for everyone, everywhere, forever — are not only “ineffective” but “downright harmful”? That would probably be a mistake and, as Adrian argues in the piece, risks preventing us from innovating and experimenting.
Tom Gash, a long-time friend of CPI and expert on crime, wrote a thoughtful response. Do read both pieces if you can.
With regards to the limitations of evidence Lars Peter Hansen, a recent recipient of the Nobel Memorial Prize in Economic Sciences, made the obvious but necessary point that purely evidence-based policy simply isn’t a thing:
“While we want to embrace evidence, the evidence seldom speaks for itself; typically, it requires a modeling or conceptual framework for interpretation.”
The animation below visualizes this in the most beautiful way possible:
(Gary Banks made some related points in a speech for the Australia and New Zealand School of Government late last year)
A timely HBR article by Christopher Worsham and Anupam B. Jena, both physicians, on the “art of evidence-based medicine” further reminds us that knowing the average effect of an intervention of a population only takes you so far:
“For instance, the publication of high-quality, randomized controlled trials that demonstrate a given treatment is, on average, less efficacious than a comparator, should not necessarily result in observed rates of that treatment’s use falling to zero. Rather, the clear prediction is that treatment rates should fall as informed physicians update their prior beliefs about the treatment’s efficacy.”
The words “update their prior beliefs” do more work than may be readily apparent.
The debate around “evidence” is often held within the unacknowledged confines of the traditional, so-called “frequentist” statistical model. It’s a model of hypothesis testing and the results only allow for a binary result: either something’s statistically significant or not.
If I believed yesterday that “Scared Straight” reduces crime and I learn today that there’s a study showing that it doesn’t then the frequentist model offers no way of reconciling these two seemingly contradictory positions.
There is, however, a different way of thinking about evidence: Bayesian statistics. Bayesian methods allow us to systematically revise (or “update”, to use the technical term) our view of the world as new evidence becomes available. Bayesian statistics moves us out of a binary world where I either accept a new piece of information and reject my prior world view or reject the new information and maintain my opinion. Instead, it allows us to update our beliefs about how the world works and integrate new information as and when it becomes available.
While Bayesian statistics isn’t particularly exotic it still isn’t taught as a matter of course in many public policy schools. Is that fact maybe also to blame for some of the more simplistic interpretations of how evidence-informed policymaking ought to work?
Public policy made tangible and exciting 🎢
After writing “The Big Short” and what may well have been the first popularly accessible account of collateralized debt obligations Michael Lewis tackled another subject not known for its sex appeal: government.
In “The Fifth Risk” Lewis tells the hair-raising story of the presidential transition under Trump and, in the process, explains many obscure-yet-utterly-vital parts of the US federal government. It’s an entertaining read and well worth your time. If you want to sample before you commit: one of the chapters was previously published in Vanity Fair.
Govern complex systems with simple principles (and not complex processes) ✨
There seems to be a universal logic at work with regards to regulating complex systems. When trying to influence a complex system simple principles work, complex rules don’t. This logic seems to hold true no matter how large or small the system is we are looking at.
Stanford’s d.school, the home of “design thinking”, has moved from teaching a particular design process to teaching principles. Why? Presumably, because the method of design is useful precisely because it lends itself to ambiguous, complex situations. Those, however, ask for general principles that guide our choices rather than for overly detailed rules.
Similarly, many vanguard organisations are moving towards “small, relatively autonomous multidisciplinary teams that are coordinated more through shared mission, culture and principles than traditional management”. In other words, they rely on broad principles rather than the detailed-rules-based logic of traditional management (this excellent Nesta blog post on “Nine emerging trends in the management and support of innovation” has more detail on this. It also includes the d.school piece referenced above).
Andy Haldane, Chief Economist at the Bank of England, made the same argument but in a much larger context in a speech to the annual central banking conference in Jackson Hole in 2012:
“As you do not fight fire with fire, you do not fight complexity with complexity. Because complexity generates uncertainty, not risk, it requires a regulatory response grounded in simplicity, not complexity.”
The intuitive truth of this is confirmed by our own experiences. We moved to a system of self-managing teams in one of our offices at the beginning of this year. Simple principles that we have given ourselves — like “work in the open” — have changed the way we work more deeply than any fifty-page manual could have.
Two sharp questions to end with. 🎪
Did you know that one of the first Randomized Control Trials caused the Sicilian Mafia? No, really. (via Mark Egan)
Which hot tub is the most judgmental one? Of course the j’accuzzi.
You’re welcome.
