Further data on the Peterborough Social Impact Bond

The Office of National Statistics provided further data on Peterborough at the end of July, this time on the complete first cohort of 1,000 prisoners.

While this is largely confirmatory information, the Ministry of Justice found a closer matching baseline, by focusing on local prisons rather than all national prisons. This responds to the concern that Peterborough may be hard to emulate or unrepresentative as it is local and therefore returns more of its prisoners to the local area.

The updated data looks like this:

Peterborough (and national equivalent) interim re-conviction figures of cohort 1 with a 6 month re-conviction period

 Peterborough

National local prisons

Discharge Period

Cohort size

Binary

Frequency

Binary

Frequency

 Sep05-Jun07

837

40.4%

74

36.60%

66

 Sep06- Jun08

1028

40.6%

81

37.80%

71

 Sep07- Jun09

1170

41.0%

85

38.30%

74

 Sep08- Jun10

1088

40.3%

84

37.30%

75

 Sep10- Jun12

1006

38.6%

78

39.30%

84

Binary: Reconviction rate over six months
Frequency: Frequency of reconviction events per 100 offenders within six months

A few topics to cover:

– Is this a better baseline and therefore does it give us greater confidence in the effect that Peterborough is having?

– Is this data good, or mixed as some have reported?

1. Is this a better baseline?
It should be, as it better matches the Peterborough cohort. As an experiment, I thought I would put together similar graphs to the ones before and compare them.

Data to March, with National baseline
Rebased reoffending data

Data to June with National local baselineRebased reoffending 2

And now the relative change graphs

Data to March, with National baseline
Peterborough relative to national

Data to June with National local baselinePeterborough relative to national2

What this shows visually is that the new baseline appears to be a better fit. Movements in the baseline prior to the intervention are closer to the movements in the Peterborough cohort, in other words the baseline appears to explain more of the movement in the Peterborough data. So it gives us greater confidence that we are seeing an intervention effect.

It also gives us a degree of greater confidence that we will get paid. The previous data ended with Peterborough’s frequency number equalling the national average. This one ends with Peterborough at least improving upon it. The propensity score matching process should bring out a comparison cohort that is even more similar, but of course we still haven’t tried it.

So, is it time to pop open the champagne and celebrate? Not yet. This is good news, but it is still only on six month data. We will be measured on whether we reduce offending over twelve months. What we can say is that our intervention appears to at least delay reoffending behaviour.

We should also say, this is only the first cohort of the first Social Impact Bond. It is incredibly early days so drawing significant conclusions at this stage is premature. On the other hand, we are learning and developing all the time, so the fact that we see a significant impact on such an early group is clearly exciting.

2. So is this data good, or mixed as some have reported?

We are cautious, because this is early days and early data. It isn’t a randomised control trial, sure. Nor is it the formal comparison cohort that will be developed for payment purposes using propensity score matching. But this is very positive data, on the best available information.

In the first set of results, which were also good, one of the caveats people put forward was that the Peterborough frequency was now only the national average. On this closer baseline this is no longer the case.

Another concern was that the jump in re-offending frequency in the national data should be treated with caution. I understand that, and see the potential for regression to the mean, but comparison with national data is more precise than looking at a comparison with historical data. Thus the 20% relative decline is the more useful figure than the 8% decline against historical figures, particularly given the strong correlation between the local prison data and the Peterborough data historically.

It is important to draw a distinction between responding with caution, on the basis of the caveats outlined above, and saying that results are “mixed” as we have seen in a few quarters. They’re not mixed, they’re surprisingly strong – but early and indicative at this stage.

A Step In the Right Direction But Not Enough

This blog was also posted on Social Finance’s blog here

Slowly we are getting to know more about the plans for probation and prisoner rehabilitation reform. We can see that some effort is being made to make the model work better following the consultation but is it enough to allow for a level playfield for all providers to take part? Will it achieve the ultimate goal reducing crime?

The key positive changes are as follows:

  • The whole idea that rehabilitation of prisoners and short sentence prisoners in particular has moved to centre stage and has become a key government policy is terrific and long overdue. Many should feel embarrassed that this has taken so long.
  • The idea of resettlement prisons, which will have a requirement to work with those providing rehabilitation services, and to which prisoners will be transferred at least three months before release. This resolves a key issue that effective rehabilitation needs to start in prison, rather than after.
  • The acceptance that a mix of binary and frequency measurement is required to make this system work. This may sound an esoteric point but is vital. A binary measure only pays in the event that a prisoner stops committing crime for twelve months. A frequency measure pays for reductions in offending across a cohort. Some people go through the prison system 10 or more times a year. If you have only a binary measure as was originally suggested the only correct financial decision when faced with such an individual (and required to give them a service) would be to give them a leaflet and tell them to go away. Further investment would invariably be loss-making as intense effort over a period of time and money is needed both to gain trust and thereafter help to move their lives in the right direction. Being paid on frequency and therefore acknowledging “distance travelled” will make it worthwhile financially.
  • The shift in number of contract areas from 16 to 21, and the different area sizes, are positive changes. This should mean that social organisations acting as primes, or a probation trust mutual, can bid.
  • But the positive impact of other elements in the response is less clear. The idea of standard contracts is interesting, but will they have to be finalised before the last round of bidding? In other words is there wriggle room?
  • The transparency piece sounds like a step in the right direction, but only a step. Input cost data and outcome payments should be transparently available across the market. In the public sector we see how much is spent on what and hopefully also get an idea (not always!) of the outcomes generated from that money. In the circumstances where you are building a new market, this data is even more important, not less. There will be enough benefit to incumbency without letting providers keep hold of this data. This will also make it clearer if a provider is not finding it economic to work with a particular cohort and is parking them.
  • The comments about women in prison were good to see, but they didn’t seem to imply that anything would be done to make the model work for women.

And there are areas where we simply don’t know anything:

  • What are the potential pricing expectations?
  • Will there be significant segmentation of the cohort and the pricing that goes with it?
  • Will men and women be priced the same? Needs and complexity are very different.
  • Is there room for alterations of pricing for specific groups as we learn over time? It seems deeply unlikely that it will be right first time.

So, at the end of this, what are concerns?

1. This is an incredibly complex, risky and ambitious programme of change. Tom Gash at the Institute for Government has written on this issue in his blog, with sensible recommendations for reducing the risk.

2. Bidding process and pace will favour incumbents

I was told by a private sector provider considering bidding for prisons that they had understood they should expect to bid in one round to learn how to possibly win in a later round. In other words, spend £1 million plus on a learning process before you stand a chance. Social organisations or probation trust mutuals don’t have that luxury. Those who know what the MoJ expects in large contracts will score better than those who are learning on the process. So the answer that it’s a level playing field simply doesn’t wash.

If charities are going to invest upwards of £250,000 of charitable funds and a considerable proportion of senior management time on a bidding process, they need more substance from the MoJ that they stand a realistic chance.

In addition, while there are some limited resources available to help test the mutual option, developing such a strategy and capacity takes time. So would developing a social prime and investment for it. The focus on the timetable above all else is in danger of defining the answer.

It should be a strategic imperative for the MoJ to end up with a mixed economy of private and social provision (and not just in the supply chain, at the prime level). There will be more learning, more constructive competitive tension, and probably greater investment in rehabilitation. European law should allow the MoJ to actively manage the market and they should do so, explicitly.

3. There is still room for gaming in the bidding

Gaming bidding is where an entity bids on the basis of having little intention of doing much rehabilitation, and makes money from input revenues without generating very many outcomes. Some seems to have occurred in the work programme, particularly around harder to reach groups. There are a number of ways that the MoJ can avoid this, examples include:

  • Requiring a certain level of investment in rehabilitation and monitoring it.
  • Scoring bidders on how much they say they will invest in rehabilitation and monitoring it.
  • Requiring transparency on input and outcome data and stating that bidders authenticity to what they said they would do will be assessed and those below a certain threshold won’t be allowed to bid again.
  • Without such measures, a sense that a low cost, gaming bid is likely to do better than a higher cost, rehabilitative bid will prevail

4. I’m not convinced the numbers add up

I can’t see into probation numbers, so I don’t know if it works to take out 20% of cost and provide an effective rehabilitation service for a wider community on top. But my instinct is that real rehabilitation will require real money. This money  is presently tied up in prisons. There should be a sense that these contracts can, if very successful, eat into the prison budget. What is fundamentally up for grabs here is what is the right allocation of resources between processing and punishing people, and trying to stop them doing it again. I wrote about this more substantially on another occasion. Read it here. My view is that the allocation that gets the number of future victims of crime to be as low as possible. In other words this is not about being nice to prisoners, or not nice to prisoners. It is about stopping crime and helping avoid further victims.

In conclusion, the MoJ is making some effort to allow this to work for a wider community than simply their incumbent private sector providers but not enough. The perceived need for speed and the inaccurate perception that they are building a level playing field are likely to undermine social sector interest in bidding at the top tier. The rehabilitation revolution should be about creating social value, reducing crime and reducing the costs of justice overall, and not simply about providing a lower cost privatised probation service. It would be a shame if at the end of the process this was how it was perceived.

Social Impact Bonds – why so slow?

Could Social Impact Bonds be happening more quickly? There seems to be a great deal of interest in the structure and the potential for bringing innovative services to bear on difficult social issues at lower risk to government. But they seem to take a long time to develop. In this blog post I explore why.

Firstly, I don’t buy the theory that this is just how long it takes. The idea is that government is not used to developing outcome based models, service providers are not used to working to them, investors are not used to taking outcome risk so it just takes a long time. All true, but our experience is that each of these can be broken down by effective intermediation and support, in other words, a third party that guides the others through the process to reach a deal. So the real question is – where are the intermediaries and why are they not making more SIBs happen?

Here are some possible answers:

a)      The role of the intermediary is poorly understood. What do they actually do?

b)      Who pays for it? When nobody is sure a deal can be reached, who should pay for its preparation, particularly as early deals are going to feel expensive as everyone is learning by doing.

c)       How does this fit with procurement? Government procurement puts a wall down the middle of the process making consensus based co-development very difficult.

d)      Distrust of those outside government by those inside. A sense that an organisation is focused on its commercial interest so cannot be asked for support.

All of these have some truth to them but in a certain sense they all hinge on a) the role being poorly understood. So why is that the case?

Governments generally don’t quite get intermediation, seeing it as an expense in the middle that is possibly unnecessary. I discussed this with Gary Sturgess, adviser to the New South Wales Government, and former State Cabinet Secretary. His theory was that government doesn’t “see” certain types of organisation and intermediaries are certainly one. Government understands service providers: they do services; and investors: they provide money. But intermediaries? In a certain sense their role is more obviously invading the space normally taken up by government itself.

One of the things I have always found slightly strange in SIBs is how few people ask the simple question: “How did you do it?” You would have thought that the various governments interested in doing them, or the various nascent intermediaries around the world, would have got in touch to find out the activities that were necessary to get the job done.

Holding that for a second, I’m wondering whether this is connected to another surreal debate around SIBs, that of who first thought of them. If you look at Wikipedia for example, and various other publications around the place, there is an attempt by various people to claim credit for the idea. I’ve always found this irritating didn’t want to wade into because a) it feels petty minded and b) we’re grateful to a range of people who contributed to early thinking on SIBs so didn’t want to look like we were joining in the me, me, me thing.

But there is something more fundamental going on that is worth touching on. Why is the debate about who thought of the idea, rather than who delivered it? In every other walk of life the definition of invention surely includes implementation. If you look up the invention of flight or the light bulb, the credit typically goes to those that made it work. But in this context the debate seems to be who first mentioned it in a paper or publication. Does this say something deeper about the social sector that we should all be aware of? Is implementation simply not respected in the same way as it might be in other areas?

To take a different example, when we talk of supermarkets in the UK, we talk of specific brands, of implementations – Tesco, Sainsbury’s, Asda or Morrison’s. The idea that someone could just make their own supermarket to the same standard without access to those companies internal body of knowledge is slightly absurd. Yet in the social sector or governmental context codified models that can be replicated by others are the goal or perceived norm. Imagine if the government was procuring a “social” supermarket. Would it research supermarkets carefully, run a detailed procurement process and hand it to the bidder who wrote down the best answers to its questions (as assessed by someone who had never run a supermarket) and offered the best price? Or would it use information gleaned from going to a bunch of supermarkets and talking to their customers? My guess is the former (I will expand on this idea in a later blog).

I can see the value of codified models from a research and evaluation perspective, then you can run Randomised Control Trials comparing different models. But built up know-how has value in the normal course of business and the micro-developments that individual organisations will learn and deliver should lead to better outcomes.

One of the other impacts I have noticed from this focus on ideas is that much of the commentary on SIBs is focused on the idea aspects without a focus on implementation. So one for example said  something like “the Peterborough SIB is interesting, but I prefer Ronnie Horesh’s SIBs because they can be bought and sold on a secondary market”. All very interesting, but missing the point that one of the key reasons governments may not choose to implement the policy bond model (which I’ve always liked by the way) is the ability that the counterparty has to change through buying and selling the instruments. Government likes to know who it is dealing with. In fact the only way to such a secondary market, I might suggest, is by giving government confidence in the value of the model first, and then start thinking through the implications of fungibility and whether the benefits in terms of scale of available capital outweigh the concerns of policy makers.

So is there a cultural reason why SIBs are taking time? That implementation experience and knowledge is overlooked as those looking to do things start from a focus on the theory, the idea, rather than how it happened? Do procurement processes exacerbate this problem by building the divide further and reducing the perceived value of real world experience? Is the social world obsessed with newness and innovation and ideas, at the expense of effective implementation?

Answers on a postcard please… (or a comment, or a tweet, or an email…)

I know by the way that I am meant to be doing a series on Social Enterprise more widely… And I will, this has just been mulling round my head for a while so I wanted to get it out there. I will also blog on the role of the intermediary …

PS For the avoidance of doubt quite a few people have at various times thought about outcome based contracts or outcome based commissioning, others about government paying for outcome notes that could be bought and sold (Ronnie Horesh), others about structures where there were outcome payers and investors (Arthur Wood and Guillermo Maclean), and again others who were wondering about whether government could pay for preventative work out of the cost savings from reduced acute costs (Peter Wheeler and David Robinson). Social Finance combined its own thinking with the ideas and thoughts of Arthur, Peter and David in developing the SIB and heard about the others later.

Eight ways to muck up PbR … and a few ways to improve it

Payment by results has been getting a lot of grief recently. Some of it deserved. The compulsion for revolution rather than evolution, and poor contracting and procurement practice have combined to create a heavily loss making industry that is not generating outcomes. The fear is that when government doesn’t seem comfortable admitting to any previous mistake it may be in danger of repeating the mistakes it has made. So as a source of Friday entertainment, I thought I would help by consolidating my eight best ways of mucking up outcomes based contracts. 

1)    Set the maximum outcome payment at or below the value of the previous revenue contract: This way providers will be nervously trying to retain revenue, rather than innovating to improve outcomes

2)    Set outcome measures and values without engaging the marketplace: Measures take a while to achieve buy-in, and to motivate change, they need it…

3)    Run a price focused rather than quality focused procurement process: This favours naive or optimistic bidders over competent ones

4)    Transfer risks that the bidder is in no position to control: It may make you feel better, but it increases cost disproportionately, or ensures you only get naïve bidders through who haven’t thoughts about it

5)    Provide detailed information only after bidders have been asked their price, and then offer them only the option to continue or withdraw: This is simply a way of ensuring poor services by desperate bidders

6)    Seek to control interventions, inputs and processes as well as outcomes: Again tempting, but not leaving room for innovation takes away much of the point

7)    Maximise proportion paid on outcomes on principle: This just restricts bidders and increases cost. There are times when a full outcomes model is a good idea, but it shouldn’t be the default.

8)    Don’t test or learn by staggering your start: Doing everything when you know least is simply irrational…

And a few ideas for doing it well, which I may expand on another time:

1) Have a way to alter prices over time to allow for learning: for example you could have a maximum and minimum outcomes pot that would be distributed to providers according to level of outcomes achieved. This would still reward quality and improvement but would limit downside and upside risk.

2) Manage your market more aggressively: If you are creating a new market, you should decide what you are looking for and create it. I would intentionally create a mixed economy, reserving some slots for say mutuals or social enterprises. That way you ensure diversity, avoid oligopoly and maximise learning and innovation.

Anyone like to add or amend any of the above?

Are Payment by Results and Management Targets dangerously idiotic?

Occasionally one reads an article and feels compelled to respond. Today was such a day when I read Payment by Results – a ‘dangerous idiocy’ that makes staff tell lies by Toby Lowe. It’s not the criticism of PbR or targets I dislike, there are plenty of concerns that need to be thought through in that area and I start on some of those in the next few blogs. It’s the lack of presented alternative that bothers me. Payment by results gets muddled up with management targets, with a sense that if all of this went away and you relied on frontline staff to do the right thing all would be well. This sentimental harking to a golden age before management and targets is magical thinking.

There are two reasons it is dangerous. Firstly it fails to acknowledge that we live in a resource constrained system. We don’t have infinite money, so we have to make choices. Choices exclude and we have to have some way of making them and some people to make them. Managers are typically rationers. Not an easy or popular task so one that we should support rather than seek to undermine.

Secondly it presumes that front-line staff invariably does the right thing by service users, patients or customers. While I have huge regard for people’s basic ethic to each other, moral compass and sense of decency, this is a sentimental and paternalistic view that we can’t run our public services on. Professions and professionals get inured and set into defined practice. When working for a foundation, I had the disturbing role of going round orphanages in Romania being told by professionals within them why they should stay open and how the alternatives we were proposing were dangerous. In South Africa there were separate cases of wards filled with crying abandoned AIDS orphans and nurses sitting in their station beside them doing nothing. It requires management at certain times to ensure focus on patients, end users, to ensure a lack of institutional bias, to ensure people aren’t forgotten or lost.

So the article attempts to unseat the unpopular but important act of putting in place targets and management to benefit service users. In the end, most of us don’t like to be managed. Few people like being told what to do, so it is always tempting to bash management. But services need to be adapted and resources need to be allocated. Better decisions are taken on the basis of data, not anecdote, intuition and here say. So information needs to be gleaned, measures used and targets set.

There are two serious issues that need to be thought through. What are good management vs bad management targets, and when if ever does PbR help? I will address those in my next blogs.

 

Is Chris Grayling missing the revolution?

Chris Grayling yesterday published his consultation on reforming probation, focusing more on rehabilitation and bringing in the private sector. This is double edged for us. It directly builds on our work in Peterborough, but does so very early and before there has been wide testing of PbR in criminal justice. He does on the other hand have a limited political window before the next election and it has put rehabilitation firmly on the political agenda.There is also at least some money available for short sentence offenders which is a break through. Probation is cross, understandably, and most others aren’t commenting in public as the detail isn’t out and they may be bidding. So far so good.
The story bombed out pretty quickly, but you can read about it here. Social Finance issued the statement here.
The issue that I’m interested in is what we alluded to in the paragraph:
“We encourage the Secretary of State for Justice to look beyond the Probation budget to finance this work. The criminal justice system costs the taxpayer £6.3 billion a year. At present we spend 13p in the pound on probation and rehabilitation, and 87p in the pound on locking people up. With more resources, better rehabilitation could cut crime andreduce wider criminal justice costs. The allocation of resources between these two areas should therefore be part of this consultation. This is possible as any new money would be paid on a results only basis. Without adequate resources, it will be difficult to ensure that support for rehabilitation is delivered properly and at scale.”
The probation budget of course overstates the amount going to rehabilitation, as probation does a lot of work around public safety, managing community sentences, advising the court and so on. So I think it is safe to assume 90-95p in the pound goes on processing and punishment and 5-10p max goes into rehabilitation. Now many might argue that is the role of the justice department. The clue, as they say, is in the title. It ain’t the fluffy put people back together department, it is the Ministry of Justice. I can see their point, but effective rehabilitation is in the Ministry’s interests. It cuts crime and should reduce their costs.
So my question is, why is even a discussion of the allocation of resources between punishment and rehabilitation unlikely to be on the agenda, and what can we do about it?
First let’s look at how it could be done.
Payment by results is being used in two different ways at the moment. One way, like the work programme in the UK, is to try to improve the efficiency/effectiveness of a present service area that is perceived to be providing poor results. This is done at scale and typically ratchets up the outcome focus over time, so that providers have time to improve or get taken over if they can’t sort themselves out.
The other way, like the Peterborough SIB, is to try innovation. Government would not pay for the service under normal circumstances, as it is perceived as too risky. But if you only ask them to pay for the outcomes achieved, then with a bit of luck, you can develop a contract and test out a service in a rigorous way and with the government’s blessing.
There is a potential third way(!). Outcome based models can be used to test the appropriate allocation of resources between a preventative spend and an acute one. So an outcome based contract is set up for the preventative spend. It doesn’t have to be entirely outcomes based, in fact it is probably better if it isn’t, but any spend over and above expected preventative spend would need to be on an outcomes basis, as it would be being taken from the potential spending on the acute service. In other words the additional preventative money would only be available if it had successfully shrunk demand for the acute services.
This method is potentially the most radical, as it allows service effectiveness to decide on budget allocation, an unusually rational way of doing things.
So why is this not being done? In all the conversations we have had with government insetting up SIBs of all shapes and sizes, they always have an expectation of an outcomes cap. The fear of excessive success is often significant. To my mind the reasons are as follows:
  • supply in-elasticity  for a lot of acute services, there are high fixed costs and demand has to reduce significantly before supply can be taken out of the system, before a prison can close for example.
  • backfill: many acute services are perceived to have infinite demand in comparison to supply (mental health for example), so any reduction in demand will simply be filled by other people. Or spare capacity will impact on behaviours elsewhere in the system. Judges will sentence more people to prison if prisons aren’t full, social workers might put more children into care if there are places available etc.
  • fear of poor contracting: the concern that they will somehow get legged over in the contract and suppliers will succeed in getting paid without having the wider outcomes impact that would actually reduce acute service demand.
So what can be done about this? All the problems described above are soluble over time.For example, outcome caps can be put in place, but any supplier hitting them has the cap and outcome values raised after a time delay of say a year. This would allow time for supply reduction strategies to be put in place and to ensure that backfill is managed. The detail issue would be managing whether such raises to the cap would be automatic or at the government contract holder’s discretion. The former ensures pressure is maintained in the system to implement supply reduction strategies – a pressure that is typically needed as reducing service levels is hard and unpopular with stakeholders. The latter reduces the risk if the contract was poorly formed in the first place and suppliers are reaching caps but not impacting actual demand.
I’m not sure this is sufficient unfortunately. To do this also requires different accounting rules and ways of managing public expenditure. I’m not an expert, but I imagine a department wanting to retain flexibility between two significant budget line items is not the sort of thing the Treasury likes. I do know that DWP had a significant fight on its hands when it sought to do the DEL/AME switch but succeeded eventually. I’m not sure what the response would be when looking at two departmental budgets as would be the case in Justice.

Where I am sure that Treasury would agree is that a supply reduction strategy needs to be in place alongside any demand reduction if we are to make any real difference to long term numbers and cost.

When faced with Gordian knots such as this, our experience is that a migration path is needed that allows the government to take initial steps without taking on too much risk. In this instance that is about pushing to ensure at least some of the contracts allow for either quite high levels of performance before being capped, or for the caps to be expanded. It is also vital that a range of actors engage with this debate and make relevant contributions to the consultation process.

Please get in touch if you have thoughts, feedback or want to participate in this debate.