First indications from Peterborough – what do they tell us?

Last week was a big week for Social Finance as reoffending data on the Peterborough pilot was published by the Office of National Statistics. This gives the first early sense of how our first Social Impact Bond is doing. In this blog I want to explore the results a little and some of their implications.

So, first, the numbers, or to give it its full title:

Peterborough (and national equivalent) interim re-conviction figures using a partial (19 month) cohort and a 6 month re-conviction period

 Peterborough

National

Discharge Period

Cohort size

Binary

Frequency

Binary

Frequency

 Sep05-Mar07

725

39.70%

72

36.60%

61

 Sep06-Mar08

870

40.30%

81

37.80%

64

 Sep07-Mar09

1031

40.70%

84

38.30%

68

 Sep08-Mar10

981

41.60%

87

37.30%

69

 Sep10-Mar12

844

39.20%

81

39.30%

79

Binary: Reconviction rate over six months
Frequency: Frequency of reconviction events per 100 offenders within six months

Three topics to cover:

–          Is the Peterborough SIB working?

–          What do these numbers tell us about whether investors are likely to get paid?

–          Do they have any implications for developing the national recidivism PbR work?

1.  Is the Peterborough SIB working?

Put simply, it would appear so. The best way to show this is to index the results so that you can see them together and then to plot Peterborough relative to the national data:

Rebased reoffending data

So, the key measure for us is the one that we will be paid on, the percentage change in frequency of reoffending against a comparison group, in this instance the national cohort.

Peterborough relative to national

On that basis Peterborough has shown a 23% relative decline to the national data. On a sample size of 844 this is likely to be statistically significant, so on reoffending within six months, rather than a year, it appears we are making a difference.

Any caveats? A number. This is on the basis of six month reoffending, not 12 months, so one could argue that the impact of our programme may lessen over time. The comparison group, of wider national reoffending, is not as carefully defined as the comparison group that we have developed in the Peterborough model proper, where the reoffending rates of a matched cohort from the police national computer is used. Given this, the comparator group 16% increase over a two year period is something of an outlier, but it is all we have to go on.

So plenty of caveats. But however much these figures are indicative and however tentative and careful we are being; for a programme in its infancy and on its first cohort, this is a great start.

2. What do we know about whether investors will get paid?

So, two completely different numbers to note here:

a) the 23% relative decline discussed above; and

b) the fact that after this change the frequency and binary metric for Peterborough are now in line with the national average.

In other words what we have achieved so far is to move Peterborough from its historically higher rate of recidivism, to the national average. Through one lens we have done tremendously well. Through another lens Peterborough did (almost) exactly the same as the national average. Which lens will be reflected in the comparison group drawn from the police national computer?

If the prisoners in Peterborough are different and thus reoffend more, then this should be picked up in the comparison group as each individual is matched to one as similar as possible.

If the local environment is different, the prison for example, or the courts system, or the police… Then it is much less clear whether that will be picked up by the comparison group. It could be in part, if prisoners going through Peterborough are relatively local (and about 70% are) then those factors could be picked up to some extent in their criminal history and be matched to prisoners from similar environments. For those that are more transient, for example those coming through from London, such effects are unlikely to be picked up.

Locally there has been speculation for a number of years around why Peterborough’s recidivism rate is higher than the national average, and most of that speculation has focused on prisoner mix. But I don’t believe anybody has any evidence to back that up.

So, this all adds up to probably a greater uncertainty as to whether we will be paid for outcomes than we have that the programme is generating outcomes.

3. Any implications for the development of the national PbR programme?

These numbers probably complicate the development of the national PbR programme in one significant way, they give the impression of an increasing trend in reoffending, while the wider crime stats in terms of amount of reported crime and the British Crime Survey has generally been going down.

The key requirement this creates is that the Ministry of Justice needs to be completely transparent with the data and analysis that it is using to develop the counterfactual data. It simply cannot credibly develop it on its own and then tell people the answer. Regional variation needs to be understood, historical variance needs to be understood, a dialogue is needed to develop an acceptable answer.

Secondly, it increases the potential, in my view, for a proportion of outcomes payment to come from a fixed sized pot that is shared out according to relative performance amongst providers. This will resolve some of the uncertainties in bidders minds and show them that, while they may be taking a risk, there is a defined amount of outcomes payments that will be made if they perform better than some of their peers.

For such a pot to work, there should be a requirement to give a minimum spend on rehabilitation in the bidding process. Open book accounting thereafter can ensure that bidders keep to their promises, but the amount that bidders are willing to invest in reducing reoffending can then be used as part of their assessment. This can be used to counter the issue in the Work Programme – that for profit maximising providers the outcomes payments for harder to reach groups are insufficient to invest in trying to get them back into work.

So, the idea that bidders demonstrate a minimum commitment, is vital to maintaining the programmes credibility – that it is about rehabilitation, as opposed to only being about cost cutting and privatisation.

Advertisements

A Step In the Right Direction But Not Enough

This blog was also posted on Social Finance’s blog here

Slowly we are getting to know more about the plans for probation and prisoner rehabilitation reform. We can see that some effort is being made to make the model work better following the consultation but is it enough to allow for a level playfield for all providers to take part? Will it achieve the ultimate goal reducing crime?

The key positive changes are as follows:

  • The whole idea that rehabilitation of prisoners and short sentence prisoners in particular has moved to centre stage and has become a key government policy is terrific and long overdue. Many should feel embarrassed that this has taken so long.
  • The idea of resettlement prisons, which will have a requirement to work with those providing rehabilitation services, and to which prisoners will be transferred at least three months before release. This resolves a key issue that effective rehabilitation needs to start in prison, rather than after.
  • The acceptance that a mix of binary and frequency measurement is required to make this system work. This may sound an esoteric point but is vital. A binary measure only pays in the event that a prisoner stops committing crime for twelve months. A frequency measure pays for reductions in offending across a cohort. Some people go through the prison system 10 or more times a year. If you have only a binary measure as was originally suggested the only correct financial decision when faced with such an individual (and required to give them a service) would be to give them a leaflet and tell them to go away. Further investment would invariably be loss-making as intense effort over a period of time and money is needed both to gain trust and thereafter help to move their lives in the right direction. Being paid on frequency and therefore acknowledging “distance travelled” will make it worthwhile financially.
  • The shift in number of contract areas from 16 to 21, and the different area sizes, are positive changes. This should mean that social organisations acting as primes, or a probation trust mutual, can bid.
  • But the positive impact of other elements in the response is less clear. The idea of standard contracts is interesting, but will they have to be finalised before the last round of bidding? In other words is there wriggle room?
  • The transparency piece sounds like a step in the right direction, but only a step. Input cost data and outcome payments should be transparently available across the market. In the public sector we see how much is spent on what and hopefully also get an idea (not always!) of the outcomes generated from that money. In the circumstances where you are building a new market, this data is even more important, not less. There will be enough benefit to incumbency without letting providers keep hold of this data. This will also make it clearer if a provider is not finding it economic to work with a particular cohort and is parking them.
  • The comments about women in prison were good to see, but they didn’t seem to imply that anything would be done to make the model work for women.

And there are areas where we simply don’t know anything:

  • What are the potential pricing expectations?
  • Will there be significant segmentation of the cohort and the pricing that goes with it?
  • Will men and women be priced the same? Needs and complexity are very different.
  • Is there room for alterations of pricing for specific groups as we learn over time? It seems deeply unlikely that it will be right first time.

So, at the end of this, what are concerns?

1. This is an incredibly complex, risky and ambitious programme of change. Tom Gash at the Institute for Government has written on this issue in his blog, with sensible recommendations for reducing the risk.

2. Bidding process and pace will favour incumbents

I was told by a private sector provider considering bidding for prisons that they had understood they should expect to bid in one round to learn how to possibly win in a later round. In other words, spend £1 million plus on a learning process before you stand a chance. Social organisations or probation trust mutuals don’t have that luxury. Those who know what the MoJ expects in large contracts will score better than those who are learning on the process. So the answer that it’s a level playing field simply doesn’t wash.

If charities are going to invest upwards of £250,000 of charitable funds and a considerable proportion of senior management time on a bidding process, they need more substance from the MoJ that they stand a realistic chance.

In addition, while there are some limited resources available to help test the mutual option, developing such a strategy and capacity takes time. So would developing a social prime and investment for it. The focus on the timetable above all else is in danger of defining the answer.

It should be a strategic imperative for the MoJ to end up with a mixed economy of private and social provision (and not just in the supply chain, at the prime level). There will be more learning, more constructive competitive tension, and probably greater investment in rehabilitation. European law should allow the MoJ to actively manage the market and they should do so, explicitly.

3. There is still room for gaming in the bidding

Gaming bidding is where an entity bids on the basis of having little intention of doing much rehabilitation, and makes money from input revenues without generating very many outcomes. Some seems to have occurred in the work programme, particularly around harder to reach groups. There are a number of ways that the MoJ can avoid this, examples include:

  • Requiring a certain level of investment in rehabilitation and monitoring it.
  • Scoring bidders on how much they say they will invest in rehabilitation and monitoring it.
  • Requiring transparency on input and outcome data and stating that bidders authenticity to what they said they would do will be assessed and those below a certain threshold won’t be allowed to bid again.
  • Without such measures, a sense that a low cost, gaming bid is likely to do better than a higher cost, rehabilitative bid will prevail

4. I’m not convinced the numbers add up

I can’t see into probation numbers, so I don’t know if it works to take out 20% of cost and provide an effective rehabilitation service for a wider community on top. But my instinct is that real rehabilitation will require real money. This money  is presently tied up in prisons. There should be a sense that these contracts can, if very successful, eat into the prison budget. What is fundamentally up for grabs here is what is the right allocation of resources between processing and punishing people, and trying to stop them doing it again. I wrote about this more substantially on another occasion. Read it here. My view is that the allocation that gets the number of future victims of crime to be as low as possible. In other words this is not about being nice to prisoners, or not nice to prisoners. It is about stopping crime and helping avoid further victims.

In conclusion, the MoJ is making some effort to allow this to work for a wider community than simply their incumbent private sector providers but not enough. The perceived need for speed and the inaccurate perception that they are building a level playing field are likely to undermine social sector interest in bidding at the top tier. The rehabilitation revolution should be about creating social value, reducing crime and reducing the costs of justice overall, and not simply about providing a lower cost privatised probation service. It would be a shame if at the end of the process this was how it was perceived.

Is Chris Grayling missing the revolution?

Chris Grayling yesterday published his consultation on reforming probation, focusing more on rehabilitation and bringing in the private sector. This is double edged for us. It directly builds on our work in Peterborough, but does so very early and before there has been wide testing of PbR in criminal justice. He does on the other hand have a limited political window before the next election and it has put rehabilitation firmly on the political agenda.There is also at least some money available for short sentence offenders which is a break through. Probation is cross, understandably, and most others aren’t commenting in public as the detail isn’t out and they may be bidding. So far so good.
The story bombed out pretty quickly, but you can read about it here. Social Finance issued the statement here.
The issue that I’m interested in is what we alluded to in the paragraph:
“We encourage the Secretary of State for Justice to look beyond the Probation budget to finance this work. The criminal justice system costs the taxpayer £6.3 billion a year. At present we spend 13p in the pound on probation and rehabilitation, and 87p in the pound on locking people up. With more resources, better rehabilitation could cut crime andreduce wider criminal justice costs. The allocation of resources between these two areas should therefore be part of this consultation. This is possible as any new money would be paid on a results only basis. Without adequate resources, it will be difficult to ensure that support for rehabilitation is delivered properly and at scale.”
The probation budget of course overstates the amount going to rehabilitation, as probation does a lot of work around public safety, managing community sentences, advising the court and so on. So I think it is safe to assume 90-95p in the pound goes on processing and punishment and 5-10p max goes into rehabilitation. Now many might argue that is the role of the justice department. The clue, as they say, is in the title. It ain’t the fluffy put people back together department, it is the Ministry of Justice. I can see their point, but effective rehabilitation is in the Ministry’s interests. It cuts crime and should reduce their costs.
So my question is, why is even a discussion of the allocation of resources between punishment and rehabilitation unlikely to be on the agenda, and what can we do about it?
First let’s look at how it could be done.
Payment by results is being used in two different ways at the moment. One way, like the work programme in the UK, is to try to improve the efficiency/effectiveness of a present service area that is perceived to be providing poor results. This is done at scale and typically ratchets up the outcome focus over time, so that providers have time to improve or get taken over if they can’t sort themselves out.
The other way, like the Peterborough SIB, is to try innovation. Government would not pay for the service under normal circumstances, as it is perceived as too risky. But if you only ask them to pay for the outcomes achieved, then with a bit of luck, you can develop a contract and test out a service in a rigorous way and with the government’s blessing.
There is a potential third way(!). Outcome based models can be used to test the appropriate allocation of resources between a preventative spend and an acute one. So an outcome based contract is set up for the preventative spend. It doesn’t have to be entirely outcomes based, in fact it is probably better if it isn’t, but any spend over and above expected preventative spend would need to be on an outcomes basis, as it would be being taken from the potential spending on the acute service. In other words the additional preventative money would only be available if it had successfully shrunk demand for the acute services.
This method is potentially the most radical, as it allows service effectiveness to decide on budget allocation, an unusually rational way of doing things.
So why is this not being done? In all the conversations we have had with government insetting up SIBs of all shapes and sizes, they always have an expectation of an outcomes cap. The fear of excessive success is often significant. To my mind the reasons are as follows:
  • supply in-elasticity  for a lot of acute services, there are high fixed costs and demand has to reduce significantly before supply can be taken out of the system, before a prison can close for example.
  • backfill: many acute services are perceived to have infinite demand in comparison to supply (mental health for example), so any reduction in demand will simply be filled by other people. Or spare capacity will impact on behaviours elsewhere in the system. Judges will sentence more people to prison if prisons aren’t full, social workers might put more children into care if there are places available etc.
  • fear of poor contracting: the concern that they will somehow get legged over in the contract and suppliers will succeed in getting paid without having the wider outcomes impact that would actually reduce acute service demand.
So what can be done about this? All the problems described above are soluble over time.For example, outcome caps can be put in place, but any supplier hitting them has the cap and outcome values raised after a time delay of say a year. This would allow time for supply reduction strategies to be put in place and to ensure that backfill is managed. The detail issue would be managing whether such raises to the cap would be automatic or at the government contract holder’s discretion. The former ensures pressure is maintained in the system to implement supply reduction strategies – a pressure that is typically needed as reducing service levels is hard and unpopular with stakeholders. The latter reduces the risk if the contract was poorly formed in the first place and suppliers are reaching caps but not impacting actual demand.
I’m not sure this is sufficient unfortunately. To do this also requires different accounting rules and ways of managing public expenditure. I’m not an expert, but I imagine a department wanting to retain flexibility between two significant budget line items is not the sort of thing the Treasury likes. I do know that DWP had a significant fight on its hands when it sought to do the DEL/AME switch but succeeded eventually. I’m not sure what the response would be when looking at two departmental budgets as would be the case in Justice.

Where I am sure that Treasury would agree is that a supply reduction strategy needs to be in place alongside any demand reduction if we are to make any real difference to long term numbers and cost.

When faced with Gordian knots such as this, our experience is that a migration path is needed that allows the government to take initial steps without taking on too much risk. In this instance that is about pushing to ensure at least some of the contracts allow for either quite high levels of performance before being capped, or for the caps to be expanded. It is also vital that a range of actors engage with this debate and make relevant contributions to the consultation process.

Please get in touch if you have thoughts, feedback or want to participate in this debate.