First indications from Peterborough – what do they tell us?

Last week was a big week for Social Finance as reoffending data on the Peterborough pilot was published by the Office of National Statistics. This gives the first early sense of how our first Social Impact Bond is doing. In this blog I want to explore the results a little and some of their implications.

So, first, the numbers, or to give it its full title:

Peterborough (and national equivalent) interim re-conviction figures using a partial (19 month) cohort and a 6 month re-conviction period

 Peterborough

National

Discharge Period

Cohort size

Binary

Frequency

Binary

Frequency

 Sep05-Mar07

725

39.70%

72

36.60%

61

 Sep06-Mar08

870

40.30%

81

37.80%

64

 Sep07-Mar09

1031

40.70%

84

38.30%

68

 Sep08-Mar10

981

41.60%

87

37.30%

69

 Sep10-Mar12

844

39.20%

81

39.30%

79

Binary: Reconviction rate over six months
Frequency: Frequency of reconviction events per 100 offenders within six months

Three topics to cover:

–          Is the Peterborough SIB working?

–          What do these numbers tell us about whether investors are likely to get paid?

–          Do they have any implications for developing the national recidivism PbR work?

1.  Is the Peterborough SIB working?

Put simply, it would appear so. The best way to show this is to index the results so that you can see them together and then to plot Peterborough relative to the national data:

Rebased reoffending data

So, the key measure for us is the one that we will be paid on, the percentage change in frequency of reoffending against a comparison group, in this instance the national cohort.

Peterborough relative to national

On that basis Peterborough has shown a 23% relative decline to the national data. On a sample size of 844 this is likely to be statistically significant, so on reoffending within six months, rather than a year, it appears we are making a difference.

Any caveats? A number. This is on the basis of six month reoffending, not 12 months, so one could argue that the impact of our programme may lessen over time. The comparison group, of wider national reoffending, is not as carefully defined as the comparison group that we have developed in the Peterborough model proper, where the reoffending rates of a matched cohort from the police national computer is used. Given this, the comparator group 16% increase over a two year period is something of an outlier, but it is all we have to go on.

So plenty of caveats. But however much these figures are indicative and however tentative and careful we are being; for a programme in its infancy and on its first cohort, this is a great start.

2. What do we know about whether investors will get paid?

So, two completely different numbers to note here:

a) the 23% relative decline discussed above; and

b) the fact that after this change the frequency and binary metric for Peterborough are now in line with the national average.

In other words what we have achieved so far is to move Peterborough from its historically higher rate of recidivism, to the national average. Through one lens we have done tremendously well. Through another lens Peterborough did (almost) exactly the same as the national average. Which lens will be reflected in the comparison group drawn from the police national computer?

If the prisoners in Peterborough are different and thus reoffend more, then this should be picked up in the comparison group as each individual is matched to one as similar as possible.

If the local environment is different, the prison for example, or the courts system, or the police… Then it is much less clear whether that will be picked up by the comparison group. It could be in part, if prisoners going through Peterborough are relatively local (and about 70% are) then those factors could be picked up to some extent in their criminal history and be matched to prisoners from similar environments. For those that are more transient, for example those coming through from London, such effects are unlikely to be picked up.

Locally there has been speculation for a number of years around why Peterborough’s recidivism rate is higher than the national average, and most of that speculation has focused on prisoner mix. But I don’t believe anybody has any evidence to back that up.

So, this all adds up to probably a greater uncertainty as to whether we will be paid for outcomes than we have that the programme is generating outcomes.

3. Any implications for the development of the national PbR programme?

These numbers probably complicate the development of the national PbR programme in one significant way, they give the impression of an increasing trend in reoffending, while the wider crime stats in terms of amount of reported crime and the British Crime Survey has generally been going down.

The key requirement this creates is that the Ministry of Justice needs to be completely transparent with the data and analysis that it is using to develop the counterfactual data. It simply cannot credibly develop it on its own and then tell people the answer. Regional variation needs to be understood, historical variance needs to be understood, a dialogue is needed to develop an acceptable answer.

Secondly, it increases the potential, in my view, for a proportion of outcomes payment to come from a fixed sized pot that is shared out according to relative performance amongst providers. This will resolve some of the uncertainties in bidders minds and show them that, while they may be taking a risk, there is a defined amount of outcomes payments that will be made if they perform better than some of their peers.

For such a pot to work, there should be a requirement to give a minimum spend on rehabilitation in the bidding process. Open book accounting thereafter can ensure that bidders keep to their promises, but the amount that bidders are willing to invest in reducing reoffending can then be used as part of their assessment. This can be used to counter the issue in the Work Programme – that for profit maximising providers the outcomes payments for harder to reach groups are insufficient to invest in trying to get them back into work.

So, the idea that bidders demonstrate a minimum commitment, is vital to maintaining the programmes credibility – that it is about rehabilitation, as opposed to only being about cost cutting and privatisation.

Advertisements

4 thoughts on “First indications from Peterborough – what do they tell us?

  1. Hi Toby
    Thanks for this detail – I really appreciate Social Finance’s transparency on this.
    In terms of national justice PbR contracts, an elephant in the room that I keep bumping into lately is that the direct reducing reoffending work of probation trusts or the ONE project may not be the biggest determinant of reoffending rates – at least in the short term. The national increase in re-offending rates for short term prisoners, the considerable fluctuation of local probation trust re-offending rates all make me think that policing may be the most important factor.
    I’m in no way disparaging the work of probation or ONE, we would be lost without it and reducing reoffending work has improved substantially over last decade in particular but if we are to measure against a baseline which typically jumps or down, there will be some nasty shocks along the way for both MoJ/NOMS and new providers.

    • Hi Russ,
      Thanks for this contribution. I broadly agree. Two separate points there, what are the determinants of reoffending rates and can we find stable baselines?

      On reoffending rates, there are definitely a range of elements that make a difference, and a good service should create better coordination and higher trust across the different participants in the marketplace. I wonder for example, whether a classic for-profit provider would build the same relationships with the police and the probation service that are necessary for our work in Peterborough. But it is still clear, or at least getting clearer that the ONE service is making a significant difference, and one that pays for its economics.

      On stable baselines, I agree this is an issue, thus my request for total transparency from the Ministry of Justice. Local variation has to be taken into account when setting the base or it is simply going to be meaningless.

      We also have to be slightly careful of two other distinctions. The first is how we reduce offending overall, rather than reoffending. Target hardening and co-ordinated action between local authorities and the police could both make bigger differences to offending rates, and could be looked at carefully. The second is how the PbR works. Good PbR as in Peterborough, allows flexible investment in an area, and learning on the go to adapt the programme. Poor PbR sets the values for outcomes, or bids the values for outcomes, so low that no work takes place in practice, as any work would simply lose money. Thus providers try to make money from the revenue elements and ignore outcomes.

      Assuming MoJ have to go forward with a programme for cost reasons, I see our job as trying to ensure that the rehabilitation element is actually serious, rather than cover for cost saving and privatisation!

      All best
      Toby

  2. Pingback: The 5th Commandment of Payment by Results: Thou shall not pay for deadweight

  3. Toby – thanks for posting this and for your analysis. Really, really interesting and a clear insight into the Peterborough pilot. Much to take from it but one key point is that we don’t let the PBR system create an ‘underclass’ where the hardest to work with – an therefore drive revenue from – are given less focus. Best, Rick

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s