Greater Manchester Centre for Voluntary Organisation

Evidence Based Commissioning: It's not what you do, it's who you work with

As voluntary organisations we are continually encouraged to present evidence of our impact on the basis that it will help us gain funding to work with our clients. Talking to many organisations there is a feeling that evidence isn’t always accepted by funders but when attempting to understand whether those funders are acting appropriately we need to ensure that our evidence is robust. In the last article on this subject I looked at the issue of confirmation bias and how it weakens an argument and in this article I’ll look at how selection bias also has an impact.

Selection bias is often called “sample bias” and is a fairly literal description. If the sample you are basing your research on is skewed then any conclusions drawn may be mistaken. Often, the selection bias in evaluation I see occurs where the client group researched does not match the standard population. This can be a particular problem with voluntary organisations who will only work with people on a voluntary basis.

Let me give an example:

We know that over 60% of young men leaving a prison sentence of lower than one year will tend to be reconvicted within 12 months. An organisation working with such offenders may be able to demonstrate that the client group they work with has an reduced reconviction rate of say 40% but this only makes a convincing case for that organisation if they can demonstrate that they’re working with a broad range of offenders that are representative of the wider population.

If the organisation only works with offenders who are willing to change and want to work on their behaviour then they may be more likely to select clients to work with from the 40% who wouldn’t reoffend naturally. Their success may be less down to their work and more related to their ability to identify those who would naturally address their behaviour. 

And of course a sample bias works both ways. You may find that the group you work with fares no better, or sometimes even worse, than an representative sample of the wider population. However, if you are working with people who face more significant challenges on average or have complex needs then this is a significant improvement in performance.

This may not just have an impact on an organisations ability to make a robust case but could create some significant liabilities, especially in delivering payment-by-results contracts.

So if we take the example organisation above – if it accepted a PBR contract on the basis of its ability to reduce reoffending from 60% to 40% but instead of the group it was used to working with a prime contractor instead referred a group of individuals that were more likely to reoffend then there would be significant problems in achieving previous performance and a likely risk of losing money on the contract.

The key to addressing sample bias is to understand if the client group you work with may be different to an average sample of the wider population. In many cases, detailed research into the group of people you work with may do more to strengthen your case than just the evaluation of that work.