Greater Manchester Centre for Voluntary Organisation

Evidence Based Commissioning: Confirmation Bias

On this blog we are going to continue take a look at some of the common problems with how evidence is shaping commissioning and how we understand our impact. The next in this series looks at confirmation bias.

Many of us believe that if we can present the facts about an issue we care deeply about we can change the minds of others. This can lead us to look for evidence that supports our point of view or evidence of the impact of our services in addressing the serious problems that we see.

There’s a danger that such approaches will be subject to confirmation bias and from my own experience this is seen in many research and evaluation reports produced by the voluntary sector. This bias is not deliberate but can often be a direct consequence of the desire to make a compelling case.

Confirmation bias is sometimes called “myside bias” – it’s the tendency to look for facts which reinforce a belief and can cause you to not notice or even ignore information that presents a challenge to your point of view. Look through the newspapers (and especially comments sections below articles) and you can see people with a range of political views on any argument dismissing any evidence produced by those on the other side. This is often seen with discussions of crime statistics, where the statistics produced by the very reliable British Crime Survey can be dismissed out of hand by people who so believe crime is rising all the time.

There are two things we need to understand about confirmation bias. The first is that if the person who’s mind you want to change is emotionally committed to their point of view then a rational argument based on statistics isn’t going to cause them to give up that emotional stake – they’ll be likely to have some form of confirmation bias. Any service with a community setting will have some impact that's down to the environment someone lives in, with a range of external factors at play. A commissioner who is deeply sceptical of your work and who has such a confirmation bias, will look at any impact report and be able to find a way to express doubt about it. That doesn’t mean that they can’t change their mind but that you need to provide emotional information as well, understand why they are so committed and find some common ground on the issue that will enable you to have a discussion about the facts. The most important technique in persuasion is to first listen and understand motivations. You’ll need more than an evaluation report to do this – you’ll need stories and a human perspective first. 

The second issue is to understand your own confirmation bias. I’ve heard colleagues from a range of organisations make the statement “We know our interventions work – we just need the evidence to prove it to commissioners”. This is a real warning sign. If you undertake an evaluation on this basis or hire an evaluator to do the same you’re likely to miss some of those inconvenient truths that can be deeply important. This is especially the case if the sceptical commissioner you wish to persuade is aware of them and is able to dismiss your case because you haven’t addressed what to them are obvious weaknesses. I’ve seen many cases where an organisation does work with a range of people whose lives are improved but where commissioners are able to suggest that other factors have contributed to these changes because an evaluation has had too narrow a focus. Often this is because the provider is so certain they make the difference that they haven't looked at the case against.

Ultimately, the only person you can ever guarantee you’ll change the mind of is yourself. Evaluations are incredibly important when done well, as every organisation needs to understand how to improve its services and as a duty to beneficiaries it must check that the services it has so much faith in actually do work. Importantly, if you want to persuade a challenging commissioner then you need to be just as challenging to the service yourself.  When you develop an evaluation process as a marketing tool, rather than as an opportunity to learn more about the things you do, then you’re immediately reducing its effectiveness to actually evaluate. Good evaluations could and should form part of the public case you make and will result in better marketing but that should be seen as a secondary benefit rather than their prime purpose.