How to measure attribution in impact evaluations

In a recent Catalyst Support and Training circle meeting, I mentioned that I had seen an increasing interest in ‘impact evaluations’ over the last year. I thought it would be helpful to explain what is generally understood to be an impact evaluation, and what it involves.

Cause and effect – what caused the car to crash?

In the picture below you can see a car crashed into the lamp post. There are four steps that led to the crash and the story is more complex than at first glance. If you want to look at the clues and have a guess at what happened you can find the results at the end of this article.

In trying to understand why the car crashed we are untangling cause and effect and looking to piece together an accurate story of what happened.

Contribution or attribution?

It's similar in the social sector. We want to untangle cause and effect and understand the extent to which the services we run lead to changes in society (impact) - people and their communities.

Let’s say that you are running an activity that helps young people to get into employment. You might measure that 68% of your participants enter into meaningful long-term employment or education. It is important to know to what extent this outcome has occurred as a result of your service. This aids your learning and helps you account back to funders and stakeholders.

The terms to the right can be helpful here:

‍Contribution - you can say you contributed to the change but you are not sure to what extent, and

‍Attribution - you can say more exactly to what extent your activities contributed to the change.

Why focus on attribution?

Social change can often take place within a complex ecosystem. The map was developed to understand the range of different factors that influence obesity levels and how they interact. We want to understand our particular role in this ecosystem, so that we know if our activities are effective.

This relates more to attribution than contribution, as expressed in the quote here:

‍“There are two reasons why it is a mistake to emphasise contribution. Firstly, it’s far too often used as a get-out clause. The phrase “we need to assess contribution, not attribution” is typically used to mean “Something good has happened. We want to imply that it was thanks to us, without trying to work out exactly how.” Even if you’re assessing contribution, you still need to understand the extent to which you contributed, and the process through which this happened. Of course, the contribution experts understand this. All too often, however, their disciples just use contribution as an excuse to avoid doing any actual thinking.” - Aidleap

Attribution and impact evaluations

Impact evaluations are one of many different types of evaluation.  This definition from the American University in Washington is a common one:

“An impact evaluation relies on rigorous methods to determine the changes in outcomes which can be attributed to a specific intervention based on cause-and-effect analysis. Impact evaluations need to account for the counterfactual – what would have occurred without the intervention”.

So impact evaluations look to measure both:

• outcomes
• the extent to which these outcomes can be attributed to a particular service.

How can we conduct impact evaluations?

Impact evaluations can be the most comprehensive and expensive type of evaluation. Particularly if using a Randomized Controlled Trial (RCT). You might have heard of RCTs in relation to drug trials: one group is randomly selected to receive the drug (the intervention or treatment group) and results are compared to another randomly selected group (the control group) that does not receive the intervention.

RCTs are also used in the social sector. One group of participants accesses a service, and another doesn't. Both are measured and the results are compared. This is an example of a counterfactual approach:

‍“A counterfactual approach involves developing an estimate of what would have happened in the absence of a programme or policy, and comparing this to what has been observed in the presence of the intervention.” - UNICEF

But counterfactual approaches are not the only way to conduct impact evaluations and measure attribution. Nor are they suitable for every situation or budget.

Smaller steps you can take towards measuring attribution

There are alternatives to RCTs: 

1. Developing a theory of change

‍Developing a theory of change is a great starting point for measuring attribution. It lays out your theory of:

• how your activities lead to outcomes
• how the outcomes lead to each other
• the role of other stakeholders in achieving those outcomes.

You can then look for evidence across the chain of outcomes to prove (or disprove) your theory.

In their guides on impact evaluations UNICEF describe the use of a theory of change to explore causal relationships as one of the ways to conduct an impact evaluation.

Theories of change can also be used in combination with an RCT as it explains the theory that the RCT will test.

2. Ruling out alternatives

Finding out about similar products and services. One step you can take towards measuring attribution is to understand the extent to which those accessing your service are accessing services similar to yours.

For example if you are the only provider of employment programmes to young people in your area, it can be easier to attribute back to you. Note: this is much harder with ‘softer’ outcomes such as increases in confidence.

For example, you can explore this through a survey question to service users and/or desk research.

Wrapping up

Measuring attribution can be challenging. But it doesn’t need significant time and expense to take the first steps towards it. And it can help you understand the role of your service within a wider system.

If you can ‘walk the talk’ and show that you are making an attempt to measure attribution (and you can show you understand the importance of attribution) it can help you attract funding.

Answers to what happened to cause the car to crash