An evaluator’s view of Catalyst

For eight months, Tom Keyte of inFocus has been leading Catalyst’s learning and evaluation work. He and his team have talked to every initiative lead, and heard from hundreds of participants. We caught up with Tom to ask him a few questions.

What have you and inFocus been doing? 

Tom: I would break it down into different components:

• Looking at what has worked/not worked and the emergent outcomes
• Looking at the learning that has emerged
• Building capacity - helping the Catalyst team develop better indicators and measures of success

It’s been both enjoyable and challenging in equal measure. Evaluators usually approach a project with a fixed structure - plans and tools they stick to with very little deviation. But Catalyst needed a more iterative approach, because the work was developing so fast as we were evaluating it. We basically had to take an agile approach, which I have really enjoyed, and we’ve been given the room and flexibility to explore different aspects of Catalyst. 

At the same time it could be overwhelming at times. It's been difficult to keep track of everything happening, and building a coherent picture has been challenging. As we come to the end I’m in need of a holiday! But I’m also exhilarated by the fascinating experience of evaluating this type of work. 

Take a step back and tell me about one big insight you’ve had into Catalyst

Tom: Ha! I’ve got 50 pages of insights. But let's focus on just a few.

1. The sheer scope of support to meet different charity needs

I was impressed by the scale and range of ways Catalyst worked to meet charities' needs, especially in response to the pandemic. This included lighter touch services such as Digital Candle, to six-figure funding programmes. Like: wherever your digital need is, we can help!

We sliced them in this way: 

• Grant making schemes
• Training support options, including advice services
• Data and encouraging the use and sharing of data
• The reuse projects and sector level impact
• Collective aspects across everything e.g. network building projects

There is something there for everyone. However, charities didn't always understand the range of options available to them. That left me wondering whether it had stretched itself too wide. This phase of Catalyst ends with a strong suite of services and good research but uncertainty around next steps and how they might best be sustained. So, would it have been better to do less? I don’t know. Their response to the pandemic would probably have been weaker if they had and the need was clearly demonstrated.

2. Reuse, reuse, reuse!

Another big insight was around the amount of reuse that took place. The principle of reuse is about reusing existing solutions and only doing what you need to do, even if you are on a funded programme. It's like taking a step away from developing things that aren't needed, by testing and using existing things first.

Reuse came out as a strong principle within the core Catalyst team and initiative leads. They demonstrated an attitude and approach that others could follow. They gave people permission to think more confidently about reusing others’ work. 

It also showed in some agencies’ approach, where they took the attitude that reuse should be promoted ahead of developing something new (if this was the better option for the charity). At the same time Catalyst funded agencies to help charities work out what they needed, respecting that it can take time to work out what you need and that its okay to take the time to find out you need something that already exists. 

3. Collaboration

Watching collaborations between agencies and charities was interesting. They built strong relationships where learning flowed both ways. I found myself hoping that some of these might last because there was so much good work to build on. 

We also saw good feedback about collaboration in those initiatives that had a big focus on this (e.g. Sector Challenge programme). We learnt where it went well and also the challenges it raised. 

4. Pressured timings

I watched how quickly Catalyst responded to needs arising from the pandemic. For example, their Definition programme - which was rapidly formed and very highly rated. And the number of experts who offered advice through Digital Candle (nearly 350!) It was impressive.

At the same time, the need to rapidly develop and deploy programmes led to feedback from some charities that grant making programmes were too short or required too fast a response time. They consistently said it would have been good if each programme was 2-4 weeks longer. 

Can you tell me about something that was very different or unexpected to your past evaluation experiences?

Tom: Yes! There were some cultural differences. At first I didn’t understand why people waved their hands in the air at Catalyst meetings. Then I worked out that it meant they were in agreement with the speaker. Then Suraj sent me a guide to video call hand signals. I enjoyed finding out about them but I am not quite sure I adapted fully to the hand signals. 

I also like how Catalyst does check-ins at the start and end of calls. Like temperature checks on how people are doing. I’m wondering if that’s good practice coming in from the digital sector as it's less familiar to me working in the charity sector more generally.

What is Catalyst to you?

Tom: That's difficult to answer because I've heard so many different answers! I also know I’ve been tormenting people with that question. I’ve read 25 answers from Catalyst leads and 91 from charities. It's hard to shut out 116 voices and come up with my own definition!

Catalyst is a young, emergent force that hasn’t yet worked out what it is. After listening to people’s answers I would frame mine around the building of digital capacity in the charity sector and improving efficiency and effectiveness of how charities deliver their work which in the end impacts on beneficiaries.

I’ve heard others frame it according to their interests. For example, people interested in reuse would give a definition around reuse and people from a network would say it's about driving collaboration. 

For me, perhaps because my work revolves around supporting charities to measure impact, I would simply explain it in terms of end impact rather than how it happens. And that end impact is about how it improves the lives of people through better digital understanding and services from charities.

Thanks Tom, we’ve really appreciated having you with us on the journey. You and inFocus are part of Catalyst now!