Evaluating in complex policy environments - a practitioner’s perspective

Why evaluation is vital

Evaluation is no longer a nice-to-have buzzword; it is vital. Evaluation helps us to identify the risks and benefits with any chosen programme or policy, and it guides important decisions such as ‘Who will get the intervention and why?’, ‘How will the policy be funded, measured, and implemented?’. It therefore ensures accountability and allows organisations to respond to external scrutiny.

...there is a non-negotiable when it comes to evaluation irrespective of what approach you take and that is developing a Theory of Change.

There is lots of guidance available on evaluation types (e.g., process, impact, economic), what you should evaluate (e.g., policies, programmes, people), and the methodologies you should adopt (e.g., contribution analysis, randomised controlled trials, economic assessments).  However, in my opinion, there is a non-negotiable when it comes to evaluation irrespective of what approach you take and that is developing a Theory of Change. A term that is now part and parcel of our everyday language in policy development. But what is it and why do we need one?

A Theory of Change

A Theory of Change explains how and why a desired change is expected to occur in a specific context. It links what a programme or initiative does to its expected outcomes, addressing the 'missing middle.'

Traditionally, a Theory of Change will capture the inputs that allow delivery of a set of activities, the outputs of those activities, the short or medium term effects (outcomes) of these outputs, and the long term change or impact of achieving your strategic objective.

However, they can be criticised as being too linear and simplistic. Often used interchangeably with the idea of a ‘golden thread’, theories of change can be over reliant on the logic of a programme from start to finish. They often assume that one thing directly leads to another, but as is often the case in issues like homelessness, poverty, or radicalisation, it’s a little more complicated.

Evaluating programmes which are highly sensitive and sit across complex areas, have many challenges:

1. It is hard to define success in counterterrorism and prevention. Measuring things like the number of prevented attacks or the reduction in radicalisation is tricky and often lacks clear standards.
2. Reliable data is often hard to access due to security concerns, and the data available can be inconsistent, making it tough to draw solid conclusions.
3. Terrorism and violence are influenced by many social, political, and economic factors, making it hard to isolate the impact of a single action or intervention.
4. Counterterrorism policies are often rushed due to public and political pressure, leading to actions based on untested assumptions and lacking thorough evaluation.
5. The effects of counterterrorism measures may take years to show, complicating the evaluation process. Short-term assessments might not capture the full impact or genuine behavioural change.

How to deal with this complexity – top tips from a practitioner

Practitioners operating in these hard-to-measure policy areas will always need to demonstrate the impact and effectiveness of their programmes and policies, and it is likely that they will continue to need independent support to do so.

Do not underestimate the power of a Theory of Change as a tool to help bring practitioners, academics, and policy makers onto the same page

Here are my top tips for individuals who are involved in evaluation projects:

1. Do not underestimate the power of a Theory of Change as a tool to help bring practitioners, academics and policy makers onto the same page.
2. Consider using a systems thinking approach. This allows us to look at the connections between different factors that could affect policy results.
3. Impact may not be defined to one area. For example, counterterrorism and security policies might also impact public safety, civil liberties, or international relations. Defining the scope and boundaries of evaluation is key to understanding the full extent of a policy’s effect.
4. We need flexibility in complex policy areas. The unpredictable nature of social systems means that evaluation methods must allow for adaptation as new information becomes available or as conditions change. This is why early and continuous monitoring will help provide valuable feedback loops. Do not be afraid to go back and adapt.
5. Engaging with a range of stakeholders is essential – they provide critical insights into how policies are functioning on the ground, including whether there are any risks or unintended consequences that may be emerging. This is particularly important when dealing with vulnerable groups or communities. For example, a well-intended policy might inadvertently harm a particular group or sector.
6. Think like a practitioner. We will always ask the ‘so what?’ as well as the ‘what next?’. Don’t wait to be asked!

Can you help demonstrate this through a Theory of Change, articulate the complexities in evaluating, and provide options for evaluation? As a practitioner – if you give us the moon on a stick – we will take it, only to be disappointed that it was not actually feasible or realistic. So be honest.  We know evaluations can be hard, but that doesn’t mean we shouldn’t give them a go. We need to build up our understanding of what works in these complex, hard-to-measure areas, so that we are on the front foot to develop proactive policies that have real-world impact.

The views expressed are those of the author, and not necessarily those of DSIT.