Evaluating Terrorism and Targeted Violence Prevention Programmes: From What Works to How Programmes Work

Despite a growing body of research evaluating CVE programmes, practical and ethical challenges mean that it is rarely possible to use experimental and quasi-experimental methods that demonstrate programme effectiveness (Lewis et al., 2024). Realist evaluation provides a way of addressing these challenges by reorienting evaluation away from ‘what works’ to focus more clearly on ‘how’ CVE programmes work.

Realist evaluation

Realist evaluation makes the mechanisms by which programmes influence change more explicit and more open to evaluation (Gielen, 2020). In doing so, they make it possible to assess a programme’s contribution to a desired outcome by testing assumptions about how programmes work (Fisher & Busher, 2023). Realist evaluation also takes greater account of the contexts within which programmes are delivered than many other approaches to evaluation. Doing so helps programme managers and funders determine whether programmes might be transferred or scaled up across contexts and answer the question of ‘will it work for us’ in ways which, for example, randomised control trials cannot (Cartwright & Munro, 2010).

By developing and testing ‘context-mechanism-outcome’ configurations, realist evaluation makes it possible to determine whether and how programmes are contributing to outcomes by identifying:

• Outcomes produced by a programme - both intended and unintended.
• Mechanisms through which programme activities produce these outcomes.
• Contextual factors that impact how these mechanisms function.

This type of analysis can be operationalised at the individual level, to understand a specific client’s progress; at the programme level to determine if a particular intervention is working as intended; or at the strategic level to assess whether a series of projects are contributing to broader counter-terrorism goals.

Realist evaluation also takes greater account of the contexts within which programmes are delivered than many other approaches to evaluation. 

Case management processes

Assessing whether a programme can meet its ambitions is supported by breaking down the case management process into its constituent parts (see Table 1). Our Campbell systematic review of the implementation and effectiveness of case-management tools and approaches used in secondary and tertiary CVE intervention provides a way of organising this process (Lewis et al., 2024).

Table 1. Stages of the case management process (based on NCMN, 2009)

Realist evaluation and case management

Combining the structure of the case management process and a realist evaluation approach has a number of advantages that can help address the challenges facing efforts to evaluate CVE/ TVTP programmes including by:

• Providing a holistic way of understanding both how individual stages of the case management process function and assessing how the programme works as a whole.    
• Identifying gaps in provision and gaps in evidence across the case management system.
• Specifying how outcomes from one stage of the case management process can become contextual factors for subsequent stages.
• Clarifying the contribution a programme makes to observed outcomes by specifying the causal mechanisms at work.

Specifying mechanisms

The evidence we synthesised in the systematic review on what facilitates or creates barriers to the effective delivery of case managed interventions can help identify those mechanisms which have attracted sufficient evidence to warrant their inclusion in an evaluation design. This helps address the challenge of identifying which measures and metrics are most important to evaluate; something which is a perennial problem in CVE evaluation (Fisher & Busher, 2023).

Breaking down the mechanisms that are at work across the case management system draws attention to the different types of mechanism that are relevant to effective interventions. These include:

1. Relational mechanisms: Covering the relationships between the intervention provider or case worker and their client, and amongst members of the multi-agency teams and programme partners that help deliver the intervention.
2. Knowledge-based mechanisms: Relating to the knowledge that those delivering, managing or interacting with the case management system hold, for example, about how to refer someone to a programme.
3. Systems-based mechanisms: Referring to the availability of relevant tools, systems and structures to support the case management process, for example, whether validated risk assessment tools are available in a given context.
4. Methodological mechanisms: Concerning the methods that are used to undertake different aspects of the process, for example, whether data is triangulated from different sources to develop a more holistic understanding of an individual’s progress.

Next steps

In research funded by National Counterterrorism Innovation, Technology, and Education Center (NCITE), we’re developing a realist evaluation framework and toolkit that will provide a means of evaluating CVE/ TVTP programmes. The framework specifies the context-mechanism-outcome configurations at work across the case management system and at each stage of the process. In doing so, it will provide a holistic approach to evaluating how programmes work that aims to overcome the persistent challenges associated with understanding what works in CVE and TVTP practice.