Key considerations when evaluating counterterrorism strategies

Evaluating counterterrorism strategies is a daunting task. While both academics and practitioners acknowledge the value in being able to understand the impact of these strategies (Silke & Schmidt-Petersen 2015; Hoffman and Shelby 2017; Schmid & Forest 2018; Schuurman 2019; Brady 2021), their complexity, perceived scarcity of high quality and independent data, and methodological challenges means the field lacks an understanding of what works in counterterrorism. This article outlines some key areas that should be considered when attempting to evaluate counterterrorism strategies. 

1. Applying a holistic (or whole of strategy) approach

Counterterrorism is complex and multifaceted. Strategies need to protect the public in a range of ways and at multiple levels. While some valuable research has explored the various programmes and activities of counterterrorism strategies (Battersby, Ball and Nelson 2020; Lewis, Marsden and Copeland 2020; Nkata 2023), there is a need for research that explores counterterrorism holistically. By understanding how the different facets of the strategy function and interact, the impact of the strategy as a whole can be better understood.

Taking a holistic approach is very challenging in its scope. It requires an understanding of the range of data sources that might be relevant, and an appreciation of the variations in quality and applicability of such data. Additionally, a range of methodologies can be applied through qualitative, quantitative or mixed methods research. However, the value of the holistic approach should be apparent when public inquiries following a successful terrorist attack are considered as a holistic example of best practice and real-world impact.

The public expects a clear understanding of what went wrong following a terrorist attack, and this is best achieved through thorough investigation of the agencies, individuals and strategies which were relied on in the response to such an attack. For this reason, it is usual for public inquiries to take place in the UK to better understand what happened and provide transparency to the event. These inquiries typically involve interviewing the individuals involved in responding to terrorist attacks as well as those involved in preventing them. The UK’s counterterrorism strategy, CONTEST, consists of four pillars (known as the four Ps): Prevent, Pursue, Protect and Prepare. A wide range of actors operate within these four pillars – first responders, security firms, transport agencies, counter-radicalisation teams, the intelligence community to name a few – and following an attack, failings need to be identified, best practices highlighted, and lessons identified for onward learning. If this multi-faceted approach is applied to understand individual terrorist attacks, it makes sense that understanding the effectiveness of counterterrorism strategies also requires a holistic approach.

2. Quantity and quality of data

Given the public perception that counterterrorism relies on intelligence and classified data, there is often an assumption that publicly accessible data is limited. This is not necessarily the case. A significant amount of useful, public facing data is available which can assist in the evaluation of counterterrorism strategies. However, it remains important to interrogate the quality of this data. While government reports might provide updates on progress and may include numerical data in support, this data is not always contextualised. For example, when a government states that it has prevented a certain amount of people from travelling to conflict zones as foreign terrorist fighters, they might omit the wider data relating to those who did manage to travel, which makes it difficult to know if the government figures are actually good and an indicator of effectiveness. Questions around bias within the data and the rigorousness applied to methodological approaches also exist. Some of this is understandable given the range of policies and government departments which generate and present such data, and certainly there are also examples of good analysis and data quality in some instances. However, in areas of counterterrorism dominated by industry, the complex and sometimes unsystematic way that this data is collected and presented can make information more challenging to understand and analyse effectively (Brady 2021).

Transparency in the data selection process is essential to address some of these issues and there needs to be a clear purpose for analysing the data in order to evaluate counterterrorism. In other words, each variable explored needs to contribute actively to an overall understanding of the effectiveness of the strategy. The challenge is in treating the various kinds of data to make it comparable and useful to the goal. Both quantitative data (numerical) and qualitative data (text/spoken word) are equally important in creating a full picture of the issue and so a mixed methods approach is warranted. For example, numbers are measurable and useful when trying to understand increases and decreases in terrorism-related arrests and prosecutions, but without the context which can be provided in text sources, the value of the analysis and thus the evaluation itself is limited.

3. Identifying metrics and measurables - can AI help?

A key issue that’s needed to explore the effectiveness of counterterrorism strategies relates to valid and reliable metrics. This is particularly the case when taking a whole of strategy approach to evaluation. Ensuring that metrics are measurable and meaningful is a challenge and requires the application of a variety of tools. One such approach would be the use of a research team or expert panel of researchers (Feller 2013; Cameron and Hynes 2024). Another would be to use AI to more efficiently explore the options. Using AI to identify metrics, the applicability of these metrics to the relevant topic, and the sources which might supply the data for these metrics allows the researcher to explore such a broad and complex strategy in a more effective way (this researcher is currently exploring this area). It cannot replace the final decision-making around evaluating counterterrorism strategies, but it can help support and structure the analysis.

...following an attack, failings need to be identified, best practices highlighted, and lessons identified for onward learning.

The criteria for appropriate metrics depends on the type of counterterrorism strategy and the overall focus of the research. Taking a thematic approach is useful where key characteristics can be identified based on qualitative analysis which may then be refined and enhanced through the use of AI. This enables the research to develop a framework for the analysis of counterterrorism at a strategy-wide level, at which point AI can help identify appropriate measurables (both direct and proxy) to best understand the issues at hand. Through such research, metrics relating to transparency, resilience and ideological principals can be identified and sources collected.

The UK’s counterterrorism strategy, for example, has the overarching goal of reducing “the risk from terrorism to the UK, its citizens and interests overseas, so that people can go about their lives freely and with confidence” (Home Office 2023). While many factors contribute to the effectiveness of counterterrorism strategies, the population’s belief that a government is working appropriately to safeguard against the risk of attack is also important and yet examples of research that combines public perspectives with hard data is elusive. Identifying useful metrics in some of these areas is challenging. However, an example of the category of ‘public trust and confidence’ could be measured through public opinion polls or surveys which explore public approval of counterterrorism policies or public perception of fairness in relation to counterterrorism operations.

4. Clarity of outputs of the research: who will use the results and how

Finally, it is important to understand what outputs are required for the potential audiences of the evaluation. Given the complexity of counterterrorism strategies, different outputs are likely to be relevant. Academic papers and contributions to academic conferences and wider research projects are critical to ensure in-depth theoretical and methodological approaches to the topic are explored and provide enhanced understanding. At the same time, easy-to-digest reports with actionable findings in the short and medium terms, as well as presentations to key groups is another useful output. The development of frameworks for evaluating counterterrorism which can be used by practitioners and academics, datasets which can be accessed by the public and potentially a global counterterrorism index which can be produced either annually or bi-annually to explore, compare and evaluate counterterrorism strategies globally are examples of some relevant outputs for all audiences.

The research on evaluating counterterrorism strategies needs to be targeted at multiple audiences in order to have the best impact. Therefore, the research needs to have real world and tangible impacts with clear and actionable outcomes and recommendations. This approach will help ensure that not only do we understand the effectiveness of counterterrorism strategies, but that we will also be able to identify lessons to be learned and best practice going forward. Keeping the lines of communication open between what has traditionally been seen as two very different audiences and understanding the intersection points will be important in enabling this. 

These key areas for consideration when evaluating counterterrorism strategies are clearly challenging but provide significant opportunities for future research. Being open about these challenges and sharing best practice will ensure that the phenomenon of counterterrorism, the terrorist threat it addresses, and the wide range of tools available to best explore this area will contribute to transparency, accountability and learning going forward.