3. Developing a Methodological Framework for Analysis in:

Christiane Suchanek

Digging into Chaos, page 27 - 38

Security Sector Reconstruction and State-Building in Afghanistan

1. Edition 2018, ISBN print: 978-3-8288-4187-1, ISBN online: 978-3-8288-7080-2, https://doi.org/10.5771/9783828870802-27

Series: Tectum - Masterarbeiten

Tectum, Baden-Baden
Bibliographic information
27 3. Developing a Methodological Framework for Analysis The study aims to elucidate the contribution of security sector reconstruction to state-building in Afghanistan. Under examination is therefore the specific security sector in the country in order to identify the impact and prospects of – and on – state-building efforts. In doing so, the study follows the OECD’s call for the development of an “in-depth understanding of SSR processes and sector-specific reform needs” (OECD 2008, 23) in order to inform and review international actions. Due to the complexity of the agenda, this requires the creation of a methodological framework on how to analyse and compare such processes in a specific context like Afghanistan. With regard to the theoretical foundations presented, this chapter discusses the methodological considerations to SSR assessment. Subsequently, a conceptual framework is developed to facilitate the case study’s feasibility and comparability for a set of selected evaluation criteria. 3.1. A Closer Look at Methodological Implications In social sciences, the first step of diving into research is to decide for a qualitative, quantitative or mixed-method approach depending on the research question. In order to assess the contribution of SSR to state-building in Afghanistan, a sole quantitative approach does not fit the need to assess the historic context of the country, the legacy and perceptions of the institutions in question, and the SSR-approach applied by the different lead nations. These points are highlighted by state-building theories as described in chapter 2. The general progress of SSR in Afghanistan, or public perceptions towards institutions are measurable with a quantitative study. This requires a solid data foundation (including a baseline measure) as well as carrying out 28 surveys with Afghan people in the field. Such data is, however, hardly available for conflict-affected areas. Barton (2010) presents that this is a manageable challenge, yet the studies presented for Afghanistan are largely overrated considering the recent developments. Moreover, the availability and quality of data in conflict-torn countries is oftentimes incomplete as data collection mechanisms fall short in times of combat and even beyond (OECD 2008, 73-74). For example, in Afghanistan the latest census data dates back to 1979 and was described as unreliable at that time (Barton 2010, 26). On the other hand, there is broadly available literature published by Afghanistan experts (such as Antonio Giustozzi), governmental bodies, and non-governmental organisations or research institutions focusing on Afghanistan (as the AREU). Hence, certain statistics are collected and analysed, but they still require careful handling and critical reflection. On this account, the study makes use of an extensive qualitative document analysis as its primary approach for assessing the contribution of SSR to state-building in Afghanistan. At the same time, the analysis is complemented – in other words triangulated – by existing quantitative research for gauging the progress of SSR and statebuilding (e.g. capability ratings). In a broader view, the Bertelsmann Transformation Index provides aggregated data on state transformation complemented by country profiles (Bertelsmann Stiftung 2016). As an external evaluation of a reform agenda, the study is subjected to certain advantages and limitations. First, the outside evaluation of SSR in Afghanistan is not affected by regulatory restrictions. Additionally, the analysis is not linked to any of the programmes yielding politically independent results (OECD 2011b, 46-47). On the other hand, this external analysis cannot draw on first-hand experiences or self-conducted field research. Hence, the study relies on observations and findings published by others. In order to mitigate this drawback, the sources of information are transparently reported. Given the complexity of the agenda, it is not possible to cover all aspects and cross-sections of SSR and state-building. To moderate the risk of omitting central contextual information, the analysis orientates at analysis guidelines provided by the OECD’s Handbook on Security System Reform (2008, 2011b). The guidebook presents a theoretical and practical footing for security sector reconstruction and evaluation. It evaluates experiences from former state-building approaches and presents guidelines for assessing the past, current and future SSR activities. For the purpose of this study, the handbook presents methodological guidance for conducting SSR assessments including analytical categories, evaluation criteria and exemplary questions (OECD 2008, 41-57, 241-242). In 2011, the assessment of SSR activities has been updated and published in an extended manual for monitoring and 29 evaluation (OECD 2011b). With this, the OECD DAC serves as a central point of reference in the international development and state-building community (Schmidt 2013, 285). This does not automatically guarantee a sound analysis, but it enhances reproducibility and transparency with regard to the research procedure. It is, however, important to acknowledge inductive fallacies concerning the transferability of results, i.e. it is hardly possible to conclude from the particular to the general (Blatter, Janning and Wagemann 2007, 185). Yet, the single case analysis of SSR in Afghanistan does not aim at the generalisability of country-specific findings (e.g. the historical or social context), but at a thorough comprehension of the processes studied. This in turn adds to the general SSR and state-building debate beyond Afghanistan and facilitates future research. The preceding explanations illustrated that the research method combines inductive and deductive approaches. The basic conception of the study is thus related to the conceptual framework analysis of policy research (Blatter, Janning and Wagemann 2007, 30). Following its eclectic logic, a set of evaluation criteria, categories and guidelines serve as our analytical framework (ibid.). Designing this framework is rather based on an inductive approach; its application on a deductive logic. The explanatory as well as exploratory character of the study reflects the complexity of SSR and the need to establish comparable theory-based analysis. However, this requires in a next step to clearly set out the research focus, the evaluation criteria, and resulting steps of analysis. 3.2. Developing a Conceptual Framework for SSR Impact For developing the conceptual approach to SSR assessment, it is necessary to specify the research objects of the study. While acknowledging the complex interdependencies within the security system of a country, the evaluation concentrates on two major suppliers of state security: the police and the military. These two institutions have been selected because they are highlighted as core providers of state and citizen security, e.g. by Sisk (2013, 87) or Hänggi (2010, 87). Furthermore, the police and the military are two critical bodies at the top of international reform agendas when it comes to regional stabilization or transformation in the wake of war (Sisk 2010, 70). The selection of the two research objects serves heuristic considerations as it allows for a focused study within a convoluted field. At the same time, the sector- 30 specific assessment enables the observation of intersections to the aforementioned multiple security or development related features and actors, especially the involvement of the local civil society.21 Moreover, the approach facilitates a comparison between two related reconstruction agendas in the same regional context, but (originally set up to be) led by different stakeholders. In Afghanistan, the United States headed the military reconstruction, whereas Germany took the lead for police reform (Hammes 2015, 279). The basis of different lead nations facilitates the juxtaposition of chosen strategies, priorities, programming and normative frameworks. It also takes into account critical remarks to state-building such as the impact of foreign engagement – more explicitly, the unintended consequences on various sectors (Sisk 2013, 100). While the police and the military constitute two distinct bodies including their very own legal framework and reform needs, a comparison of the reform activities promotes an understanding of common problems and objectives (OECD 2008, 16). This meets the claims of forging a more integrative approach to SSR and its assessment without ignoring context-specific particularities. Up to this point, the main challenge of SSR and state-building assessment has been emphasised a few times: its complexity. In order to examine the developments in Afghanistan since 2001, it is necessary to divide the reform agendas into manageable units that comprise the impact of SSR activities on state-building in the country. For evaluating the development programmes, the OECD DAC established five standard assessment criteria in 1991: relevance, effectiveness, efficiency, impact and sustainability (OECD 2008, 242). With the 2011-update of the handbook, they have been supplemented and expanded to eight criteria for SSR evaluation. The committee included ownership to sustainability in recognition of its role for the longterm success of measures (OECD 2011b, 49). Furthermore, they added coherence, coordination and consistency with values with respect to the manifold and interconnected SSR activities in target countries (ibid., 48-49). In the international community, the standard criteria are regularly applied for reviewing assistance programmes. They are used, among others, by the United States Agency for International Development (USAID; see Purcell, et al. 2016), the German Corporation for International Cooperation (Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH 2015), or the Kreditanstalt für Wiederaufbau (KfW 2017). Thus, these criteria provide an applicable and internationally comparable framework for the assessment of military and police reform in Afghanistan. In the following, 21 See section 2.3 31 Table 1 summarises the eight evaluation categories as an overview of our assessment scheme. Evaluation Criteria Definition in the Context of SSR Points of Consideration 1 Relevance / Appropriateness The extent to which the SSR activity is suited to the priorities and policies of the target group, recipient and donor Addressing main security threats and needs of the state and the citizens Based on up-to-date context assessment Consistent with domestic system Technically adequate solution Balance of long-term and immediate / short-term considerations Inclusion of non-state actors / mechanisms Responsive and adaptive to changing circumstances Consistent with donor policies and priorities 2 Effectiveness The extent to which the SSR programme achieved its objective Match with intended outputs, purpose and goal Changes resulted from international intervention or other factors Reasons for the results (delivery or nondelivery of specified objectives) Recommendations or options to make intervention more effective 3 Efficiency The extent to which outputs – qualitative and quantitative - of the SSR initiative have been produced by the least costly inputs in order to achieve the desired results Efficiency aspects taken into account Measures during planning and implementation to ensure efficient use of resources Outputs efficiently delivered (options to reduce resources without losing quality and quantity of outputs; to increase output with the same amount of resources) Different type of intervention more efficient in terms of costs but with same results Was a worthwhile intervention given possible alternative uses of the resources 32 4 Impact The positive and negative changes produced by SSR, directly or indirectly, intended or unintended Effects of the intervention on people and institutions (intended/unintended; positive/ negative) Effect on well-being of different groups (women, men, boys, girls, etc.) Perception of the effects of the intervention by insiders / affected people Strengthening of institutions (capacity, accountability) Development and improvement of relevant policies due to the intervention Identification and measurement of changes Attribution of changes to intervention 5 Sustainability & ownership A measure of whether the benefits of SSR are likely to continue after donor funding has been withdrawn with special emphasis on the involvement of locals Access to justice and security over longer term by the population (creation of processes, structures and institutions) Human & institutional capacity Consistency between donor's priorities and domestic effective demands / needs Support by local institutions (Well) integration into local social and cultural conditions Management and leadership of reforms by locals Participation of partner country stakeholders in the planning and implementation of the intervention Appropriateness of goods, services and technologies provided appropriate to the economic, educational and cultural conditions in the partner country Maintenance of benefits after withdrawal Credible exit strategy envisaged or in place 6 Coherence Whether the support and initiatives of Cooperation within individual donor governments (or between if one programme) "Whole-of-government" mechanisms in place to support SSR 33 different international actors and departments are operating together in a coherent and strategic fashion Collision (tensions) or reconciliation of donor priorities: "hard" security issues (e.g. counter-terrorism) versus development-style SSR objectives 7 Coordination / Linkages The extent to which the SSR activity incorporates and promotes the coordination between different parts of the security system and government strategies in the country Steps / Measures to forge strategic engagement across the security system on different levels Forge of links between different programmes, frameworks, plans and operations (peacebuilding, development, recovery, poverty reduction, gender equality, etc.) Consistency and complementarity with activities supported by different donor organisations 8 Consistency with values The extent to which the SSR programme has promoted key principles and values in security governance (e.g. democratic governance, respect for human rights, respect for the rule of law, transparency, accountability) Promotion of norms of good and democratic governance, respect for human rights and the rule of law Intervention designed and carried out in accordance with basic governance principles of transparency and accountability Promotion of equitable access to justice and security for populations Table 1: SSR Evaluation Criteria (Source: Adaption based on OECD 2008, 241-242, 2011b, 48-50) 34 This catalogue of evaluation criteria facilitates the analysis by providing a limited, albeit wide-ranging set of critical dimensions. Reviewing the eight categories, they cover the essential aspects of state-building and SSR programmes which have been outlined in the theoretical sections 2.2 and 2.3. Yet, it has to be acknowledged that these criteria overlap in some aspects. For example, effectiveness enquires the achievements of programme objectives, though this could also be incorporated under “impact”. This category observes, in a broader sense, the changes occurred in relation to the reform initiative. Still, the criteria forge an adequate framework for reviewing reform activities which facilitates the comparison of such complex and diverse agendas. For evaluating the reform activities with the eight categories, a conflict profile is crucial as a first step of the case study, provided in section 4.1. This revision is essential to provide basic insights to the underlying drivers of conflict, such as the social, political, regional and international roots of security threats (OECD 2008, 50). The conflict portrait lays the foundation for central assessment aspects. For example, the appropriateness of actions can only be gauged against country-specific circumstances. The conflict analysis further reveals conflict parties, leading stakeholders, and developments on the ground to which the security sector reform and actors are expected to adapt (Schnabel and Born 2011, 6). Moreover, it outlines the structures of the security system in the country, i.e. the main state and non-state provider of security which constitute central components of SSR according to the OECD (2008, 22). As a second step, the case study examines separately the police and the military reform in the country since 2001 (section 4.2. and 4.3.). First, the analysis reviews reform activities to give a topic-based overview. That includes the entry point of reform, key plans and measures, actors, and developments. Then, the eight evaluation criteria appraise the particular reform agenda, especially in view of external assistance. With this, main reform achievements, failures, challenges, and options, hence lessons learned from the engagement are concluded. The study does not list in detail each and every activity of the military and police reform in the country as it should provide a condensed assessment of the overall process.22 The main focus will be laid on thematically structuring the advisory efforts which facilitates the application of the evaluation criteria. For this purpose, the categories for reconstruction are graded by a 22 Such a detailed overview including the broad historical context has already been compiled in the publications of Giustozzi and Isaqzadeh (2013) concerning the police, and Giustozzi (2015) for the army of Afghanistan. 35 model the KfW uses in its ex post evaluation reports (e.g. in KfW 2015). The KfW applies a six-point rating scale to the projects ranging from “very good” results (level 1) to no impact of or a deteriorated situation caused by the project (level 6). For “sustainability”, however, the bank introduces a four-point scale ranging from “very good” to “inadequate sustainability” (KfW 2015, 7). This rating system presents a rather qualitative assessment as it employs a nominal scale without clear benchmarks. On the plus side, it provides a useful rating tool in order to assess the reform agenda and its outcomes. The KfW (2017) insufficiently explains the application of a separate scale for sustainability.23 The special scale complicates the comparison of results and calculating an average score for a reform. The KfW does not adjust the different scales in this calculation, but accepts the different weight for the four-point sustainability scale. Despite this complication, the paper follows the KfW’s approach for the sake of comparability to its assessments and the limited scope to develop a diverging rating scheme. On the positive side, the scalebased system enables a graphical presentation of the ratings, thus facilitating the comparison of results – with the limitation of the four-point scale to be included in the figure. In order to give an overview of the rating scale applied to the reform process, table 2 summarises the KfWs evaluation scale. Rating Rating Description Four-Point Scale for Sustainability Level 1 Very good result that clearly exceeds expectations Very good sustainability: The developmental efficacy of the project (positive to date) is very likely to continue undiminished or even increase. Level 2 Good result fully in line with expectations and without any significant shortcomings Good sustainability: The developmental efficacy of the project (positive to date) is very likely to decline only minimally but remain positive overall. (This is what can normally be expected). Level 3 Satisfactory result – project falls short of expectations but the positive results dominate Satisfactory sustainability: The developmental efficacy of the project (positive to date) is very likely to decline significantly but remain positive overall. This rating is also assigned if the sustainability of a project is considered inadequate up to the time of the ex post 23 The KfW states: „For the criterion of sustainability we only use a four-point scale, which mainly reflects the anticipated future trend (albeit with a certain degree of uncertainty). A score of 4 indicates ‘insufficient sustainability’” (KfW 2017). This description, however, does not provide an explanation for abandoning the six-point rating for sustainability. 36 evaluation but is very likely to evolve positively so that the project will ultimately achieve positive developmental efficacy. Level 4 Unsatisfactory result – significantly below expectations with negative results dominating despite discernible positive results Inadequate sustainability: The developmental efficacy of the project is inadequate up to the time of the ex post evaluation and is very unlikely to improve. This rating is also assigned if the sustainability that has been positively evaluated to date is very likely to deteriorate severely and no longer meet the level 3 criteria. Level 5 Clearly inadequate result – despite some positive partial results, the negative results clearly dominate Level 6 The project has no impact or the situation has actually deteriorated Table 2: Rating Levels for Project Evaluation (Source: Adaption based on KfW 2015, 7) In a third step, these assessments are merged for estimating their impact on the overarching state-building venture in Afghanistan (section 4.4). Having said that, there is no “end state” of statehood to measure a state’s development against as all states are subject to constant change (Sisk 2010, 58). State performance research is incorporated to assess the functionality of state institutions, for example by the Bertelsmann Transformation Index (Bertelsmann Stiftung 2016). Such data adds to the overall assessment of state-building activities in a country. Still, it is necessary to clarify the key dimensions for impact assessment. Sisk (2013, 53) suggests four categories: autonomy, authority, legitimacy and capacity. Autonomy describes the freedom from foreign control or narrow partisan interests. Authority means the sole and rightful use of coercive power. Legitimacy in the sense of Jellinek24 refers to the internal and external perception of the right to rule. The concept of capacity captures the state’s ability to plan and provide certain services to its citizens – e.g. security and justice (ibid.). The World Bank (2012, 26) presents a similar set of dimensions in its tool-kit for state-building guidance. In contrast to Sisk, however, the autonomy dimension is not separately listed. It is 24 See section 2.1.1. 37 implicitly covered by the other three categories, particularly by state authority as it demands autonomy for making and implementing decisions on its own.25 Therefore, the impact of SSR – with regard to military and police reform efforts – on state-building is gauged by appraising the three dimensions authority, legitimacy and capacity as suggested by the World Bank (2012, 26). Subsequently, the results of the case study and the methodological approach are discussed, reflected and concluded in the final chapter. By this means, the analysis is of interest for stakeholders concerned, development workers and researchers, and even for the training personnel on the ground (OECD 2008, 25-26). Moreover, the study informs about the long-term nature of SSR, its limitations, and contemporary challenges to both political decision makers and the public. Analysing the foreign engagement in SSR in Afghanistan also enhances transparency and accountability in the donor countries. Eventually, the research approach and the answers presented aim to guide future investigations and international engagement. 25 Sisk (2013, 46) underscores that the aspects are oftentimes interconnected and sometimes even mutually reinforcing. He gives the example that the ability to govern in terms of authority is at times contingent on the consent of the ruled as an expression of legitimacy.

Chapter Preview



With the terrorist attacks of 9/11, international attention has shifted to Afghanistan. This event also marked the beginning of a fundamental transformation in the country: It was and is target of an enormous military campaign that toppled the Taliban in a “War on Terror”. With the previous government deposed, a new one had to be built. Guided by a multinational coalition, the Afghan state was quickly reformed, but instabilities and conflicts have not only been inherited but also unintentionally triggered. Even after more than 15 years of international intervention, neither peace nor political stability or security seem to have taken roots in the country. The book takes a closer look at the international state-building regime and evaluates the impact and outlook of such efforts in Afghanistan. In order to facilitate the analysis, the study concentrates on Security Sector Reform (SSR) with special emphasis on the reconstruction of the Afghan military and police. With this, the author presents critical insights into state-building prospects in Afghanistan and introduces an analytical approach that may also be transferred to other cases of state reconstruction.