[ Evaluation ]

Effects measured objectively

Fragile statehood threatens not only the people of the country concerned but also the international community. It is clear that development and security are inter-related, but very little research has been done so far to measure precisely the effects of development aid on stability. An evaluation of the situation in North East Afghanistan provides some initial pointers.

[ By Christoph Zürcher, Jan Koehler, Jan Böhnke and Cornelius Graubner ]

As the fourth-biggest donor and third-biggest provider of troops for the International Security Assistance Force (ISAF), Germany is a major actor in the multilateral effort to rebuild Afghanistan. German understanding of reconstruction includes not just military and diplomatic support but also, crucially, development assistance. So far, the Federal Ministry for Economic Development and Cooperation (BMZ) has spent € 650 million on that purpose; the money has gone into areas such as sustainable promotion of the private sector, basic education as well as the supply of drinking water and energy, especially from renewable resources.

The evaluation focused on Northeast Afghanistan, the core area of German engagement, but the cumulative effects of development aid of all actors engaged are subject of the evaluation. On the basis of broad-based data, a team from Berlin’s Free University in cooperation with the BMZ evaluation department evaluated effects for the first time (see article on p. 110 for methodology). This article summarises the findings of the first representative survey.


Improving utilities

In spring 2007, we conducted a survey among 2,034 households. Respondents were asked to assess different actors’ contributions to the provision of basic services. The findings reveal a considerable need for action. Only nine percent of households get their drinking water from a pipe. The rest resort to water from wells, open canals or streams. Only around 25 % of households have electricity. Seven percent of respondents have difficulties getting food.

Aid programmes matter. Despite the dire state of infrastructure, all communities reported that they had benefited from development cooperation in the two years preceding the survey. Around 61 % of households believe that international agencies have contributed positively to drinking water provision, while 66 % consider them instrumental in improving roads. According to 47 % of the households development agencies have helped to raise the standard of schooling.

A gloomier picture is painted in terms of agriculture, electric power and employment. Only 16 % of respondents reckon that development projects have a positive effect on agricultural production; only 12.2 % see them improving access to electricity; and a mere 2.6 % think development agencies have helped to create jobs.


Little confidence in the state

An interesting finding is that Afghans feel their government’s contribution to progress in the areas mentioned is limited. 34 % of respondents think the government plays a role in improving the school system, but only 13 % of households credit it with responsibility for better roads. Six percent feel state authorities have a positive impact on agricultural production; five percent give them credit for improving drinking-water supply. Three percent state the government has helped improve to power supply and only 0.3 % think it has created jobs.

How little confidence Afghans presently have in their government is also revealed by the comments on how often district and provincial authorities respond to local needs. Only three percent agree to the statement that this happens regularly or invariably, whereas 31 % feel it is rare. A full 37 % expressed the view that district and provincial authorities do not care about the needs of the community.

A surprisingly high percentage of respondents approved of school attendance for girls. Only 1.9 % took the view that education for girls had a negative impact on local life. A large majority of households also agreed that it was a good thing to create more opportunities for men AND women to earn a living outside agriculture. That view is particularly prevalent wherever aid agencies have been active.
Enhanced security

The survey’s most surprising result is that 23 % of respondents stated that security had improved somewhat in the past two years, while 76 % mentioned substantial progress in terms of enhanced security. Both the presence of foreign troops and the government in Kabul were said to have contributed to this trend. On the other hand, part of the local population see traditions and Muslim values in danger because of the strong foreign presence. In this respect, development agencies were considered a threat by 21 % and foreign military by 43 % of respondents. Apparently, a reserved, cautious attitude towards foreigners somewhat reduces the positive effects of tangible and specific support provided by international agencies.

We also wanted to establish how development cooperation impacts on security, state legitimacy and attitudes towards international actors. We therefore looked at three issues of key concern to the international community: security, governance and public support for international actors.

In these fields, the responses from communities which claimed to have benefited much from development projects were compared with those from communities that felt they had not benefited much. Specifically, we sought to establish whether the former
- felt safer,
- saw the state in a more favourable light, and
- took a more positive view of international actors.

No correlation was found between development projects and people’s sense of security. And it should come as no surprise that respondents who thought they had profited from development projects more than most did not feel safer than others. From the survey and from open interviews, we know that people’s greatest fear is organised crime. What is more, perceived threats vary considerably from one district to another. In the short term, there is no direct correlation between community security and development projects.

Second, development projects have only a minor impact on public perceptions of the government. Whether communities have benefited from projects more or less than most others, they show an equal tendency to see the national administration as unhelpful. This suggests that development agencies have failed so far to promote state capacities at the local level.

The answer to question number three is an unambiguous “yes”. Development projects enhance the standing of international agencies among the people. Communities that had profited from development cooperation more than most generally saw international actors in a more favourable light.

The data available at present does not permit any statistically conclusive statement on causality. Did projects influence public opinion, or did development aid flow mostly to communities that already had a positive attitude towards international aid agencies? That question will be answered by the next survey, scheduled for 2009.

For various reasons, however, it can be assumed that development projects impact positively on public attitudes, and thus contribute generally to a more favourable environment for the mission. The geographical spread of development projects may show regional differences, but these can be explained by climatic and topographical conditions. There is no pattern to suggest that development projects were distributed on the base of any political bias. Furthermore, nothing in the interviews with aid-agency staff indicated that communities were selected or discriminated against according to such attitudes.

The results of this research are fairly positive. Perhaps more important, however, is the fact that despite a host of logistical and methodological difficulties, it is possible to analyse the impact of development projects even in conflict regions.

Broad data base

Suitable methods are needed to assess the effects of development projects aimed at improving security, conflict management and governance in crisis regions. To develop such methods, a three-year cooperation was established between the SFB 700 research centre at Berlins Free University (FU) and the Evaluation Division of the BMZ. The aim is to design a tool-set that, if used systematically by development agencies, can objectify the debate on opportunities, benefits and risks of complex international reconstruction missions. The methods were first tested by assessing impacts of development cooperation in the north-east of Afghanistan.

Our analyses are based on comparing groups – setting households who have profited from development projects against others that have not done so or only to a much lesser extent. If the groups are otherwise indistinguishable, differences can be interpreted as an outcome of the projects. However, such an assessment depends on sufficient data to allow for a statistical analysis; but such data is typically not readily available in crisis regions. It must be collected on the ground.

In a country like Afghanistan, doing so is a daunting logistical challenge, given the appalling state of the roads, the tough security situation and harsh weather. Moreover, development agencies on the ground often tend to be unwilling – or unable – to provide information. That does not make the task any easier. On top of that, many villages are not marked on maps, there are no population statistics, and administrative jurisdictions are constantly changing.

The part of Afghanistan selected for the first evaluation was the main area of German engagement outside the capital Kabul. Bordering on Tajikistan, China and Pakistan, it includes the provinces Kunduz, Takhar, Baglan and Badakhshan. The population is ethnically diverse. Tajiks, Uzbeks and Hazara are numerically dominant but Pashtuns are also present. The Pashtuns are either members of nomadic groups, or were resettled here by the government by the middle of the 20th century. The focus on rural areas makes sense because around 80 % of Afghans live in such areas. Without the cooperation of this majority, national reconstruction cannot succeed.

The questions we asked focused on the positive and negative effects of development projects, with households serving as the units of analysis. The number of households surveyed was large enough to ensure that the sample was representative.

To assess how development projects have impacted on a community, the effect of development cooperation perceived by households was cross-checked against the actual number of projects. These were identified from documents provided by the agencies in the area, and then attributed to communities with the help of a geographic information system (GIS). To ensure the various groups were comparable, data on ethnic composition, economic situation, geographical location, standard of education and household sizes were also collected.


Multi-layered survey

The data was collected in several manners. First, in February 2007, 2,034 households were surveyed by specially trained local interviewers from CoAR (the Coordination of Afghan Relief) in 77 communities in the districts of Imam Sahib, Aliabad, Taloqan and Warsaj. Half of the communities were picked by random sampling; the remainder were selected to make sure that the sample reflected the geographic and ethnic diversity of the region. Within the communities, households were selected at random; but care was taken to ensure that the number of respondents constituted a representative sample of the community. The interviews consisted of 57 questions and lasted 90 minutes. The response rate was around 95 %.

So-called community profiles served as a further source of data. For these, another group of interviewers talked to local administrations, municipal councils and village elders, compiling data about the recent past, demographics, ethnic make-up, local political and social organisations, available facilities and the scale of development projects in the individual communities.

A third data-collection tool was furnished by quarterly reports for the selected communities, which recorded major events such as military operations, new development projects or even natural catastrophes. Data that already existed, such as databases of locally active agencies or the ISAF, were also analysed. In addition, 52 open interviews were conducted with international stakeholders in Afghanistan. In ten localities, detailed focus group interviews were conducted with village elders. Weeks long participant observation on the ground rounded off the data collection. On this broad basis, it was then possible to cross-check descriptive survey results, statistical analyses and qualitative comparative analyses.

D+C, 2008/03, Focus, Page 108-111