In my development economics class today, we had Michael Clemens (bio) from the Center for Global Development as a guest speaker. He talked about the impact evaluation of aid. Poverty reduction is appealing especially to people living in the developed world; hundreds of millions of dollars are donated for poor countries and regions around the world. The impact evaluation, however, is often neglected.
By “impact evaluation”, we mean a comparison of the post-project situation with what would have happened had the project not taken place that is cost-effectively rigorous. Clemens suggested three ways to increase the rigor of impact evaluation: independence, consistency, and transparency Firstly, the evaluation should avoid insider judgements as they tend to be biased; evaluation by outside parties such as other scholars/academia or consulting firms are usually more convincing. Secondly, the project should be evaluated for its fixed goals rather than shifting objectives. Thirdly, the information should be made to the public for further inspection.
Jeffrey Sachs advocates the Millenium Village Project (website) and the project is aimed to reduce extreme poverty by half by 2015. The information on the website about impact evaluation sounds encouraging:
Improved drinking water, with household usage increasing from 21% to 68%; Expanded access to HIV testing, from 8% to 28% of adults tested in the last 12 months; Reduced malaria prevalence, from 22% to 5%
But such information can be misleading because (a) it does not show how other regions and districts are doing in the same period and therefore neglects the fixed time effect acting behind both treated and untreated groups, and (b) it does not address the central objective of the project — poverty reduction.
The counterfactual is always difficult to measure because history only happens once. But beneficiaries of the project should be able to provide some suggestions of their satisfaction towards the program and therefore give researchers some idea about its effectiveness.
Clemens also mentioned that the right rigor of impact evaluation does not equal randomization. Difference-in-Difference estimation can be useful in some contexts as well.