Randomized control trials (RCTs) have been at the forefront of development economic research in recent years. How well do these inform us of policy alternatives to reduce poverty?
On the bright side, RCTs allow us to identify the causal impact of policy interventions, and a lot of studies provide evidence that some simple nudging can make a big difference on behavior (see Esther Duflo’s work on encouraging Kenyan farmers to use fertilizers). However, there are also a few caveats in interpreting RCT results:
Publication Bias: only significant results — either positive or negative — get published. Are we learning about the truth or the truth we WANT to know? For instance, microfinance has been applauded as an innovative and effective way to increase savings and investments, encourage entrepreneurship, and reduce poverty. But a recent working paper has found zero effects of access to microfinance on long term development outcomes.
Pre-analysis plan vs. manual selection after study is initiated: Here is a philosophical discussion in the Journal of Economic Perspectives by Ben Olken.
Heterogeneous treatment effects: the magnitude of the effects of policy varies a great deal. External validity is often a concern. Here is a thought-provoking paper by Eva Vivalt, the founder of AidGrade, a database on impact evaluations.
Experimental arms race: Are we simply adding more technical details into the same experiments without shedding light on fundamental channels of how they change behavior? Here is an article by David McKenzie on the tpoic. More specifically, Rachel Glennerster writes about what this implies for RCTs involving governments.
Your thoughts and comments are welcome.