My Two Cents on Randomized Controlled Trials

Randomized control trials (RCTs) have been at the forefront of development economic research in recent years. How well do these inform us of policy alternatives to reduce poverty?

On the bright side, RCTs allow us to identify the causal impact of policy interventions, and a lot of studies provide evidence that some simple nudging can make a big difference on behavior (see Esther Duflo’s work on encouraging Kenyan farmers to use fertilizers). However, there are also a few caveats in interpreting RCT results:

Publication Bias: only significant results — either positive or negative — get published. Are we learning about the truth or the truth we WANT to know? For instance, microfinance has been applauded as an innovative and effective way to increase savings and investments, encourage entrepreneurship, and reduce poverty. But a recent working paper has found zero effects of access to microfinance on long term development outcomes.

Pre-analysis plan vs. manual selection after study is initiated: Here is a philosophical discussion in the Journal of Economic Perspectives by Ben Olken.

Heterogeneous treatment effects: the magnitude of the effects of policy varies a great deal. External validity is often a concern. Here is a thought-provoking paper by Eva Vivalt, the founder of AidGrade, a database on impact evaluations.

Experimental arms race: Are we simply adding more technical details into the same experiments without shedding light on fundamental channels of how they change behavior? Here is an article by David McKenzie on the tpoic. More specifically, Rachel Glennerster writes about what this implies for RCTs involving governments.

Your thoughts and comments are welcome.


Pascaline Dupas on Decentralization and Efficient Targeting of Subsidies

Pascaline Dupas, a development economist at Stanford University, gave a talk in our department this afternoon on how decentralization affects the targeting of subsidies in Malawi.

The allocation of public subsidies to targeted beneficiaries is an important task of governments in developing countries. Three conditions need to be met to maximize the effectiveness of these subsidies:

1. They should be assigned to people with the highest returns.

2. There should be limited leakage on the way to assigned beneficiaries.

3. Beneficiaries should put subsidies in appropriate (intended) use.

The second and the third conditions have been explored extensively in the development economics literature. For example, economists have examined the effectiveness of conditional cash transfer (CCT) programs on school enrollment of children in poor families. By making school enrollment a precondition for cash transfers, the government hopes to nudge parents to send their children to school, which they probably wouldn’t do if the transfers were unconditional.

The purpose of this paper is to investigate the first condition. More specifically, they look at the effectiveness of centralized (proxy means tested, or PMT-based) vs. decentralized (through chiefs) allocation of subsidies in Malawi. Dupas and her coauthors ask the following questions:

To what extent does the poor performance of local authorities in the aggregate come from inefficient allocation of resources or subsidies? How well do chiefs target? Could a centralized (PMT-based) system do better?

In Malawi, like in many other African countries, chiefs play an important role in allocating resources within local communities. There are advantages and disadvantages of letting the chiefs assign the subsidies instead of the government. PMT-based systems assign subsidies based on asset owned by individuals/families, but asset is a poor predictor for consumption levels of the poor, and the latter is a much more reasonable measure of poverty. Chiefs, on the other hand, are likely to have more local information, especially regarding the relative characteristics of the households within the community. Therefore, they might do a better job assigning the subsidies to targeted beneficiaries. However, chiefs might also favor their kins and dampen the intended equalizing effects of the subsidies.

The authors assess the efficiency of subsidy allocation along two margins: poverty targeting and productivity efficiency. The former is giving subsidies to people who are eligible based on their consumption of perishable food, while the latter assigns subsidies to people where the returns are the highest. If chiefs know that there will be income pooling between community members after subsidies are put to use, they will have more incentive to allocate subsidies to the most productive households in order to maximize efficiency. Such an outcome will also be more likely the more heterogeneity there exists in productivity.

Dupas showed us the error rates of three systems of subsidy allocation: a hypothetical PMT-based system, an allocation through chiefs, and a random allocation. There are two types of errors: type I error where an eligible individual is not given a subsidy, and type II error where an ineligible individual is given one. Surprisingly, chiefs seem to do as well as a PMT-based system at identifying the targeted beneficiaries.

How to incorporate productivity and equality considerations into the decision process of the chiefs? The authors proposed a general framework where the chief maximizes a weighted sum of households’ benefits from the subsidy (in terms of expected additional profit) within the community. They divide households into different classes based on their relationship with the chief, poverty status, and returns to the subsidy. The relative weights of the chief towards different classes can be recovered through comparing the allocation patterns of food versus input subsidy, with the assumption that the former doesn’t involve productivity considerations while the latter does.

Although the question they are investigating is interesting and important, I wish the speaker were clearer about what alternative allocation systems could generate similar effects as allocating through the chiefs. In particular, if the only advantage of the chief relative to a centralized agency is that he (or she, in the rare case) has updated local information, would it be possible to create a self-monitoring system within the community such that people divide the subsidy optimally among themselves? Another related concern is that the chief is likely to base their judgment on self-reported characteristics of the household when allocating resources. If they don’t observe the true productivity of households in the community, how might that affect the findings about their productivity efficiency consideration? I look forward to reading their paper when it comes out.

Book Review: The Idealist by Nina Munk

I heard about this book from Marginal Revolution. Here are some excerpts. It’s a concise and interesting book, great for anyone interested in economic development and foreign aid in Africa.

Journalist and author Nina Munk presented a vivid narrative of how Professor Jeffrey Sachs at Columbia University initiated the Millennium Village Project, a large-scale development program aimed at eliminating extreme poverty and creating opportunities of economic development in Africa. The project has been subject to wide scrutiny especially from development economists like Esther Duflo ever since its birth. It is criticized as untransparent, poorly enforced, and sloppily evaluated.

Sachs expected the project to eradicate extreme poverty and provide a pathway for sustainable economic development among the ultra poor in Africa. This can be seen in his speech in Ruhiira, Uganda:

In five years we are going to end hunger in this community. In five years we are going to bring malaria completely under control. In five years we will have hospitals and clinics through the whole community. In five years you will have beautiful crops. Step by step, poverty will become something of the past!

He probably still holds this aspiration now. But from what I have read in this book, there are four main reasons why this project is unlikely to achieve its goals.

First, it is fundamentally hard to teach people how to be self-reliant, and simply giving them money is not a solution. Many African countries have been receiving aid for as long as they can remember, and their incentive to be self-dependent is low. As Ahmed Mohamed, the former director of MVP’s Dertu project, puts it:

Our people have refugee syndrome. There are so many handouts here. Free food, free medications, free water, free education. And now we come in and talk to them about empowerment.

Second, development projects initiated by foreigners but without the supporting local infrastructure is doomed to fail.

Third, development programs that are not aligned with preexisting market conditions are likely to fail. David Siriri, the person in charge of MVP’s Ruhiira office, once encouraged farmers to grow maize using effective farming methods, hoping that increased yields will lead to higher incomes for them. While the farmers did enjoy a big harvest, they were unable to sell their surplus maize because of the limited demand for maize. To increase the incomes of the poor, development agencies need to consider the dynamics of local markets as well.

Last but not least, cultural conventions and religious beliefs sometimes prevent program beneficiaries from making the most out of development projects. Believing that “everything is written” gives an individual little incentive to work hard because anything can be taken by a supernatural power at anytime. Traditions can also make people economically irrational. For example, Somalis hoard camels even when it makes no economic sense to do so, only because camels symbolize wealth in their culture.

It is a bit poignant for me to see villagers not appreciating the efforts of the MVP staff but asking for more. Common complaints about MVP include lack of transparency, inadequate community participation in decision-making, vaguely defined ownership, etc. Well, all the benefits brought by MVP are essentially a windfall for them, so I guess their high hopes for follow-up work is warranted.

Banerjee’s Review on Micro-finance Research


Picture from Engaging Justice GWSS, University of Minnesota.

MIT development economist Abhijit Banerjee wrote a nice and short review of micro finance. Great reading for every development-economist-to-be.

Two decades of research has provided us with a much clearer view of why and how micro finance works. Yet there are still fundamental questions that call for more careful investigation. Does group borrowing help the poor’s long term growth more than individual borrowing? Why does micro credit have limited impact on business creation and income growth? How do the incentives of micro finance institutions (MFIs) affect the debt contracts that borrowers take up?

The answer to the first question is yes and no. It is widely accepted that group borrowers tend to monitor each other in terms of undertaking profit-generating projects and making timely repayments. However, groups formed out of family ties can be more reluctant to press their members in loan repayments. Indeed, an individual in such groups might find it easier to default anyway and let family members cover for him. Fortunately, economists are identifying innovative ways to combine favorable features of group borrowing to enhance the performance of individual loan contracts. For example, letting some of the group members get loans and the others paid to share the joint liability has been recently proved to decrease default rates.

The answer to the second question is a bit trickier. Development economists are tempted to believe that micro-finance to be a silver bullet for all the problems of the poor. But in reality how well an individual utilizes micro credit depends critically on his or her intangible personal traits such as financial literacy, self-control, entrepreneurship, and even ambition. Not all micro-credit borrowers are good at growing businesses. Indeed, the fact that they have difficulty getting credit might have already proved they are poor entrepreneurs. These borrowers might also have poor financial judgment   due to lack of education. In terms of motivation, some borrowers might take loans to facilitate consumption needs instead of long term investments. In addition to these “personal characteristics” reasons, Banerjee also suggests a set of more objective reasons. Frequent repayment might restrict borrowers’ ability to make lump sum investments, and production and consumption might exhibit increasing returns only when investment exceeds a threshold which micro loans cannot reach.

The third question was largely ignored in the early micro-finance literature but is gaining attention recently. MFIs possess the ultimate power to decide the terms of loan contracts, process of approval, enforcement methods, and punishment for defaults. It is important to bear in mind that the current loan contracts are already selected to help the MFIs maximum profits (even though many such organizations usually claim they have a social responsibility) in all economic analyses of micro-finance. For example, lenders might make the terms of individual loans more attractive in order to reduce default rates.

At the end of the article, the author points out several directions for future research. On the theory side, like many other fields in development economics, much remains to be done. Xiao Yu Wang, a new Assistant Professor in our department, has written her job market paper on the matching of risk types within social networks. On the purely empirical side, two questions stand out. First, is it possible to help borrowers make better use of the loan? If so, how should we do it and what should we expect? Some development organizations (e.g. BRAC) offer financial literacy curriculum together with loans, but the impact of such educational programs are rarely rigorously evaluated. Second, is it possible to create a mechanism that makes larger loans than micro-credit loans and still maintain high repayment rates?

If you want to read more about micro finance, The Handbook of Micro-finance provides a thorough review for subject, and Due Diligence by David Roodman (which I blogged about) cites research in sociology and anthropology with a lot of case interviews.

Economics of the Family (1): Measuring Living Costs at the Household Level

This is the first of a series of posts on the economics of the family, based on lectures and in-class discussions of Professor Amar Hamoudi‘s seminar course on this topic.

The central discussion of our first class was a fundamental question in economic research and policy design: how should we define poverty? An economics student might think naively construct a minimum income threshold as the poverty line, but this effectively classifies all infants as poor. While children do not earn an income, they enjoy food and housing which are shared among family members. This extremely example highlights the public good nature of domestic goods and services and calls for measures of well-being that takes demographic composition into consideration.

To make households with different demographic characteristics comparable, we need to make select a reference household structure and use equivalence scales, which are “measures of the relative costs of living of families of different sizes and compositions that are otherwise similar” (Citro and Michael, 1995). For example, if the equivalence scale of a single adult family is 0.5 and the reference family has two adults and two children, then a single adult can live as well as a family of two adults and two children while spending only half as much. For economists, equivalence scales directly measures the impact of changes in demographic composition on the cost of living.

An example of the policy application of equivalence scales is Mollie Orshansky’s calculation of poverty thresholds in the US (Orshansky, 1965). Orshansky used USDA “economy food plan” to compute the food costs for families of different size and composition, and then adjust for the fraction of expenditure spent on food.

It is useful to narrow our scope to calculating the costs of an additional household member, in particular, an additional child. This calculation can be done using the Engel Curve or Rothbarth measure (Nelson, 1993). Engel equivalence scales draw from empirical evidence that the share of food expenditure decreases as families become better off, and compute the costs of a child to be the compensated income needed for a family to restore its share of food expenditure before the childbirth. By contrast, Rothbarth estimates child costs by selecting a group of adult goods (such as alcohol and adult clothing) and calculating the income needed to restore the consumption of these goods. Deaton and Muellbauer (1986) innovatively modeled changes in the demographic composition of the family as variations in the prices of goods (food and nonfood, for the simplest case). Consumption demand for different goods is regressed on the adjusted “prices” and duality is used to interpret the results.

At the end of the lecture, Amar raised two interesting points for further discussion. First, why do we measure everything at the household level? Assume there are two households, each including one elderly couple with the same level of household income. Couple A cooks dinner for their son who lives just next door (but not in their house), while couple B only takes care of themselves. Which couple is better off? Maybe they are equally well off because couple A might gain utility from cooking for their son. But the question does not stop here. We need to ask if the son brings additional income to the household or share resources with his elderly parents. In this case, public goods provision and resource sharing might expand well beyond the boundary of the household. In essence, we are assuming that the household is a shared consuming unit where everyone inside shares resources and reaps utilities that are inter correlated. Second, households do not only consume, they also produce. It is more realistic to model the household as a firm where production and consumption are jointly decided.

Our topic for the next class is gender and resource allocation. Looking forward to discussing about the interesting literature on intra household bargaining and gender bias in child investment.

Citro, C. and Michael, R., eds. (1995). “Measuring Poverty: A New Approach“.
Deaton, A. and Muellbauer, J. (1986). “On Measuring Child Costs, with Applications to Poor Economies”. Journal of Political Economy 94(4): 720-744.
Nelson, JA. (1993). “Household Equivalence Scales: Theory versus Policy?” Journal of Labor Economics 11(3): 471-493.
Orshansky, M. (1965). “County the Poor: Another Look at the Poverty Profile.” Social Security Bulletin (January 1965): 3-29.

Nancy Qian on China’s Great Famine

Image from NY Times.

I was surprised and pleased to find Nancy Qian on my department’s list of speakers for development and labor seminar series. I remember her paper on the missing women and price of tea for how well-articulated and convincing her argument was. This afternoon I finally had the pleasure to listen to her talking about the institutional causes of China’s great famine from 1959 to 1961. The working paper is here.

China’s great famine is not a new topic. Due to its unprecedented scale and the special policies China was pursuing at that time, it has attracted a lot of scholarly attention. While previous research has sought answers from rigidity of institutional arrangements and local officers’ zealousness to impress the central, this paper argues that the procurement policy of the central government is the main cause of the famine.

Here are a few facts about China’s great famine that you should know before reading the paper: First, deaths are mostly rural. Interestingly, the central government seemed to have controlled the spreading of news extremely well so that rural residents thought their urban relatives suffered just as much they did (from interviews with survivors). Secondly, estimated per capita average food availability is too high (2000+ calories) to be compatible with a famine. Lastly, there was considerable variation in food availability — a fact largely ignored by previous research but is the focus of Qian and her coauthors.

I was most impressed by how Qian and her coauthors navigate through different data sources for their purposes. They constructed two benchmarks of caloric levels for “food needed to survive”: a higher one for heavy adult labor and healthy child development, and a lower one for staying alive (from the Minnesota starvation experiment). They used USDA guidelines and adjusted total population by demographic breakdown. For grain production, they used post-Mao corrected data. Historic per capita consumption data were acquired from National Bureau of Statistics and aggregate procurement data from Ministry of Agriculture.

Measurement error is obviously a major concern here. Chinese statistics are known to be unreliable, and the fact that these data were collected in a period of unrest exacerbates this problem. to check the robustness of their findings, they used the 1990 birth cohort census (at the county level) as another proxy for the severity of the famine. The idea is: during the famine, couples are less likely to have children, and those who were born immediately before or during the famine were less likely to survive than individuals born after the famine. They addressed the measurement error problem in grain production by constructing predicted production from data on temperature, rainfall, and suitability to produce grains.

Nancy Qian is an impressive presenter. She provided sufficient background knowledge on the subject before going to the key model. She showed only graphs and figures which contained important facts to bear in mind or key insights from their model. The importance of helping your audience visualize your results cannot be understated.

More tips on focus group discussions

My friend Aine McCarthy, a PhD student at University of Minnesota who is doing field research in Tanzania, offered me some advice on focus group sampling methods and arrangements. I also blogged about focus groups (here and here).

/picture: me in focus group discussion with farmers in Katente village./

The bottom line is that focus group discussions should be informative on the topic you are going to investigate in your subsequent field work (surveys, individual interview, etc). I did three focus group discussions before conducting my survey. Two of them consisted of farmers, and one of boda boda drivers. Only the boda boda driver group can be considered successful. For the first farmer’s group, I did not specify my requirements (only members of the SACCO are eligible). So the village chairperson gathered twelve farmers, among whom only four were members. The focus group discussion turned into a mobilization trip. At last, I had to separate the group into members and non members and let out boda boda driver to explain products at the SACCO (he’s an old member here). The second time for farmer’s group, I did not communicate with the villager leader well and I went to the discussion site empty-handed. This greatly limited my ability to probe meaningful questions.

Based on my experience and Aine’s advice, here are a few more tips:
1. Seek help from village leaders and local organizations which know the population well. They will save you a lot of time finding participants. Remember that focus group participants need not be randomly selected.
2. The criteria to group people should be relevant to your topic of interest. Aine’s research is about family planning, so grouping participants by gender can make the conversation more natural.
3. Choose a suitable group size which allows you to generate enough useful information. On the one hand, a group of only 3 or 4 people (my first farmer’s group) is unlikely to yield rich information. On the other hand, if a group is too large (the definition of “large” depends on the number of investigators and assistants), you will fail to capture all the information. It will also make the discussion longer, which might make participants impatient for the last few questions.
4. Make a list of questions, but be flexible. It doesn’t harm to try a few questions in your first focus group if you are not sure whether they will yield meaningful information.
5. Notify the participants the length of the focus group discussions beforehand. I think it’s basic respect for others’ time.
6. Compensate participants appropriately depending on their contributions and the cultural setting. Sometimes cash works better, but a soda can be enough.

Field research is a process of trial and error. And this process may be particularly lengthy and unpredictable in an unfamiliar environment. Learning by doing is the correct attitude.