Managing Impact Evaluations Strategically

Achieving higher living standards is a perennial aspiration of any society, and one that drives governments to spend ever-scarce resources on infrastructure, promote productive activities and deliver social services like health and education amongst others.

This is a noble objective—but good intentions don’t always ensure that we reach our goals. Governments and citizens must also discern whether they are reaping the full benefits from their investments of public resources. This is not only a matter of transparency and accountability but also of effectiveness: limited resources should be invested where they can generate the greatest benefits.

Impact evaluations are an important tool for measuring whether resources are used effectively and efficiently in the development process. They allow countries and organizations like the IDB to determine to what extent development objectives are actually achieved, and to detect and address unintended consequences. Properly designed, impact evaluations can also shed light on why programs work and how to make them better. As such, they are a valuable tool to inform, design, adapt, and implement development interventions. Furthermore, the lessons learned in one program can also benefit other countries or institutions working on development globally.

Impact evaluations establish a causal link between an intervention and an outcome by teasing out the intervention’s contribution from other external factors. This approach to evaluation is unique in its ability to quantify and attribute impacts. But carrying out an impact evaluation successfully can be a complex process requiring strong commitment of all parties involved. Qualified professionals to closely supervise and properly manage practical and logistical hurdles, as well as to collect specific purpose data and to carry out analytical and computational work are required. Such an endeavor can be costly, particularly if sensitive data need to be collected and so impact evaluations, despite their importance, often stretch the means of cash-strapped governments.

Since the IDB introduced its Development Effectiveness Framework in 2008, the Bank has approved 196 projects in the public sector and 3 in Opportunity for the Majority that planned to use an impact evaluation. 27 percent of those decided not to conduct the evaluation, changed the selected methodology or, as of now, are in standby. In addition, 11 projects changed the evaluation methodology from reflexive to experimental or quasi-experimental after the approval, with 8 of them currently being conducted.

Of the 154 that continued with the planned evaluation, 77 are in the design phase, 67 are being executed, and 10 have been concluded. 35 percent of the evaluations follow experimental methods and the rest conduct them using quasi experimental or other methodologies.

Since the inception of the Development Effectiveness Framework in 2008, the importance of evaluating the impacts of our operations has been growing in the IDB’s DNA. Now that the evaluation seed has been planted, and given their cost and complexity, we need to move up another step and become more strategic in our evaluations. For the IDB, this means prioritizing staff time and money on evaluations that can make a meaningful contribution to the development knowledge base and, especially, to the development challenges faced by our borrowing member countries.

The IDB is committed to continually learning from impact evaluations and sharing this knowledge with our borrowing member countries and the international community facing similar challenges. Decades of development practice by governments and international organizations have shown that the price of ignorance is steep. As we move forward, impact evaluations complement other monitoring and evaluation tools to help enhance the evidence base for development. The box that follows explains more about how the IDB’s supported impact evaluations are helping to ensure effective social and economic development in the region.

What We Learned

Cash Transfers Programs:

Cash transfers are one of the most widely evaluated interventions, with much of the international evidence in lower and middle income countries coming from Latin American and the Caribbean. The current cohort of IDB supported evaluations contributes to our understanding of how these programs work differently in urban and rural areas, how to design incentive schemes more effectively, and the impacts of cash transfers on maternal-child health and to reduce teacher attrition in hardship areas. Cash transfer programs are a dynamic platform for testing innovative solutions to reduce poverty and build human capital and we expect many more exciting learning opportunities in the years to come.

Strengthening Mexico’s Oportunidades Human Development Program in Urban Areas

This series of evaluations was geared toward increasing the effectiveness in urban settings of Mexico’s conditional cash transfer program known as Oportunidades. Previous evaluations showed a significant development effectiveness gap in the program between rural and urban settings. Lower urban effects were associated with mirroring the successful rural design of Oportunidades to localities where child obesity and high school dropouts were the main development challenges instead of the problems of under-nutrition and primary school dropouts that drove the rural design. The evaluations demonstrated the cost-effectiveness of an alternative urban nutrition supplement that reduces anemia by 7 percentage points in children from 6 to 59 months old without the side effect of weight gain of traditional supplements. They also showed that budget-neutral variations in the structure of urban school transfers could lead to positive effects. Eliminating primary school transfers and increasing lower and upper secondary school transfers by 25 percent, in urban settings, would have no negative impact at the primary school level, and could reduce school dropouts at higher school levels by approximately 6 percentage points, although this positive effect varied among beneficiary groups. Paper forthcoming.

Expansion of Colombia’s Familias en Acción Conditional Cash Transfer Program – Phase II

This evaluation analyzed the impact on nutrition, health, and education of a cash transfer program in Colombia known as Familias en Acción. The evaluation constituted a long-term assessment of the program in terms of the accumulation of human capital in municipalities of less than 100,000 inhabitants. It concluded that the program increased the height of children between 9 and 12 years of age by 0.16 standard deviations. It also showed that the program reduced chronic malnutrition of children between 9 and 15 years old by 6 percent. However, obesity in children between 9 and 12 years increased by 5.6 percentage points. The program had a positive impact on the management of food when children suffer acute infectious diarrhea—the probability of providing the same quantity of food to those who are sick increased by 48 percent. Finally, the program had a positive and strong impact on education. Among young adults (18+) in rural areas, the probability of graduation increased by 64 percent and years of schooling rose by six months. The conclusions of the impact evaluation led to a change in the amount and structure of subsidies in order to strengthen the program. Municipalities are now divided into four groups based on their social development, and subsidies now increase as students advance by school grade in order to reduce the opportunity cost of studying. Read the paper on Colombia’s Conditional Cash Transfer Program

Building Social Equity through Mexico’s Community Education Program – Phase II

In remote villages of rural Mexico, one of the biggest challenges to improve the quality of education is to decrease the attrition rate among teachers. Mexico is looking to enhance its incentives to retain teachers in its Community Education Program after a baseline survey conducted in cooperation with the IDB identified reasons why teachers leave. Those reasons include difficulties adapting to life in the community where they’ve been assigned, and a stipend that is insufficient to cover living expenses. The evaluation aimed to better understand teacher attrition, with a focus on the process of adaptation in the community and the role of the stipend in attrition. The analysis resulted in several important policy recommendations. First, efforts to reduce teacher attrition should concentrate at the beginning of the school year and in municipalities with difficult conditions in terms of geography and low access to services. Second, teacher training should include techniques to adapt to teaching and living in communities where food, shelter, and transportation may be rudimentary. Third, to the extent possible teachers should be assigned to localities closest to their homes and teacher preferences in terms of postings should be honored. Fourth, payments should be made in lump sums while teachers are in service, without retaining funds as incentives to stay, and transportation expenses should be covered separately. Finally, the evaluation recommended improving communication and monitoring with communities in order to ensure that basic conditions for the teachers remain in place, including safety, health and sanitation, and social interaction. Paper forthcoming.

Promoting productive activities:

The four evaluations of production development programs assess the effectiveness of matching-grants in fostering technology adoption, innovation and, ultimately, production efficiency. The evaluations confirms that this instrument effectively induce beneficiaries to increase their innovation efforts and to adopt practices that can increase their productivity. These results notwithstanding, more evidence is still needed to fully assess the cost-effectiveness of such instruments depending on the specific design (including targeting), country and sector.

Panama’s Multiphase Technological Transformation Program – Phase I

This evaluation examined the effectiveness of technology development funds by analyzing the impact of matching grants provided by Panama’s National Secretariat for Science, Technology, and Innovation (SENACYT). The grants support innovation initiatives by Panamanian firms. Taking advantage of the first Panamanian Innovation Survey, the evaluation analyzed the effect of receiving support from SENACYT on both the likelihood of undertaking innovation and R&D activities and on the magnitude of the investment in these activities. The evaluation showed that the program helped beneficiary firms increase their investment in innovation and R&D three and four times over, respectively. It also allowed 30 percent of the beneficiaries to engage in innovation and R&D activities for the first time. Because of data limitations the evaluation could not assess the medium- and long-run effect of the program in terms of productivity and adoption of new products and processes. However, the findings provide further evidence on the effectiveness of matching grants in fostering innovation, adding the case of a Central American economy to the existing literature for the first time. The evaluation put forth a series of policy recommendations, with a focus on information collection and production of indicators. Read the paper on Panama’s Multiphase Technological Transformation Program

Financing Investment Projects and Productive Restructuring for Business and Export Development in Colombia

Government-owned development banks can play a crucial role in channeling public funds to firms with little or no access to credit despite their good business prospects and management. Two IDB impact evaluations of Colombia’s Bancoldex, a second-tier development bank that has received IDB funding through two loans, provide evidence of such a role. Using highly disaggregated lending and firm data, the studies analyzed the impact of Bancoldex lending on access to credit and economic performance of beneficiary firms over the past decade. The first study found that firms tapping Bancoldex credit lines got loans with lower interest rates and longer maturities than other firms. Moreover, beneficiary firms expanded their credit relationships with other financial intermediaries, enjoying better credit conditions well after receiving Bancoldex credit. The second study found that beneficiary firms in the manufacturing sector increased output, employment, investment, and productivity within the first four years after they got their first Bancoldex loan. The effects were large, ranging from around 20 percent for employment and productivity to 30 percent for output. The results show that Bancoldex offers some “additionality” beyond simply substituting credit that private sources would be willing and able to offer under similar conditions. Read the story on Bancoldex Lending and Green Financing.

Agricultural Development in Uruguay

This evaluation focused on the Uruguayan Livestock Program (ULP) designed to improve management and production practices among livestock producers. Between 2007 and 2010, the program co-financed around 1,300 business plans proposed by individual producers. Although the program supported the adoption of a wide range of practices, the vast majority of plans ended up focusing on livestock reproduction. For this reason, the evaluation analyzed two key indicators of reproductive efficiency, namely calf production and calf sale. Using an eight-year panel constructed by combining data from the Uruguayan livestock traceability system with a registry of ULP participants, the evaluation found that the ULP on average increased calf production by between 11.36 and 15.3 calves and increased net calf sales by 4.35 calves. These results notwithstanding, the evaluation showed that the program’s internal rate of return is quite sensitive to how benefits are measured—positive and large when the value of production is used, but very limited when using sales. This suggests that the program’s overall economic impact may have been rather modest. Read the paper on Agricultural Development in Uruguay

Integration of Small Producers into the Wine Production Chain in Argentina

This evaluation examined the matching grant component of a project to foster the integration of small producers into the Argentine wine value chain. The project supported coordination activities, co-financed initiatives by small producers, and strengthened institutions in the value chain such as the representatives of small and medium-sized wine growers and a network of wine development centers. It also targeted producers with low productivity, lesser-quality grapes, limited access to technical assistance and markets, and low bargaining power—all features that limit the potential contribution to the value chain. The matching grants specifically supported small producers who undertake investments in anti-hail nets, irrigation systems, machinery, and other vineyard improvements. The evaluation found that the program increased the productivity of beneficiaries by 7.9 percent. The use of anti-hail nets had a particularly significant effect, with increases in production and productivity of around 35 percent. Unfortunately, data limitations did not allow for estimating the impact of the matching grants on grape diversification and the adoption of high-value grape varieties, two expected effects that would have furthered signal improved integration into the value chain. The analysis of these effects will be the topic of an extension of this evaluation. Read the paper on Integration of Small Producers into the Wine Production Chain in Argentina

Social Services:

The four impact evaluations from the social sector span education, nutrition, social protection and active labor market programs. These evaluations contribute to learning about the effectiveness of different service delivery models, and reveal some surprising unexpected impacts. Impact evaluations can play an important role in identifying aspects of interventions that require strengthening as well as unintended consequences, both positive and negative.

Education Equity in Bogota, Colombia

The main purpose of Bogota’s Education Equity Program was to build schools known as megacolegios in poor neighborhoods on the outskirts of Bogota. The facilities included multi-functional classrooms, state-of-the-art libraries, laboratories, and sports facilities. Between 2008 and 2012, megacolegios were built in 15 poor communities with funding from an IDB loan. For the purposes of the impact evaluation, these communities were matched with a similar number of communities that were not served by the project but had comparable characteristics (location, socioeconomic characteristics, and schools’ internal efficiency). Using administrative data and an ad-hoc survey capturing current and retrospective information, the evaluation found that students at the megacolegios had higher grade promotion rates, lower dropout rates, and higher school graduation rates than those at the comparison schools. In terms of academic performance, however, standardized test scores showed no statistically significant differences between students at the megacolegios and at comparison schools. One issue of concern revealed by the evaluation is an increasing prevalence of school bullying, verbal violence, and vandalism, which were found to be slightly higher at the megacolegios. Paper forthcoming.

Improving Child Nutrition Services in Bolivia

In the 8th district of the city of El Alto, 23 percent of children under 4 suffer from stunting, a symptom of chronic malnutrition. Despite the availability of affordable and nutritious foods, many families in the area have poor diets. Starting in 2008, the Community Nutrition Project trained mothers how to better feed their children, provided them with nutritional supplements, and collected health-related data such as weight and height. The intervention also included group cooking classes using fruits and vegetables bought on the city’s streets, as well as community meetings to talk about nutrition, domestic hygiene, and other health-related topics. The impact evaluation compared children in households in the intervention area with ineligible neighbors based on the program’s well-defined geographic boundary. It showed that community health agents successfully delivered services to the targeted population. Mothers increased their knowledge of nutrition and health habits and were more likely to have received micronutrient supplements. However, greater awareness failed to trigger a change in eating habits and correct use of micronutrients, and children who took part in the program were no less likely to be stunted than children in the comparison group. A strengthened version of the intervention incorporating innovative behavioral change strategies will be rolled out under an experimental evaluation framework in early 2014. Read the paper on Improving Child Nutrition Services in Bolivia.

Support for the Dominican Republic’s Social Protection Program – Phase I

This evaluation examined the effectiveness of reinforcement messages aimed at promoting behavioral changes in poor households regarding education in the Dominican Republic. Texting 10 reinforcement messages over 10 consecutive weeks increased student enrollment by 3.2 to 3.8 percentage points among the school-aged population between 5 and 21 years of age. The messages also reduced primary school student absenteeism during the second school semester by 2.5 to 2.7 percentage points. Sending 10 reinforcement messages via e-mail during the same intervention period reduced primary student absences in the second semester by 2.4 to 3.2 percentage points. Voicemail messages also had a consistent impact on behavioral changes within the household, including a 7 percentage point increase in the number of students who dedicate more than one hour a week to homework; a reduction of 2.5 to 3.5 percentage points in the school-age population that engages in some income-generation activities for the home, and a reduction of approximately one hour in the total hours per week that they work; and an increase of 4 percentage points in the number of older siblings helping students with their homework. This brief study presents evidence to justify more comprehensive studies about how reinforcement messages can promote social objectives that benefit poor households. The findings suggest that implementation of a message platform directed to household beneficiaries of social programs could strengthen the effectiveness of social protection policies. Read the paper on the Dominican Republic’s Social Protection Program – Phase I

Program to Support the National Employment System in the Dominican Republic

The Youth and Employment Program in the Dominican Republic trained youth at risk in technical skills and provided them with the opportunity to gain job experience. Training activities were split into three parts: a life skills training module (75 hours); a vocational training module (150 hours); and an on-the-job training module in a private company for three months. According to the impact evaluation, in terms of labor outcomes the program helped beneficiaries find better jobs, with a positive impact on job formality for men of about 17 percent and an increase of 7 percent in monthly earnings among those employed. The program also reduced the probability of teenage pregnancy by 6 percentage points (to 48 percent), with a stronger effect on young and single women and those who were already mothers The channels through which the program seems to affect teenage pregnancy are psychological well-being and expectations. The program also helped young people acquire more general skills such as responsibility and teamwork that will generate greater opportunities for them in the future. Read the paper on the Dominican Republic’s Program to Support the National Employment System.

 

Read Next

IDB helps finance a comprehensive plan to improve transportation infrastructure in Costa Rica.
Or read more from Transport
MiBanco expands access to finance and training to women entrepreneurs in Peru.
Or read more from Institutions for Growth and Social Welfare