Achieving higher living standards is a perennial aspiration of any society, and one that drives governments to spend ever-scarce resources on infrastructure, promote productive activities and deliver social services like health and education amongst others.
This is a noble objective—but good intentions don’t always ensure that we reach our goals. Governments and citizens must also discern whether they are reaping the full benefits from their investments of public resources. This is not only a matter of transparency and accountability but also of effectiveness: limited resources should be invested where they can generate the greatest benefits.
Impact evaluations are an important tool for measuring whether resources are used effectively and efficiently in the development process. They allow countries and organizations like the IDB to determine to what extent development objectives are actually achieved, and to detect and address unintended consequences. Properly designed, impact evaluations can also shed light on why programs work and how to make them better. As such, they are a valuable tool to inform, design, adapt, and implement development interventions. Furthermore, the lessons learned in one program can also benefit other countries or institutions working on development globally.
Impact evaluations establish a causal link between an intervention and an outcome by teasing out the intervention’s contribution from other external factors. This approach to evaluation is unique in its ability to quantify and attribute impacts. But carrying out an impact evaluation successfully can be a complex process requiring strong commitment of all parties involved. Qualified professionals to closely supervise and properly manage practical and logistical hurdles, as well as to collect specific purpose data and to carry out analytical and computational work are required. Such an endeavor can be costly, particularly if sensitive data need to be collected and so impact evaluations, despite their importance, often stretch the means of cash-strapped governments.
Since the IDB introduced its Development Effectiveness Framework in 2008, the Bank has approved 196 projects in the public sector and 3 in Opportunity for the Majority that planned to use an impact evaluation. 27 percent of those decided not to conduct the evaluation, changed the selected methodology or, as of now, are in standby. In addition, 11 projects changed the evaluation methodology from reflexive to experimental or quasi-experimental after the approval, with 8 of them currently being conducted.
Of the 154 that continued with the planned evaluation, 77 are in the design phase, 67 are being executed, and 10 have been concluded. 35 percent of the evaluations follow experimental methods and the rest conduct them using quasi experimental or other methodologies.
Since the inception of the Development Effectiveness Framework in 2008, the importance of evaluating the impacts of our operations has been growing in the IDB’s DNA. Now that the evaluation seed has been planted, and given their cost and complexity, we need to move up another step and become more strategic in our evaluations. For the IDB, this means prioritizing staff time and money on evaluations that can make a meaningful contribution to the development knowledge base and, especially, to the development challenges faced by our borrowing member countries.
The IDB is committed to continually learning from impact evaluations and sharing this knowledge with our borrowing member countries and the international community facing similar challenges. Decades of development practice by governments and international organizations have shown that the price of ignorance is steep. As we move forward, impact evaluations complement other monitoring and evaluation tools to help enhance the evidence base for development. The box that follows explains more about how the IDB’s supported impact evaluations are helping to ensure effective social and economic development in the region.
What We Learned
Cash Transfers Programs:
Cash transfers are one of the most widely evaluated interventions, with much of the international evidence in lower and middle income countries coming from Latin American and the Caribbean. The current cohort of IDB supported evaluations contributes to our understanding of how these programs work differently in urban and rural areas, how to design incentive schemes more effectively, and the impacts of cash transfers on maternal-child health and to reduce teacher attrition in hardship areas. Cash transfer programs are a dynamic platform for testing innovative solutions to reduce poverty and build human capital and we expect many more exciting learning opportunities in the years to come.
Strengthening Mexico’s Oportunidades Human Development Program in Urban Areas
This series of evaluations was geared toward increasing the effectiveness in urban settings of Mexico’s conditional cash transfer program known as Oportunidades. Previous evaluations showed a significant development effectiveness gap in the program between rural and urban settings. Lower urban effects were associated with mirroring the successful rural design of Oportunidades to localities where child obesity and high school dropouts were the main development challenges instead of the problems of under-nutrition and primary school dropouts that drove the rural design. The evaluations demonstrated the cost-effectiveness of an alternative urban nutrition supplement that reduces anemia by 7 percentage points in children from 6 to 59 months old without the side effect of weight gain of traditional supplements. They also showed that budget-neutral variations in the structure of urban school transfers could lead to positive effects. Eliminating primary school transfers and increasing lower and upper secondary school transfers by 25 percent, in urban settings, would have no negative impact at the primary school level, and could reduce school dropouts at higher school levels by approximately 6 percentage points, although this positive effect varied among beneficiary groups. Paper forthcoming.
Expansion of Colombia’s Familias en Acción Conditional Cash Transfer Program – Phase II
This evaluation analyzed the impact on nutrition, health, and education of a cash transfer program in Colombia known as Familias en Acción. The evaluation constituted a long-term assessment of the program in terms of the accumulation of human capital in municipalities of less than 100,000 inhabitants. It concluded that the program increased the height of children between 9 and 12 years of age by 0.16 standard deviations. It also showed that the program reduced chronic malnutrition of children between 9 and 15 years old by 6 percent. However, obesity in children between 9 and 12 years increased by 5.6 percentage points. The program had a positive impact on the management of food when children suffer acute infectious diarrhea—the probability of providing the same quantity of food to those who are sick increased by 48 percent. Finally, the program had a positive and strong impact on education. Among young adults (18+) in rural areas, the probability of graduation increased by 64 percent and years of schooling rose by six months. The conclusions of the impact evaluation led to a change in the amount and structure of subsidies in order to strengthen the program. Municipalities are now divided into four groups based on their social development, and subsidies now increase as students advance by school grade in order to reduce the opportunity cost of studying. Read the paper on Colombia’s Conditional Cash Transfer Program
Building Social Equity through Mexico’s Community Education Program – Phase II
In remote villages of rural Mexico, one of the biggest challenges to improve the quality of education is to decrease the attrition rate among teachers. Mexico is looking to enhance its incentives to retain teachers in its Community Education Program after a baseline survey conducted in cooperation with the IDB identified reasons why teachers leave. Those reasons include difficulties adapting to life in the community where they’ve been assigned, and a stipend that is insufficient to cover living expenses. The evaluation aimed to better understand teacher attrition, with a focus on the process of adaptation in the community and the role of the stipend in attrition. The analysis resulted in several important policy recommendations. First, efforts to reduce teacher attrition should concentrate at the beginning of the school year and in municipalities with difficult conditions in terms of geography and low access to services. Second, teacher training should include techniques to adapt to teaching and living in communities where food, shelter, and transportation may be rudimentary. Third, to the extent possible teachers should be assigned to localities closest to their homes and teacher preferences in terms of postings should be honored. Fourth, payments should be made in lump sums while teachers are in service, without retaining funds as incentives to stay, and transportation expenses should be covered separately. Finally, the evaluation recommended improving communication and monitoring with communities in order to ensure that basic conditions for the teachers remain in place, including safety, health and sanitation, and social interaction. Paper forthcoming.
Promoting productive activities:
The four evaluations of production development programs assess the effectiveness of matching-grants in fostering technology adoption, innovation and, ultimately, production efficiency. The evaluations confirms that this instrument effectively induce beneficiaries to increase their innovation efforts and to adopt practices that can increase their productivity. These results notwithstanding, more evidence is still needed to fully assess the cost-effectiveness of such instruments depending on the specific design (including targeting), country and sector.
Panama’s Multiphase Technological Transformation Program – Phase I
This evaluation examined the effectiveness of technology development funds by analyzing the impact of matching grants provided by Panama’s National Secretariat for Science, Technology, and Innovation (SENACYT). The grants support innovation initiatives by Panamanian firms. Taking advantage of the first Panamanian Innovation Survey, the evaluation analyzed the effect of receiving support from SENACYT on both the likelihood of undertaking innovation and R&D activities and on the magnitude of the investment in these activities. The evaluation showed that the program helped beneficiary firms increase their investment in innovation and R&D three and four times over, respectively. It also allowed 30 percent of the beneficiaries to engage in innovation and R&D activities for the first time. Because of data limitations the evaluation could not assess the medium- and long-run effect of the program in terms of productivity and adoption of new products and processes. However, the findings provide further evidence on the effectiveness of matching grants in fostering innovation, adding the case of a Central American economy to the existing literature for the first time. The evaluation put forth a series of policy recommendations, with a focus on information collection and production of indicators. Read the paper on Panama’s Multiphase Technological Transformation Program
Financing Investment Projects and Productive Restructuring for Business and Export Development in Colombia
Government-owned development banks can play a crucial role in channeling public funds to firms with little or no access to credit despite their good business prospects and management. Two IDB impact evaluations of Colombia’s Bancoldex, a second-tier development bank that has received IDB funding through two loans, provide evidence of such a role. Using highly disaggregated lending and firm data, the studies analyzed the impact of Bancoldex lending on access to credit and economic performance of beneficiary firms over the past decade. The first study found that firms tapping Bancoldex credit lines got loans with lower interest rates and longer maturities than other firms. Moreover, beneficiary firms expanded their credit relationships with other financial intermediaries, enjoying better credit conditions well after receiving Bancoldex credit. The second study found that beneficiary firms in the manufacturing sector increased output, employment, investment, and productivity within the first four years after they got their first Bancoldex loan. The effects were large, ranging from around 20 percent for employment and productivity to 30 percent for output. The results show that Bancoldex offers some “additionality” beyond simply substituting credit that private sources would be willing and able to offer under similar conditions. Read the story on Bancoldex Lending and Green Financing.
Agricultural Development in Uruguay
This evaluation focused on the Uruguayan Livestock Program (ULP) designed to improve management and production practices among livestock producers. Between 2007 and 2010, the program co-financed around 1,300 business plans proposed by individual producers. Although the program supported the adoption of a wide range of practices, the vast majority of plans ended up focusing on livestock reproduction. For this reason, the evaluation analyzed two key indicators of reproductive efficiency, namely calf production and calf sale. Using an eight-year panel constructed by combining data from the Uruguayan livestock traceability system with a registry of ULP participants, the evaluation found that the ULP on average increased calf production by between 11.36 and 15.3 calves and increased net calf sales by 4.35 calves. These results notwithstanding, the evaluation showed that the program’s internal rate of return is quite sensitive to how benefits are measured—positive and large when the value of production is used, but very limited when using sales. This suggests that the program’s overall economic impact may have been rather modest. Read the paper on Agricultural Development in Uruguay
Integration of Small Producers into the Wine Production Chain in Argentina
This evaluation examined the matching grant component of a project to foster the integration of small producers into the Argentine wine value chain. The project supported coordination activities, co-financed initiatives by small producers, and strengthened institutions in the value chain such as the representatives of small and medium-sized wine growers and a network of wine development centers. It also targeted producers with low productivity, lesser-quality grapes, limited access to technical assistance and markets, and low bargaining power—all features that limit the potential contribution to the value chain. The matching grants specifically supported small producers who undertake investments in anti-hail nets, irrigation systems, machinery, and other vineyard improvements. The evaluation found that the program increased the productivity of beneficiaries by 7.9 percent. The use of anti-hail nets had a particularly significant effect, with increases in production and productivity of around 35 percent. Unfortunately, data limitations did not allow for estimating the impact of the matching grants on grape diversification and the adoption of high-value grape varieties, two expected effects that would have furthered signal improved integration into the value chain. The analysis of these effects will be the topic of an extension of this evaluation. Read the paper on Integration of Small Producers into the Wine Production Chain in Argentina