Chapter 2: Development Effectiveness at Approval
In this chapter:
- Sovereign-Guaranteed Operations
- Project Approvals in 2014
- Assessing Projects at Entry
- Box 2.1: BELIZE – Improving Teachers to Benefit Students
- Box 2.2: PANAMA – Equitable Access to Electricity
- Box 2.3: JAMAICA – Citizen Security and Justice Programme III
- Box 2.4: PERU – Using Information technology to Improve the Agriculture Sector
- The DEM & IDB Priorities
- Non-Sovereign Guaranteed Operations
Project Approvals in 2014
In 2014, the IDB’s Board of Directors approved 105 sovereign guaranteed loans, with a value of $11 billion. Investment projects accounted for $7.8 billion, and policy-based projects that support the design and implementation of policies or specific reforms accounted for $3.2 billion (see Figure 2.1). One project under the Contingent Credit Line for Sustainable Development was also approved for $300 million.
Given the complexity of the region’s development challenges, the IDB is increasingly aiming to capture a multisectoral view, starting with its sector notes. More and more, sector notes reflect the multi-faceted reality of the region’s challenges thus better supporting country strategy development. The overarching goal is to allow for the design and execution of projects that address challenges comprehensively. As a result, lending volumes reported by specific thematic divisions offer only a partial view of how the IDB allocates its resources. Figure 2.1 shows the lending volume of IDB divisions that approved projects in 2014, without detailing cross-collaboration, but instead by the division of the primary booking unit. Figure 2.2 illustrates joint work across IDB divisions, which reflects the multisectoral approach of our work.
In 2014, 36 public sector operations (34 percent of the SG-total) were the result of joint work between different Bank divisions, a frequent way in which the IDB works to tackle multisectoral challenges. In addition, four operations were designed jointly with one of the Bank’s private sector windows. In IDB jargon, this is called “multi-booking”. Such collaboration allows for a more holistic approach of development projects as the strengths of different divisions combine and synergies between them improve the overall effectiveness of our work.
Assessing Projects at Entry
How can the IDB ensure that the 105 SG operations approved in 2014 will reach their development objectives? Within limits, the answer depends on the Development Effectiveness Framework (DEF). The main purpose of the DEF is precisely that. In order to achieve effectiveness, projects ought to be designed not only in a coherent way from the standpoint of the project logic and the best evidence-based solution supporting them, but also in a way that lends itself to evaluating them once completed. To assess if projects have these characteristics, the IDB uses the Development Effectiveness Matrix (DEM) prior to project approval.
How are DEMs designed and what do they represent? The evaluability assessment of the DEM is composed of three subsections: program logic, ex-ante economic analysis, and monitoring and evaluation. For each of these subsections the DEM generates a score between 0 and 10. The simple average of these three scores corresponds to the overall DEM score. All projects must reach a minimum of 5 in each of the evaluability subsections and in the overall DEM score before being submitted for approval by the Board of Executive Directors.
Figure 2.3 shows the evolution of DEM scores between 2009 and 2014. DEM scores increased considerably during the period. Since 2012, all projects have had overall DEM scores above 7, meaning that their design is supported by an evidence-based economic logic and in a way in which their results can be evaluated.
In 2014, 60 percent of approved projects were rated as “evaluable”1 and 40 percent as “highly evaluable.” Boxes 2.1 to 2.4 illustrate some of the projects approved in 2014 that had high DEM scores and point out main features in them that illustrate why they were evaluated this way. To understand what lies behind the overall DEM score, it is necessary to look in more detail at the scores of the evaluability categories. Figure 2.4 shows that average scores across the three categories have shown sustained improvement. In 2014, the highest average score was for the economic analysis category (9.5), followed by program logic (9.1) and monitoring and evaluation (7.8).
BELIZE – Improving Teachers to Benefit Students
An IDB education loan to Belize in 2014 was rated highly evaluable with a DEM score of 9.5. A distinct feature of the design of this project was the reliance of its diagnostics and proposed solutions on both novel country-specific and international evidence. The project aims to build on the success of a 2012 pilot to explore scaling the project. In addition the project will follow a rigorous evaluation plan aimed at re-confirming the success of the previous pilot on a much larger scale. The project will use two randomized control trials to assess the effects of teacher and principal training on classroom instruction and student learning, and it will also test innovative ways to identify individuals with strong teaching potential.
Even though 92 percent of school-aged children in Belize attend primary school, less than half achieve a satisfactory score on the primary education exit exam. Considering Belize’s relatively high spending on education—6.8 percent of GDP compared to 4.8 percent of GDP in the average OECD country—this poor performance can be explained not by a lack of resources but by the uneven response of the school system to a number of challenges.
With support from the IDB, the government piloted a teacher training model in 2012 called Visible and Tangible Math in the Belize City District. As a result of the pilot, standardized test scores improved. The general approach was to train the teachers in mathematics in the way they were expected to teach their own students, using hands-on activities tailored to each student.
The pilot identified that teachers often have limited competency. Teaching has lacked an active engagement of students to better support them in developing critical-thinking skills. There is ample evidence in the education field that teacher quality is the most important school factor in the quality of student learning, so addressing this problem is a priority.
The IDB loan will support Belize as it works toward improving teacher quality by strengthening the selection and training of new teachers, including on-site practice in training programs for teachers and principals, and strengthening the government’s quality assurance of education.
PANAMA – Equitable Access to Electricity
With ample evidence that equitable access to electricity remains a challenge for Panama, an IDB loan was approved in 2014 to help mitigate this challenge through the expansion of Panama’s rural electricity network which was rated as evaluable with a DEM score of 8.6.
One salient feature of the design of this project is its solid monitoring and evaluation plan. As noted in chapter 6, there is ample room for increasing the scope and range for rigorous evaluations of infrastructure projects. In this case, detailed surveys will be conducted, and a rigorous econometric approach will be used to assess the socioeconomic effects of the project. Particular attention will be given to the impact on women and children. This evaluation will contribute to the still-limited knowledge in the region on the development impact of rural energy systems.
Electricity is a basic requirement for development and a key service to improve lives. Lighting alone can bring many benefits, including increased study time, improved health, and new business opportunities.
In Panama, private companies in charge of distribution usually invest in areas that are easily accessible, leaving poor and isolated households outside of the electricity grid. Only 71 percent of Panama’s rural households have access to electricity, as reported in Panama’s 2010 census.
Since 2006, the IDB has contributed to the expansion of Panama’s rural electricity coverage. To date, 13,443 families have benefited, but more than 111,000 rural families still lack electricity.1 The project approved in 2014 will further expand electricity coverage to another 6,126 rural-households (by connecting them to the grid), plus 45 schools and 10 health centers. In addition, the program will use renewable energies to expand access to 4,218 more isolated households, plus 62 schools, 11 health centers, and 14 indigenous territories. In total, 10,344 households will benefit. The project will encourage public-private partnerships.
Panama underlined the importance of equity in access to energy by approving Law 58 on March 30, 2011. The law establishes that the government will continue to support the electrification of underserved rural areas.return
JAMAICA – Citizen Security and Justice Programme III
Crime and violence is a daunting problem in Jamaica and the IDB has been supporting the country’s efforts to mitigate this challenge since 2001. In 2014, the IDB approved Jamaica’s Citizen Security and Justice Programme (CSJP III), the third program in a series to mitigate these challenges in the country.
The project was rated highly evaluable, with a DEM score of 9.2 based on the empirical evidence of the problem, the strategy to address it, the quality of the program logic, the economic analysis and, in particular, its evaluation plan. This is especially important, since, as mentioned in the 2013 Development Effectiveness Report, one of the great weaknesses of projects in this area is precisely the lack of access to evidence-driven analyses to support the intervention. The program itself will add to the knowledge base of the sector through the inclusion of two randomized control trials. In addition, CSJP III benefited from the evidence found in prior phases.
The 2014 CSJP program focuses on using solutions that have proven successful elsewhere. The solutions include: (1) providing skills and opportunities that enable residents to change their attitudes toward tolerance instead of violence; (2) delivering youth programs to promote life skills, education, and job preparation; and (3) increasing access to services that complement the formal court system to promote reconciliation and alternative dispute-resolution mechanisms.
Since 2001, the IDB has supported the first two phases of the Programme (CSJP I and II). CSJP has expanded from nine to fifty communities in eight parishes since its inception, and a recent tracer study of CSJP II (2009-2013) by the IDB´s Office of Evaluation and Oversight reported that the murder rate in the eight parishes where the program is functioning has declined 43 percent, compared to a 35 percent decline nationally. Additionally, 44 percent of targeted residents said that crime in their community has decreased in the past five years, compared to 28 percent of residents from other communities.
PERU – Using Information technology to Improve the Agriculture Sector
The IDB approved a project in 2014 that aims to strengthen the capacity of farmers in Peru to negotiate prices, identify marketing alternatives, adopt new technologies, improve hedging against risk, and gain more access to credit. The project was rated highly evaluable, with a DEM score of 10, not only because of the quality of the evidence based diagnosis and identification of a solution, but also thanks to a rigorous experimental evaluation design, as well as the results of other similar approaches elsewhere, provided to gain insights on the effectiveness of the intervention that will be piloted through the project.
In Peru, agriculture accounts for 7 percent of GDP and provides 80 percent of the nation’s food supply. However, a significant share of the agricultural output is sold at a lower price than it could have been sold in a higher-priced market, given that only 8 percent of farmers use some type of market information on prices to make decisions. This translates into lost earnings for farmers.
The project is based on a research study in Peru that involved providing cellphones set up to receive text messages with prices of 17 crops in nearby markets. Farmers who received the service sold their crops at prices 13 percent higher compared to those who did not receive it, thus earning a higher value for their agricultural output.
Producers who lack resources or do not understand the benefits of the cellphone service are not always initially willing to pay for it, so the IDB project makes financing available to private cellphone providers until the service becomes sustainable.
In benefitting farmers, the project aims to increase GDP growth while also helping to reduce poverty and improve equity. When the project is completed, an evaluation of its short- and long-term impact will provide information for the scaling of this in Peru as well as the design of similar projects in the region in the future.
The program logic category is divided into three dimensions: problem diagnosis, proposed interventions and solutions, and results matrix quality. One crucial element assessed by the DEM in this section is if there is an evidence-based assessment and solution for the project. What does this mean? As a first step, projects ought to clearly identify a problem to address, and then make sure there is enough evidence to guarantee it is indeed a priority challenge. For illustrative purposes, let’s imagine that a government faces the challenge of low crop yields in a particular plot of land. To address the issue jointly with the IDB, the project team would ask a series of questions to understand the problem. For example, is there evidence that indeed the crop yields are low? How do they compare with average crop yields in other nearby regions?
Once there is enough evidence to substantiate the claim that crop yields are indeed lower than desirable; the DEM ensures the project explores the root causes of this circumstance so that the proposed solution can be optimized.
When the team has gathered enough evidence to identify the root causes of the problem the DEM also serves as the tool that questions whether there is evidence the solution proposed by the team will work.
For this purpose, let’s imagine that the solution of the IDB team centers on the identification of a promising staple that is less water-dependent than average. Has this promising seed thrived in other similarly dry lands? If yes, how do the two pieces of land compare? If the seed thrived in a dry land that nonetheless enjoyed higher annual rainfall, are we adapting the technique and increasing the frequency of irrigation? The DEM examines whether there is evidence the problem we are trying to address indeed exists, and evidence that our proposed solution has been engineered to be as effective as possible. In other words, this amounts to assessing whether the project is based on a sound theory of change. In many cases, this analytical work relies on the knowledge produced by the Bank, such as impact evaluations, technical notes, and Sector Framework Documents, among others. The financial support of Technical Cooperations has been crucial to produce this analytical work.
Figure 2.5 shows that in 2014, ratings in all three dimensions of the program logic section were quite close to the possible maximum score (that is, very close to or above 90 percent of the maximum possible score).
A significant portion of the progress in evaluability scores in recent years is related to the economic analysis category. Following our example, it is essential to consider whether the value of the mature staple, or benefits of the project, is equal or greater than the initial cost of the seed being planted and the materials used to fertilize and care for it as it grows. The valuation of benefits is done from the vantage point of society. If firms benefit from the intervention, the analysis also considers effects on employees, competitors, suppliers, etc. Likewise, the costs included go beyond IDB-project financing to include all costs necessary for the intervention’s implementation. Economic analyses carried out by project teams can use either a cost-benefit or cost-effectiveness methodology, while for policy-based loans a general economic analysis is used.
Figure 2.6 shows the proportion of projects with the different economic analysis methodologies used. In 2014, a cost-benefit analysis was carried out for the vast majority of projects. This type of analysis is a very valuable tool to gauge the efficiency in the use of Bank funds. A properly implemented cost-benefit analysis, must be able to show that the economic benefits generated by the project not only is greater than its costs, but also that its return is higher than the cost of capital. Figure 2.7 shows the average economic rate of return (ERR) by sector, and the number of projects in the sector for which the ERR was estimated.
Finally, in the monitoring and evaluation category, the IDB has made great progress in using the most rigorous methodologies to evaluate its projects. In 2014, impact evaluations based on experimental or quasi-experimental methods were proposed for 43 percent of projects approved (compared with 49 percent in 2013)2. Ex-post cost-benefit analyses were proposed for 49 percent of projects, ex-post cost-effectiveness analyses for 5 percent, and before-after methodologies for 4 percent (Figure 2.8).
As described previously, the IDB uses the DEM as a tool to enhance project evaluability during design. Generally speaking, a higher DEM score is desirable. However this may not always be the case. For example, if an innovative program is designed, it will be natural that it cannot be supported by empirical evidence regarding its effectiveness, since it has never been tried. In such a case, the project may score low on the section rating whether the solution is evidence-based. When this happens, the project should be accompanied by a rigorous impact evaluation that will document if the proposed intervention was effective or not. As pioneers of development, it is desirable for the Bank to forward such innovative solutions. Thus, managing for development results cannot rely on a single tool, but must complement this with other relevant and pertinent information. The DEM score is used as a tool to proxy project evaluability of projects during Board Approval, but these discussions are rich in nature and extend beyond the DEM score.
The DEM & IDB Priorities
The DEM is also used during project design to ensure alignment with the IDB’s priorities, such as lending targets, as well as alignment with country priorities, as they have been outlined in the Country Strategy. Together, IDB staff and borrowing countries define how a country’s priorities coincide with the Bank’s development goals. The Country Strategies provide a common reference framework for the medium-term that guides the Bank’s actions and gears it toward results.
Specifically, the DEM’s strategic alignment section captures the extent to which projects contribute to (1) the IDB’s Results Framework, which establishes the Bank´s corporate priorities given the region’s development challenges, and (2) country strategy objectives.
Of the 105 public sector projects approved by the IDB in 2014, 99 were aligned both with the Bank’s institutional priorities and the corresponding Country Strategy. Five were not aligned to the Country Strategy, and one was only aligned to the Country Strategy but not with the institutional priorities, but they were approved because they addressed important development challenges. In 2013, there were three projects that were neither aligned to the IDB’s institutional priorities or the Country Strategy.
Non-Sovereign Guaranteed Operations
The IDB fosters development through its private sector operations in order to create opportunities for individuals, and foster economic growth. Private-sector led growth tends to be sustainable growth, as ventures must stand the test of markets.
The IDB supports the development of the private sector through its sovereign guaranteed (SG) operations, which channel resources to support policies aimed at improving the business environment, and overcoming other challenges that constrain investment. The IDB also provides direct finance to private entities through its non-sovereign guaranteed (NSG) operations.
This work is done through the IDB’s Structured and Corporate Finance Department (SCF), in charge of primarily large-scale projects, and its Opportunities for the Majority Sector (OMJ), which invests in business models that can be scaled up and benefit the Base of the Pyramid in the region.3 In 2014, the IDB approved 63 private sector projects for a total of $2.8 billion up from $2.1 billion in 2013. Figure 2.9 provides a breakdown of the 2014 approvals by their respective IDB divisions.
The IDB has had a development effectiveness framework in place for its non-sovereign guaranteed (NSG) operations since 2008. This framework has its foundations on frameworks at other development finance institutions (DFIs), and in particular, is harmonized with the Evaluation Cooperation Group Good Practice Standards (ECG-GPS). The basic elements consist of an ex ante development effectiveness scoring tool, a supervision reporting tool that tracks actual results annually, and self-evaluations (conducted for mature projects) which are validated by the independent Office of Evaluation and Oversight (OVE). Since its inception in 2008, this framework has been adjusted to sharpen its focus and increase its relevance. Currently, in the context of the private sector reform discussions, the IDB Group is working on the harmonization of the different development effectiveness tools used by each private sector window.
In order to determine which private sector projects the IDB should support, and how the IDB can best add value to those operations, all NSG operations are carefully screened and analyzed to determine their potential to bring development benefits to Latin America and the Caribbean. In order to ensure rigor and consistency in assessing NSG projects, the IDB uses a comprehensive Development Effectiveness (DE) toolkit, which is applied to virtually all operations.4 This toolkit has been refined and improved over the years, and the start of 2014 marked the introduction of a completely revamped approach to assessing development effectiveness for NSG projects. Boxes 2.5, 2.6 and 2.7 highlight promising NSG projects approved in 2014.
PARAGUAY – Interfisa: “Nde Vale”
Nde Vale is an expression that comes from the Guaraní dialect and, although it does not possess a direct translation, it is attributed to a person that is both persistent and courageous in the activities they undertake.
Through its customized “Nde Vale” product, Interfisa—a specialized financial institution—seeks to address the unmet needs of women entrepreneurs in Paraguay. Micro-entrepreneurs at the base of the pyramid face significant obstacles in obtaining financing for their businesses due to a lack of appropriate records that can be used as a means for verifying their repayment capacity, a scenario that is worse for women-owned businesses.
The lack of collateral, such as access to land titles—which are generally in the name of the male head of household—and traditional structures for investment decision making in the family, put women at a greater disadvantage when applying for any kind of financing. Such market failures are often exacerbated for the following reasons: (1) the market often does not recognize women entrepreneurs as relevant target customers; (2) the informal economy in which most women operate limits their productivity and restricts their access to formal sources of financing; (3) the amount of collateral required to obtain a loan under these circumstances is up to three times greater than the loan amount; and (4) a large percentage of women do not have the technical knowledge to apply for a loan or even know what their capital needs are.
With this project, Interfisa seeks to contribute to their clients’ development, particularly in rural areas, by extending loans to micro, small, and medium enterprises (MSMEs) owned or led by women. In order to cater to these beneficiaries, “Nde Vale” does not require collateral, has a flexible repayment schedule adjusted to the customers’ cash flow, and a streamlined loan contract. Moreover, Interfisa has designed an incentive system to motivate its loan officers to reach women who reside in areas outside of urban centers and are currently not part of the financial system. Overall, with the support of a US$5 million loan from the Opportunities for the Majority Sector (OMJ), this project aims to benefit approximately 5,072 women micro-entrepreneurs and 9,105 small and medium enterprises (SMEs) owned or led by women in Paraguay, who previously had little or no access to formal sources of financing.
ECUADOR – Banco Guayaquil: A multi-booked SCF-OMJ project
Agriculture in Ecuador represents the main economic activity in rural areas, where 41 percent of the population still lives in poverty. This is in part due to the lower levels of productivity in this sector, which consists primarily of micro, small, and medium-sized farmers who lack access to knowledge on productive best practices, technology, and effective financing to become more competitive in both domestic and foreign markets.
With the main objective of improving access to finance for agricultural producers in Ecuador, the Inter-American Development Bank (IDB), acting jointly through its Structured and Corporate Financing Department (SCF) and its Opportunities for the Majority Sector (OMJ) joined forces and—along with the “China Co-Financing Fund for Latin America and the Caribbean” as well as diverse B lenders—is providing Banco Guayaquil (BG) with US$100 million in medium-term financing to support its expansion strategy into the agroindustry sector.
To achieve greater penetration, BG seeks to develop partnerships with various anchor companies, thereby using an alternative channel to reach producers that previously lacked access to formal sources of financing. These anchor companies, in addition to sharing their lists of generally small producers with BG, will also provide information on their various crop specific production cycles to ensure that the terms and conditions of the loans placed by BG fit the needs and repayment capabilities of its producers. Additionally, they are also expected to provide technical assistance on best production practices to loan recipients in their value chain. The productive sectors identified as potential recipients of support from this program are the cocoa, corn, dairy, meat, palm oil, and sugar sectors.
ARGENTINA – Autopista Urbana in Buenos Aires
The Autopista Urbana S.A. (AUSA) program represents a pioneering NSG operation, as it is dedicated specifically to addressing the critical public health issue of road safety. It is the first time a private sector concessionaire raises funds especially to finance road safety concerns.
Every year in Latin America and the Caribbean, approximately 150,000 people are killed in road accidents with another five million injured. The economic cost of crashes in the region is estimated at $60 billion per year, which amounts to 1-2 percent of regional GDP. In response, global leaders have espoused the Decade of Action for Road Safety (2010-2020) with the stated goal of reducing global fatality projections by half by the year 2020.
In the city of Buenos Aires road incident rates among users and pedestrians are at approximately 10,000 per year with nearly 25 people injured daily. In particular, the most problematic infrastructure issues in Buenos Aires are “at-level” road/rail crossings where vehicle and bus traffic crosses the urban train network. During 2005-2010, there was an average of 714 incidents per year at these road-rail intersections resulting in over 350 fatalities per year across the city.
The AUSA project provides infrastructure investment, private sector management and coordination with public authorities to improve road safety in Argentina. The approach includes two road safety and mobility components: (1) a large-scale infrastructure investment program; and (2) a partnership between the Bank and Autopista Urbana S.A. (AUSA) which commits to a program of continuous improvement with the goal of reaching international best practices based on the Bank’s Highway+ program.
Expected development impact includes: (1) reduction in the number of traffic incidents; (2) time savings for both the road and rail systems; (3) enhancements to the environmental sustainability of the road network; and (4) temporary construction jobs. Also, the Road Safety Action Plan will: (1) increase accountability and transparency of road safety performance; (2) improve data collection, management and analysis of road safety data and (3) move AUSA towards international best practices in road safety.
The NSG Development Effectiveness toolkit consists of the following three key components: (1) a results matrix (for a quick overview of the main expected project results); (2) a development effectiveness assessment (resulting in the project score); and (3) an evaluability checklist (resulting in the evaluability score).
Similar to SG projects, each NSG loan proposal includes a results matrix, which—in a table format—clearly states the Project Objective and the main expected development results (project outputs and results), with baseline and target values. The results matrix is tracked throughout a project’s life, and therefore allows for a quick understanding of whether a project indeed reaches the most important objectives and results it had set out to achieve.
Beyond simply stating expected development results in the results matrix, they are scored against quantitative or qualitative benchmarks within the NSG development effectiveness assessment. This allows comparisons among different projects’ expected performance in terms of development effectiveness.
The NSG development effectiveness assessment5 analyzes the expected development results and the significance of IDB’s contribution for each project along the dimensions of (1) expected development results, (2) additionality (both financial and non-financial), and (3) alignment to IDB’s strategic development objectives and the country strategy or program.6
The development results dimension, considers a company’s business performance, contribution to economic and private sector development, and an environmental and social assessment. The additionality section assesses the importance of the IDB role in making the project feasible (or more sustainable). For example, this could be through the provision of financing on more favorable terms than available in the market (financial additionality). Also, additionality is related to enhancing a project’s development results or its design (non-financial additionality). It is particularly important to assess additionality for private sector operations, for which the intention is to ensure that the IDB always adds value to the projects beyond what purely commercial sources can offer. The strategic alignment dimension takes into consideration how well the project fits in with the IDB’s stated corporate priorities, as well as the country’s development goals as expressed in the Country Strategy or Country Program documents.
Starting at project eligibility, a project score (0-10) is generated for the performance areas of development results (with subscores for the underlying criteria mentioned above) and additionality, whereas the strategic alignment dimension is assessed as either “aligned” or “not aligned”. The overall project score is a weighted average of the performance area scores. The project score is updated and finalized at the quality and risk review meeting and at Board approval. This score is then revised annually (as part of the NSG Project Supervision Reports or PSRs) throughout a project’s life, reflecting how actual project performance compares to the expectations set at approval.
For NSG projects approved in 20147, the average overall project score was 7.7, indicating high expectations for projects’ development results. The average development outcome score was 7.4 and it is expected that the IDB will provide strong value to the transactions, with an average additionality score of 8.4. Figure 2.10 details project scores of 2014 approvals.
In terms of alignment, 34 of the 37 NSG non-TFFP8 loans and guarantees approved in 2014 were aligned to both IDB corporate strategic priorities, as well as country strategies or programs. One project was aligned only to the country priorities, and two projects were aligned to IDB corporate priorities but not to the Country Strategy or Program documents.
Similar to the evaluability tool for SG operations, the NSG evaluability checklist9 assesses whether a project is well laid-out, and set-up in such a way that it will be possible to evaluate the results once the project is finished. The checklist has three main components: (1) project logic; (2) financial and economic analysis quality; and (3) monitoring and evaluation quality. The project logic section of the checklist considers how well the project is justified—is there evidence of the need for the intervention? The section also includes an assessment of how clearly the project’s development objectives are stated, and whether there is evidence for the validity of the way in which the project addresses the identified development problem. Furthermore, this section analyzes the justification for the need for IDB involvement in the relevant private sector context. The score also appraises how well the document identifies the intended beneficiaries and the development benefits the project is expected to achieve.
Finally, a crucial component of evaluability is the verification of the vertical logic of the project, in other words how the project objective relates to the outputs which in turn lead to results, as reflected in the results matrix.
The financial and economic analysis section evaluates the quality of the project’s financial and economic analyses in accordance with the standards set forth by the ECG-GPS.10 The monitoring and evaluation section assesses the mechanisms and provisions put in place to ensure a meaningful ex-post project evaluation, as well as development results tracking and reporting.
Finally, an overall Evaluability score11 is produced based on the scores for each of the three areas described above (with each area equally weighted). For projects approved in 201412, the average evaluability score was 8.2.13 Figure 2.11 below summarizes evaluability scores for NSG approvals in 2014.
It is important to note that evaluability levels were redefined in 2014 to differentiate projects with top scores from other projects. In particular, the “highly evaluable” level was modified so as to exclusively include projects with DEM scores equal or greater than 9. This has allowed for a distinction of the highest-scoring projects as a way to incentivize continuous improvements in the evaluability levels of all other projects.return
There are cases where a given project combines different impact methodologies to evaluate its different components.return
The IDB Group also supports private sector projects through the Inter-American Investment Corporation and the Multilateral Investment Fund, which complement the Bank’s products and services to the private sector by focusing their operations on supporting the development of micro, small, and medium-sized enterprises. return
The main exceptions are individual operations under the Trade Finance Facilitation Program (TFFP), which are assessed not at the operation, but at the Program level.return
The instrument used for this assessment is the NSG Development Effectiveness Matrix (NSG DEM). Its application results in the NSG Project Score (which scores expected development results), for which there is no equivalent in SG operations. In order to avoid confusion about the meaning of the term “DEM”, it is important to point out that the “DEM Score” for SG operations corresponds to the Evaluability Score of NSG operations.return
These dimensions follow the ECG-GPS (Evaluation Cooperation Group’s Good Practice Standards) for private sector project evaluation.return
There were 63 NSG loans and guarantees approved in 2014, of which 26 were part of the Trade Finance Facilitation Program (TFFP). Due to their homogeneity, TFFP operations are assessed not at the individual operation, but rather at the program level. Therefore the figure reflects the scores of the 37 non-TFFP NSG loans and guarantees.return
Trade Finance Facilitation Program (TFFP).return
While an evaluability score had been produced for NSG operations since 2011, the evaluability checklist was completely overhauled and harmonized with the SG evaluability approach for operations in 2014. Due to the novelty of the approach, the year 2014 is considered a pilot exercise, which is currently being reviewed for feedback by the Office of Evaluation and Oversight (OVE). Based on such feedback and other lessons learned, the checklist and scoring approach might be adjusted for future operations.return
For real sector projects, an ex-ante cost benefits analysis is carried out to calculate the FRR (Financial Rate of Return) and ERR (Economic Rate of Return) or ROIC (Return on Invested Capital) and ERIOC (Economic Return on Invested Capital). For financial intermediary projects, the financial performance of the client financial institutions, as well as the economic sustainability of their portfolio are the main criteria proposed by the ECG-GPS.return
As previously mentioned, the Evaluability score is referred to as “DEM score” for SG operations.return
This excludes TFFP operations, as well as other transactions for which evaluability is not assessed at the individual operation level, but at the facility or program level.return
As the year 2014 marks a pilot of this new NSG evaluability checklist, the scoring mechanics and the average score might still be revised based on OVE’s feedback.return