Chapter 4: Assessing Results at Project Closure
In this chapter:
- Assessing Sovereign Guaranteed Operations at Closure
- Why a New Project Completion Report System?
- Box 4.1: Piloting the New Project Completion Report
- A Closer Look at the New Project Completion Report Criteria
- A Glimpse at the Types of Lessons the New Project Completion Report will Convey
- Box 4.2: COLOMBIA – The Educational Equity Program in Bogota
- Box 4.3: MEXICO – Strengthening of the Oportunidades Human Development Program II
- Social Policy for Equity & Productivity
- Box 4.4: HAITI – The Emergency Response to Contain Cholera Program
- Infrastructure for Competitiveness & Social Welfare
- Institutions for Growth & Social Welfare
- Assessing Non-Sovereign Guaranteed Projects at Closure
Assessing Sovereign Guaranteed Operations at Closure
With the revamping of the Project Completion Report (PCR) system starting in 2013, the IDB completed the adaptation of its results measurement instruments according to the Development Effectiveness Framework (DEF). The DEF, through its focus on results measurement sets high standards, and allows for a better design of projects as well as providing a mechanism for their monitoring and evaluation.1
Why a New Project Completion Report System?
The PCR system in place prior to 2014 provided limited insight into project performance at completion, and little comparability across projects. The system classified IDB projects at closure according to self-assessment ratings, which were not based on quantitative, evidence-driven, or verifiable dimensions. The new system, designed in consultation with multiple stakeholders, addresses and improves on the weaknesses of the previous system. The new PCR was piloted in 2014 (see Box 4.1) and is scheduled for rollout in 2015.
Piloting the New Project Completion Report
The new PCR system was tested and refined in a pilot run for 12 projects that spanned most of the IDB’s subregions and sectors of operation. The pilot included five projects in social protection and education, four in infrastructure and three in institutional capacity (see Table 4.1).
Findings from the pilot were incorporated into the new PCR system. For example, the list of questions used to assess the four core criteria at project completion was fine-tuned and the scoring criteria more specifically defined.
The pilot of the new system demonstrated that the new PCR indicators allow for sizable variance and differentiation among projects that overperform versus those that underperform. Using the new system, it is possible to assess with more accuracy the achievement of results and objectives as well as the degree of efficiency and sustainability of the interventions.
The fact that the system is stringent enough to rate 33 percent of the sample as modest achievers with scores of around 0.6 shows that the standard for development effectiveness has risen under the new criteria.1 Only 25 percent of piloted projects were rated as high achievers with scores in the highest range of around 0.9. Looking ahead, once there is a critical mass of completed new PCRs, such cross-sectional comparison will also be possible for sectoral and geographical clusters. Detailed results of the PCRs from the pilot projects are presented in Table 4.2.
The high variation in the scores of the pilot projects, and the number with modest performance indicators, may be due in part to a significant portion of the projects analyzed having been designed and started before 2008, prior to implementation of the Development Effectiveness Framework (DEF). The project with the highest score in each sector started in 2008 or later, consistent with the notion that the DEF was developed precisely to design more evaluable interventions. This could mingle the negative effects of a more stringent methodology on the evaluation of pre-2008 projects and the positive effects of improved design, implementation, and monitoring that the DEF methodology could have had on post-2008 projects. Taking this into account, the new PCR system will only be applied to projects approved since 2009 and thus designed using the DEF and its associated development effectiveness matrix.
For comparison, under the old PCR system, 76% of operations approved in 2014 were considered satisfactory in terms of development results.return
In the new system, projects are assessed according to four core criteria: development effectiveness, efficiency, relevance, and sustainability. The analysis supporting each criterion is evidence-driven. Noncore criteria are also analyzed, including the contribution of a project to both the Bank’s and the host country’s strategic objectives. Reflecting a concerted effort by the IDB to increase the objectivity of the criteria used, the new system sets forth clear principles and guidelines for the preparation of the individual reports. A checklist used to rate the performance of the completed intervention integrates specific rules to ensure objectivity in analyzing achievements under each core criteria. The PCR also focuses on documenting lessons learned to better understand what works for development and so that they can be used in the design of new development projects.
A Closer Look at the New Project Completion Report Criteria
The evidence-driven approach of the PCRs strengthens the objectivity and evaluability of the assessments. In addition, attribution has a clearly defined role in the assessment of effectiveness: it is important to demonstrate not only that project goals were met, but also that those achievements were at least partially attributable to the intervention itself and not to other determinants.
The new PCR also analyzes the ex-post efficiency of projects by incentivizing cost-benefit or cost-effectiveness analysis at closure. In other words, the IDB is accountable for both the accomplishment of its proposed goals and the judicious use of resources allocated to accomplish them.
The four core evaluation criteria used under the new PCR system—effectiveness, efficiency, relevance, and sustainability—constitute a structured and standardized methodology to rate the performance of a completed project. The PCR system has specific guidelines that detail how to assign a score in assessing each of the four criteria.
First, the PCR assesses project effectiveness, which is the extent to which expected project outputs were completed and attributable results achieved. The methodology verifies how many of the outputs and results that were targeted when the project was designed were achieved at its completion and how attributable they are to the Bank’s intervention. To determine the second criterion, project efficiency, the PCR analyzes whether the project benefits surpassed its costs, and whether the project’s goals were achieved at a reasonable cost. The assessment of project relevance looks at the capacity to adapt and align the project to the goals of the IDB and the borrowing country. Finally, the assessment of project sustainability rates the extent to which the operation ensures that the results achieved will not be reversed.
A Glimpse at the Types of Lessons the New Project Completion Report will Convey
The new PCR system was tested in a pilot run of 12 projects. Table 4.1 presents the projects evaluated grouped by sector, and Table 4.2 groups the projects according to the IDB’s institutional priorities and summarizes the projects’ performance indicators.
One of the goals of the new PCR system is to strengthen institutional learning. Important lessons drawn from the successes and difficulties faced by the projects evaluated under the pilot run could benefit future IDB operations. The sections below summarize those lessons, grouped according to three of the IDB’s institutional priorities (which were presented in Chapter 1). Lessons are drawn from all of the pilot projects, including the three discussed in more detail in Boxes 4.2 to 4.4.
COLOMBIA – The Educational Equity Program in Bogota
A US$60 million loan was designed to improve the equity and quality of education in the Colombian capital. In response to rapid demographic growth and migration to the city by rural populations victimized by violence in the countryside, the Secretariat of Education of Bogota set a goal of expanding the capital school district to accommodate 200,000 new students between 2004 and 2008.
In this context, the IDB loan was approved to contribute toward closing part of this gap by expanding public educational infrastructure and revamping the school system to improve educational quality for primary and secondary students. The project built 20 new schools, reaching the target of 61,100 new students enrolled, and contributed to higher completion rates as well as lower repetition and dropout rates.
Despite these gains, evidence on improved learning rates among students was rather thin. This can be related to the fact that although the program succeeded in improving several student results, it had very modest effects on improving the quality of teachers. In fact, the project trained only about a fifth of the teachers who had been expected to enroll in training. Another negative was the unexpected increase in bullying and violence among students attending the new mega-schools built under the project.
Two important lessons useful for the design of future operations stem from this project. First, improving education requires appropriate incentives to secure the participation of teachers. For example, such incentives could link teachers’ participation in training to career advancement. Second, appropriate management of the learning environment is necessary, especially in schools serving children from low-income homes, as in the case of households with families displaced by violence known in Colombia as desplazados. Future projects of this nature should include the teaching and application of civic, social and conflict resolution skills from inception.
MEXICO – Strengthening of the Oportunidades Human Development Program III
A US$800 million loan was the third in a series of operations in Mexico whose total financing was $2 billion. The initial operation helped to consolidate the Mexican flagship conditional cash transfer (CCT) program in rural and semi-urban areas. CCT programs are meant to increase human capital by making cash transfers conditional on the recipient complying with certain education or health requirements. This is done with the goal of interrupting the generational transmission of poverty. The program was successful in inducing the rural poor to use education, nutrition, and health services, which was a condition for participants to receive the cash transfers.
The second phase of the program aimed to replicate its success beyond rural areas by expanding CCTs to urban areas. However, this effort did not meet with the same success as the first program phase. Between 2002 and 2008, 43 percent of the urban beneficiaries dropped out, mostly because they did not meet their co-responsibilities to receive the transfers.
The third program phase aimed to enhance the effectiveness of the program in urban areas, particularly by helping the government improve the profiling of potential urban beneficiaries by establishing suitable co-responsibilities and participation rules. The number of workshops required for beneficiaries was reduced while the customer service centers available to the beneficiaries were increased. In order for the program to be better aligned with urban residents’ profiles, scholarships available for secondary school increased, while those intended for primary school decreased.
As a result of the third program phase, urban beneficiaries increased from around 3.2 million in 2009 to 4.9 million in 2012. However, being able to measure the long-run benefits of CCT programs, beyond the immediate assistance they provide to the poor, remains a challenge.
Among the program findings is that preventing school dropouts requires that CCT operations carefully profile potential beneficiaries and consider providing an incentive-compatible bundle of benefits and an accessible network of care/education providers. The findings also point to the importance of designing long-term evaluations of CCTs that examine their impact on building human capital and developing the capacity of beneficiaries to permanently escape from poverty.
Social Policy for Equity & Productivity
Five PCRs conducted under the pilot program examined projects classified under this institutional priority: two for projects to improve conditional cash transfers (Expansion of the Familias en Acción Conditional Cash Transfer Program Phase II in Colombia, and Strengthening the Oportunidades Human Development Program III in Mexico); one to improve educational quality and results in Colombia (Educational Equity in Bogota); one to improve living conditions of indigenous communities in Chile (Integrated Development of Indigenous Peoples, Phase I & II – Orígenes); and one to contain the cholera epidemic in Haiti (Emergency Response to Contain Cholera).
Most of these projects achieved all their intended goals, but there is room for learning. The CCT programs in both Mexico and Colombia had major accomplishments. The Mexican beneficiaries showed higher secondary school completion rates, and improved childhood nutritional levels. The Colombian CCT program improved school attendance and enrollment, but could not improve dropout rates or reverse the worsened student’s performance. The number of health center visits also fell short of the project’s targets.
In the case of the Colombian education project in Bogota, dropout and repetition rates were successfully reduced, but challenges remain in the improvement of students’ learning rates. Furthermore, there was a low participation rate for its teacher-training program.
The IDB had considerable success in several of the other social policy programs evaluated under the pilot run. Through the Orígenes project, the welfare and living conditions of Chilean indigenous communities was improved by creating tailor-made governmental structures to serve this population, including mechanisms to supply specific public services. These public services include the bilingual intercultural education program provided in 301 public schools, the intercultural medicine services provided in 10 public health centers, and the support to the use of indigenous medicine through 376 sub-projects. The quick response and effectiveness of the cholera abatement operation in Haiti was also a great success (see Box 4.4).
HAITI – The Emergency Response to Contain Cholera Program
A US$20 million grant to Haiti, provided support to the government to abate the cholera outbreak in late 2010. This epidemic hit only 10 months after a devastating earthquake, when much of the damaged and destroyed health, sanitation, and transport infrastructure around Port-au-Prince and Leogane had yet to be rebuilt. Within a month of the cholera outbreak there were over 11,000 confirmed cases and 180 casualties, and the numbers were growing exponentially.
Given the extremely virulent nature of cholera and its capacity to kill a healthy adult within hours of infection, this emergency required immediate attention. The IDB responded with this program within six weeks. In the face of the fast-spreading epidemic, the most important aim of the intervention was to reduce mortality through timely case management and improved access to safe water. For the most part, the cholera epidemic was successfully contained.
The project excelled not only in its timely responsiveness but also in its capacity to reinforce the stewardship of the Ministry of Health in ensuring rapid, effective, and transparent execution of its duties, while at the same time using the country’s own planning, project financial management, and procurement capabilities. This was achieved thanks to the integration of UNICEF as the executing agency in charge of the deployment of the aid. Over 50 percent of the grant was disbursed via UNICEF. Indeed, a significant factor in the success of the project was UNICEF’s ability to build on the strengths of its implementing partners. Upon the project’s approval, UNICEF was granted an exception to use its own procurement guidelines and processes, and this resulted in significant time savings for delivering emergency care.
The IDB’s ability to adapt and react quickly in an emergency will serve as a point of reference for the design of future emergency response operations. Among the recommendations for such operations is that they include simplified and expedited approval processes, and that they establish protocols and conditions for partnering deployment through international aid agencies.
Why Were Some of the Goals Only Partially Achieved?
Drawbacks in this group of projects most commonly stemmed from political and institutional challenges. In a few cases, design flaws, weak monitoring plans, and inertia also caused difficulties.
Political and institutional issues could have affected both the education program in Colombia and the indigenous peoples’ support program in Chile. In Colombia, several changes in the administration of Bogota’s city hall and national administration occurred during the lifespan of the education project. A change in government and the corresponding loss of political support also affected the Chilean program.
Land acquisition and the bidding process also proved challenging for Colombia’s education project in Bogota and resulted in fewer schools being built. The project built 16 of the 20 targeted schools. An additional 4 schools were built with local funding, but not with the loan as originally intended.
Both the Mexican and Colombian CCT programs built on prior and successful program rounds. Lessons captured from these projects point to a certain inertia as a result of this “follow-on” circumstance. The programs did not adequately coordinate the multisectoral components of the interventions, and it was a challenge to implement timely adjustments in the supply of services (such as health and education) upon which the programs are based.2
Orígenes in Chile, which was approved years before the IDB adopted the DEF, encountered difficulties in properly defining measurement strategies for what was a unique intervention. In turn, this affected the monitoring of progress during execution. In large part, this was due to the fact that the program was atypical. It had an unconventional design so that it could be tailored to address the interests of the indigenous communities it served. However, some needs were not apparent during design, such as the ancestral organizational structure of the indigenous population, and this created delays and difficulties during implementation, making corrective action necessary. The Bank financed a tailor-made monitoring system for the project, which has allowed monitoring of more than 1,000 individual sub-projects of the program.
What Did We Learn from these Pilot Projects?
The pros and cons of a big brother: Operations built on the successes of previous phases can benefit from experience and in some cases demonstrable effectiveness. Nonetheless, this may generate resistance to adapting the status quo to ever-evolving contexts. Such was the experience of the Colombian and Mexican CCT programs evaluated under the PCR pilot. Operations based on past successes should be thorough in making a proper diagnosis of the specific program context in order to guide potential adaptations of the pre-established model. This will be helpful in determining what the new operation wants to achieve and how it may differ from the preceding interventions.
Monitoring in less conventional contexts: When innovating new approaches and working in unconventional contexts, more attention should be given to properly designing a monitoring and evaluation system that is sufficiently tailored to capture the specific circumstances. Such is a lesson from the Chile project. When possible, the IDB should also promote the involvement of the administrative and statistics divisions of the executing agencies at national and subnational levels.
Minding the context and collateral effects: It is crucial to thoroughly assess the context and collateral effects that could influence a project’s implementation. For example, for education projects, beyond establishing the physical need for schools, it is necessary to be aware of the social context. As shown by the education project in Colombia (see Box 4.2), issues can arise when a large number of students from resettled and poor families are placed together in a mega-school. In such a context, unrest may ensue and with it bullying; when this does happen it can undermine some of the project’s achievements. For its part, the Chilean project showed that when working with indigenous communities it is advisable to consider the multimodal definition of some indigenous economies that are not only based on land cultivation.
Beneficiaries and benefactors: Active beneficiary involvement is helpful to achieve a project’s results. Projects such as the one in Chile designed to respond to the needs of indigenous communities need to take into account ancestral local authorities in the decision-making process. It is their opinion that will ensure communities’ cooperation with the program’s objectives. In addition, when possible a program should involve local communities in the fiduciary and procurement processes. In such cases, it is important to assure the consistency of these processes with the corresponding national systems and their requirements.
Hope for the best, plan for the worst: Strengthening the planning capacity within a program is critical for its success. All the loans in the PCR pilot under the social protection pillar (with the exception of the emergency loan to Haiti) experienced problems during implementation that might have been avoided with better planning. Hence, it is important to prioritize this capacity and provide the resources needed to build it. For projects with multiple executing agencies, it is necessary to clearly define the decision-making process and the sequence of the planned activities from project inception. This can help to avoid delays and ensure effective inter-agency coordination. Changes in authorities, counterparts, and priorities are day-to-day realities that public sector operations cannot control. Thus, the IDB’s project implementation plans need to be cautious in programming and take into consideration the contingencies and adverse circumstances that could affect execution.
Flexibility in Emergency Responses: The success of the IDB’s flash operation in Haiti shows that emergency grants crucially depend on flexibility and speed to accommodate alternative procurement networks managed by international aid partners.
Infrastructure for Competitiveness & Social Welfare
Four PCRs conducted under the pilot program examined projects classified under this institutional priority. They included two transport programs: the Acoyapa–Costa Rica Border Road Integration Program to link border areas with continuous, safer, faster, and cheaper transportation; and the Andean Highway Program to reduce transport costs and travel time and provide alternative routes for export and import traffic to Atlantic and Pacific seaports. The other two projects were an energy program and a social housing program. The first was Development of a New Sustainable Energy Matrix, a four-phase program in Peru to consolidate energy growth. The second was the Social Interest Housing Program in Honduras, which financed individual subsidies for new houses and housing improvements, as well as collective subsidies to build basic infrastructure in urban poor neighborhoods.
These projects varied considerably in their overall performance, but all achieved some level of success. The Andean Highway program had some successes beyond those that were captured by the indicators and targets of its original evaluation plan. Therefore, a proper effectiveness analysis of the project could not be conducted. The Acoyapa–Costa Rica road project successfully reduced transport costs and increased transport speeds. The housing program in Honduras delivered most of its intended outputs, although the sustainability of its achievements was jeopardized by the public institutional weakness of the country’s housing sector. Peru’s energy project achieved most of the proposed policy changes and serves as a successful model to guide similar projects in other countries.
Why Were Some of the Goals Only Partially Achieved?
There are a variety of reasons for the project shortcomings among this group, including a lack of definition of clear indicators to adequately capture successes and difficulties on the ground that resulted in cost and time overruns. Other weaknesses were context-driven, such as the degree of violence in the intervened area.
The Andean Highway Program was approved in 2000, prior to implementation of the IDB’s Development Effectiveness Framework, and so some indicators lacked targets and means of verification. Cost and time overruns were significant, and an inadequate assessment of land conditions resulted in design errors that required corrective action during implementation. Taken together, these problems translated into significant increases in the project workload and costs, as well as delays in execution.
The performance of the housing project in Honduras was affected by high levels of violence in the area where collective subsidies were provided. In addition, the collective subsidies component was the only project component that collected relevant ex-ante and ex-post information that allowed for a before-after comparison for evaluation. When the project was approved in 2006, the IDB’s risk analysis tool had not yet been revamped. Perhaps the project would have benefited from the new risk assessment modality. Under the new Risk Tool, which was implemented in 2010, in addition to identifying risks and corresponding mitigating actions, project teams proceed to monitor indicators related to the implementation of these mitigating actions.
Peru’s energy project exemplified good performance all around. It was the only programmatic policy-based loan evaluated in the pilot, and its satisfactory achievement of results drew on the flexibility built into the design of such loans that allows for adjustments of the agreed-upon policy conditions upon disbursement of each of the four tranches. The main result of the program was a long-term national energy policy built upon the new energy matrix. There were three other achievements thanks to the new energy matrix. First, there was a reduction of the use of fossil fuel based energy. Second, the project resulted in the approval (at the feasibility level) of public projects of gas supply outside the capital city. Third, renewable non-conventional energy projects (ethanol, biodiesel) were promoted and there were definitive concessions for wind energy projects and hydropower plants. All of these accomplishments benefited from the strong institutional capacity and coordinated work of the involved agencies (Ministries of Finance, Energy, Agriculture and Environment), which led to an integral approach during the design and implementation of reforms.
What Did We Learn from these Pilot Projects?
Diagnosing conditions on the ground matters for preemptive measures that circumvent risk and foster success. This is a key element in heading off potential conflicts that can be an obstacle to achieving results. The success of infrastructure projects requires early identification and corresponding mitigation measures to address any conflicts that might arise in the intervened area.
Supervision is essential. Cost overruns are common to infrastructure programs, so continuous supervision of the equilibrium between cost, times, scope, and quality is important. Such supervision allows for corrective and timely actions that guarantee timely completion of the works.
Institutional strengthening matters for infrastructure programs. Infrastructure programs tend to prioritize works over institutional strengthening. In the Acoyapa–Costa Rica, Andean Highway and Honduras projects, institutional strengthening components were overlooked, making adequate management of complex processes such as compensating resettled dwellers even more challenging. Institutional strengthening components need to be measurable and to define clear action plans and corresponding indicators and targets.
The higher the quality of the results matrix design, the more evaluable the project at closure. Most of the evaluated projects under this institutional priority were designed before the new DEF. Their resulting lack of clear standards for measuring results had a clear impact on the ability to measure the performance of these projects. The new PCR evaluation system will only be applicable to projects with a Development Effectiveness Matrix, that is, projects approved after 2008. These projects more clearly identify the monitoring and evaluation instruments up-front, and they have pre-set verification methods with executing agencies (see Chapter 2 for more details).
Institutions for Growth & Social Welfare
Three PCRs conducted under the pilot program examined projects classified under this institutional priority. The objective of the Integrated Development Program for Campo Grande–Procidades in Brazil was to revitalize the city center and provide institutional strengthening of the municipality. The second program provided financing to Agencia Financiera de Desarrollo in Paraguay and supported the productive sector by facilitating medium-to-long-term funding for small and medium-sized enterprises (SMEs). The Institutional Strengthening Program for the National Senate aimed to improve the parliamentary, administrative, and human resources management capacity of the Argentine Senate, with an emphasis on transparency.
The success of these interventions varied. The program in Paraguay successfully increased the maturity of loans to private sector beneficiaries from 2 to 10 years, but evidence was incomplete regarding the project’s effects on SME productivity. This evidence will be captured by an ex-post evaluation subsequent to the third project in the series. The Argentine Senate program showed modest success, but that might have been due to a lack of appropriate indicators to measure its results. Three of the five results indicators were actually products, and the indicator selected to measure the intervention’s impact was inadequate because it referred to an index with a much broader scope than the intervention’s targeted development goals.3 The program in Brazil achieved and surpassed both its output and results targets and successfully orchestrated public consultation forums to benefit project decision-making, increased the value of downtown residential properties, reduced travel time in the city areas were interventions took place, and brought down Campo Grande’s communications costs.
Why Were Some of the Goals Only Partially Achieved?
The shortcomings and failures of these projects largely stem from political changes, complex execution mechanisms, insufficient experience of the executing agencies, and the lack of effort to quantify key indicators for the problems diagnosed. The Paraguay program had great success in expanding loan maturities, but it will only be able to capture evidence on beneficiary productivity after the third project in the series. The Argentine program was not designed with proper results indicators, rendering the measurement of results nearly impossible. This project also suffered three administration changes in the Senate that affected and ultimately delayed its implementation. It was also affected by a four-level executing mechanism (strategy, direction, operations, and coordination) that introduced too many complexities in decision making and execution.
The Brazilian program’s success reflected a good alignment between the activities financed with the urban transportation master plan of the executing agency. This allowed for ample project support not only from local authorities but also from the private sector and civil society. The activities carried out within the program were opened for consultation in public opinion forums, ensuring that final decisions included the views of beneficiaries.
What Did We Learn from these Pilot Projects?
It can be beneficial for executing agencies to be more familiar with the IDB’s projects. In the case of the Argentine Senate Program, a lack of such familiarity resulted in differences in interpretation between the decision-makers of the executing agency and its coordinating unit. When familiarity is lacking, complementary training is necessary for the executing agencies.
To the extent possible, complexities in execution should be avoided. The complexity of the execution mechanism for the Argentine Senate Program resulted in implementation delays and hindered execution dynamics. Decision levels are important for the purpose of checks-and-balances, but should also be flexible enough to avoid causing delays during implementation.
Defining measurable and relevant indicators for results is key for the measurement of results as well as for the proper evaluation of impact. The Argentina program was approved before the IDB’s new DEF was in place. The Paraguay program was approved six months after DEF implementation, which might not have been long enough for teams to overcome the learning curve associated to such an institutional change. Thus these program designs did not fully benefit from the DEF, and in fact their shortcomings evidence the need for a well-defined results matrix with indicators that can measure the goals that an intervention aims to achieve.
A clear identification of the intervention area is critical. Bringing interventions together in specific geographic areas and clearly identifying the area of intervention at the time of design can avoid resources being diverted to activities that do not have a clear effect on welfare. This can also facilitate the task of selecting a rigorous control group for impact evaluations. The Brazilian program showed that putting in place a clear urban planning framework can help coordinate and articulate tasks and avoid duplicity.
Assessing Non-Sovereign Guaranteed Projects at Closure
As discussed in Chapter 2, the development effectiveness score rates projects according to their expected (at entry) and achieved (at implementation) development results (financial and economic performance), profitability, additionality, and IDB’s work quality. The development effectiveness project score is also updated during the self-evaluation conducted once during the life of NSG projects when projects are mature, fully disbursed and in operation. Self-evaluation of each project has been in place for NSG operations since 2005 and has followed the ECG-GPS4 standards for private sector operations.
During 2014, the independent Office of Evaluation and Oversight (OVE) and the IDB Group private sector windows worked together to redesign the self-evaluation guidelines in order to embrace a higher standard, and to allow for greater harmonization between NSG and SG methodologies. It is expected that the new guidelines will focus on the areas of Achievement of Project Objective, Relevance, Effectiveness, Efficiency and Sustainability. The new evaluation framework is expected to be operational by mid-2015 (hence, self-evaluation ratings are not reported for 2014).
See Chapter 1 for more details on the DEF.return
See Chapter 6 for more details on the supply-side challenges of CCTs.return
The impact of the intervention was measured by an indicator of perception from Latinobarómetro. The indicator measures the population’s confidence in the nation’s legislative body, and in the Argentine evaluation its variation surpassed the target. However, the indicator is too broad, as the referred perception involves not only the perception of the Senate but of the members of Parliament.return
ECG-GPS stands for the Evaluation Cooperation Group’s Good Practice Standards. The ECG is a group established by the heads of evaluation of multilateral development banks.return