Box 5.1

Learning From Failure:

Lessons from Project Implementation

Part of Chapter 5 - In Pursuit of Sustainable Economic and Social Development

The IDBG understands that to successfully implement its development mandate, each project it supports must be as effective as possible. The Bank’s experience — both successes and failures — in supporting countries in implementing projects under the IDB-9 sector priorities and measuring progress with the CRF Output Indicators has yielded valuable lessons that will help improve the design and execution of future operations.

1. Constructing an institutional-level results framework is a tall order.

Establishing Bank-wide indicators that fully capture the diversity of the IDB’s portfolio is much more challenging than designing project-level indicators. As Chapters 1 and 2 highlight, three factors that contribute to making this challenging are: dynamic client demand, delays during project execution, and large-scale or complex projects that require more time to yield results. For example, from 2012 to 2015 the Region’s demand for financing for teacher training programs was lower than initially expected. Delays during execution can relate to bidding processes or sudden fiscal constraints. Finally, large scale projects, such as São Paulo’s metro Line 4 (see Story 2.11 in Chapter 2), can take more than the four year CRF period to achieve substantial results: in this case, the benefits to its 198 million passengers per year. As also discussed in Chapters 1 and 2, there were also a number of challenges to target-setting due to the lack of accurate data at the time the CRF was developed. This led to some targets being either significantly lower, or significantly higher than what was ultimately achieved.

2. The importance of collaboration is not a cliché.

Collaboration and multi-sectoriality are persistent concerns at the IDB. When it comes to project design and execution, collaboration — whether internal or external or between public and private sources of funding — has not yet reached its full potential. Some of the Bank’s goals, such as closing gaps in the broadband and certain infrastructure systems, cannot be accomplished without public-private partnerships, or without teaming different areas of the Bank and of our partner countries in designing and executing multi-sectoral projects.

3. To make a big impact, start small.

Before investing in large-scale projects, it is important to implement pilots and evaluate them rigorously. Furthermore, limited resources don’t necessarily mean that a project can’t have a big impact. Especially in times of fiscal tightening measures in parts of the Region, the careful design of an operation and strategic allocation of resources needs greater prominence.

4. Traceability is the key for successful monitoring.

From the design of a project through its completion, project teams need to identify their contribution to the CRF in a consistent and unambiguous manner. With the implementation of Convergence, a tool to manage operations from beginning to end, project teams have the opportunity to report and track their contribution to the CRF through the progress they report in the Results Matrix. Through the identification of the logic of the intervention and the development of the Results Matrix, project teams should ensure the quality of the data to avoid potential inconsistencies and to ensure that achievements do not go un-reported.

5. Documenting findings and recommendations is powerful.

Gathering findings and recommendations after a project is completed is a fruitful way to learn from the Bank’s failures and successes. Nonetheless, they are only useful if they are duly documented and reported in the Bank’s monitoring systems. The Bank’s newly implemented platform — Convergence — is becoming an important tool to document findings and recommendations, and the Bank continues working to improve its performance and usefulness. Through enhanced data collection and reporting capabilities, Convergence is expected to yield a more efficient and transparent CRF reporting process, whereby it will be possible to see which specific projects contribute to CRF Indicators.

6. Communicating both positive and negative findings is important.

Highlighting both positive and negative findings is a powerful tool to improve the design of Bank-funded projects. Furthermore, the dissemination of the results from an evaluation is essential to support evidence-based policy decisions and help the Bank understand what works and what does not in development. The Bank has made great strides over the past few years, but there is still further work to do.

7. Flexibility is fundamental.

Being able to make adjustments to a project, whether in regard to a timeline, an indicator or a planned evaluation can often affect whether the operation is determined to be successful or not in reaching its desired development outcome. For example, impacts can go undetected if the timing of the impact evaluation is inadequate. At the same time, it is important to consider that changes made to the project need to be carefully documented to allow others to retrace the decision-making process and learn from the adjustments, and should not be undertaken simply to make the project a “success”.

8. Timing of project evaluations is critical.

Project evaluations need to be timed properly in order to be able to assess their full results. Evaluations carried out at very early stages will fail to capture the full outcome that the project sought to achieve.

Continue reading Chapter 5 - In Pursuit of Sustainable Economic and Social Development