Box 1.1

Learning From Failure:

Challenges in Defining Indicators, Calculating Baselines, and Setting Targets for the CRF 2012–2015

Part of Chapter 1 - Monitoring Results for Greater Development Effectiveness

The IDB places utmost importance on development effectiveness. one aspect of this broader effort entails continuously working to improve the existing performance measurement architecture. the lessons learned in the development and implementation of the CRF 2012–2015 became important inputs for the CRF 2016–2019 approved in november 2015, and the resulting improvements in both the CRF’s content and the processes surrounding it are expected to give way to a more useful CRF. In this vein, performance information emerging from annual CRF progress reports will inform corporate processes, such as business planning, budget allocations, and personnel work program management, which will in turn enable better-informed decision-making across the Bank.

In addition to a number of broader lessons discussed in Chapter 5, the Bank has learned three important insights regarding the construction of the CRF Indicators.

1. Institutional-level target-setting has limitations due to the complex nature of development work.

Factors such as the change in IDB partners’ demand for Bank-supported interventions, the extension of project execution beyond reporting periods (e.g. infrastructure projects may need longer maturity periods to generate measurable results which extend beyond a given CRF period), and the emergence of other circumstances that are challenging to forecast at project approval (for example, delays in related bidding processes) have prevented some indicators from reaching their established targets.

2. Target-setting for the first CRF was also particularly challenging due to the lack of accurate data to rigorously calculate baselines.

At the time the CRF 2012–2015 was established, the Bank did not yet have all the necessary systems in place to establish baselines for all indicators. this is an ongoing challenge when designing new indicators as systematic data collection typically begins onlyonce the indicator has been agreed upon. In the case of the CRF 2012–2015, the first DEO that reported formally on CRF results (DEO 2012) did not include baselines for a significant number of indicators, although these were calculated for later DEO editions. this ultimately means that caution should be exercised in drawing conclusions about IDB performance based on the final status of the CRF 2012–2015 indicators.

3. Selecting the right scope for output indicators, and defining indicators that truly measure IDB corporate performance are also challenging.

In some cases, indicators were defined with too narrow a scope, such as “inter-urban roads” as opposed to “roads” more broadly. In the case of IDB operational effectiveness and efficiency (Table D of the CRF), the challenge was defining indicators that capture Bank performance as opposed to the combined performance of the executing agencies with which it works (for example, regarding disbursement levels). In this sense, as seen in Chapter 4, a number of efficiency indicators capture factors that are partially beyond the Bank’s control. Thus, interpreting these indicators requires additional context and analysis.

In preparing the CRF 2016–2019, the Bank took stock of these insights and other lessons (discussed in Chapters 4 and 5) to improve the usefulness and overall quality of the CRF indicators along with the processes associated with it. special attention was given to fostering a participatory process, which involved maintaining frequent dialogue with both the Board of executive Directors and Bank staff across all Divisions in the development of the indicators. this all helped instill a sense of institutional ownership while enhancing the accuracy of the target-setting process. Better availability of data also helped the Bank to construct more robust baselines and targets.

Continue reading Chapter 1 - Monitoring Results for Greater Development Effectiveness