Box 3.1

Overcoming Challenges:

Lessons to Better Evaluate Project Impact

Part of Chapter 3 - Evaluating Projects to Enhance Learning and Policy Making

First, timing is of the essence. The timing of an evaluation is often based on the planned timeline and funding of the project, as opposed to the right time to assess the project’s effectiveness. If conducted too early, an impact evaluation might suggest that a project had no impact. However, such a conclusion can be inaccurate if the full effect of the intervention has not yet had the time to materialize. The project team therefore needs to carefully decide when to conduct the evaluation, and modify the evaluation plan based on the actual progress of the project.

Second, “the only thing that is constant is change.” This quote from the ancient Greek philosopher Heraclitus is true not only for life in general, but also for impact evaluations. Over the course of an impact evaluation, conditions often change. Institutions can change and this can affect programs. For example, an evaluation of youth programs in Jamaica ran into problems because the program was moved to a different ministry during the course of the evaluation. Anticipating and adapting to such changes is important both for a successful project and its impact evaluation.

Third, get ready to chase your control group. Control groups are often comprised of specific individuals. Keeping up with them may not always be easy, a lesson learned in the same evaluation of youth programs in Jamaica. Individuals can move because of personal reasons, a job change, or other circumstances, and they might not always inform the project team. Teams need to be aware of possible relocations and find ways to stay in contact with everyone in the group.

Fourth, short-term results do not equal long-term results. If a project shows results shortly after implementation, this does not mean that the results will be maintained in the long term. Similarly, a project might not show significant results in the short term, but will reveal its impact over the long term. Depending on the project type and the desired project outcomes, pursuing an impact evaluation over a longer period of time can be worthwhile to determine the full impact. An additional challenge is to find a funding source to conduct such an evaluation, years after a project has been completed.

Fifth, long-term evaluations and resources are needed. To be able to understand whether or not a project worked in the long term, longitudinal evaluations are needed. A strategy needs to be established to identify where knowledge gaps are and their relevance to future interventions; and to find a source of funding to ensure financing of long-term evaluations.

Continue reading Chapter 3 - Evaluating Projects to Enhance Learning and Policy Making