Evaluation approaches complementary to impact evaluations

• Theory-based evaluation

“Theory-based evaluation approaches provide an overarching framework for understanding, systematically testing and refining the assumed connections (i.e. the theory) between an intervention and the anticipated impacts.

The focus of theory-based evaluations is not only on understanding whether a policy has worked, but why, and under what conditions a change has been observed. Theory-based evaluation will therefore generally seek to identify each of the various elements in the underlying logic model, and examine the links between each element.”

Source: UK Magenta Book (“recommended central government guidance on evaluation that sets out best practice for departments to follow”), Chapter 6 “Setting out the evaluation framework” (see p.61 of the pdf: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/220542/magenta_book_combined.pdf )

For more details about the concepts and principles of theory-based evaluation and how to apply this approach, see the dedicated webpage of the Treasury Board of Canada Secretariat: https://www.canada.ca/en/treasury-board-secretariat/services/audit-evaluation/centre-excellence-evaluation/theory-based-approaches-evaluation-concepts-practices.html

More specifically to the energy efficiency field, the Intelligent Energy Europe project AID-EE (Active Implementation of the European Directive on Energy Efficiency) applied in 2006-2008 this evaluation approach to 20 energy efficiency policies across different sectors and different European countries.

The final report of the project includes general guidance about how to apply theory-based evaluation, see chapter 2 of: https://www.ecofys.com/files/files/aid-ee-2007-summary-report.pdf

This is illustrated with practical examples through the 20 case studies: https://www.eceee.org/library/conference_proceedings/eceee_Summer_Studies/2007/Panel_4/4.070/

The table below provides the list of these AID-EE case studies (per sector):

• Process evaluation

“Process evaluation, action research and case studies can be used to evaluate the implementation and delivery of a policy to provide feedback on a wide range of issues. These can include whether the policy is being implemented as planned, what is working more or less well and whether it is delivering expected outputs and outcomes.

Process evaluation cannot determine whether a policy “worked” this can only be achieved using an impact evaluation. It can, however, complement an impact evaluation by providing crucial insights into why a policy did, or did not, work and test the logic model on which the policy is based.

(…) Process evaluation, action research and case studies use a range of qualitative and quantitative research methods including one to one interviews, group interviews, surveys and observations. Multiple methods are often used to provide triangulation of data and corroborate findings.”

Source: UK Magenta Book (“recommended central government guidance on evaluation that sets out best practice for departments to follow”), Chapter 8 “Process evaluation, action research and case studies” (see p.85 of the pdf: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/220542/magenta_book_combined.pdf )

More specifically about energy efficiency policies and programmes, detailed guidelines can be found in the California Energy Efficiency Evaluation Protocols (pp.131-142): http://www.calmac.org/publications/EvaluatorsProtocols_Final_AdoptedviaRuling_06-19-2006.pdf

• Economic evaluation (or Cost-Benefit Analysis)

An economic evaluation is most often based on an impact evaluation, and goes further by comparing the costs and benefits of the policy measure.

“A reliable impact evaluation might be able to demonstrate and quantify the outcomes generated by a policy, but will not on its own be able to show whether those outcomes justified that policy. Economic evaluation is able to consider such issues, including whether the costs of the policy have been outweighed by the benefits. There are different types of economic evaluation, including:

• cost-effectiveness analysis (CEA), which values the costs of implementing and delivering the policy, and relates this amount to the total quantity of outcome generated, to produce a “cost per unit of outcome” estimate (e.g. cost per additional individual placed in employment); and
• cost-benefit analysis (CBA), which goes further than CEA in placing a monetary value on the changes in outcomes as well (e.g. the value of placing an additional individual in employment). This means that CBA can examine the overall justification for a policy (“Do the benefits outweigh the costs?”), as well as compare policies which are associated with quite different types of outcome. CBAs quantify as many of the costs and benefits of a policy as possible, including wider social and environmental impacts (such as crime, air pollution, traffic accidents and so on) where feasible. The Magenta Book uses the very general term “value for money” to refer to the general class of CBA-based approaches, but it is important to recognise the more general scope of CBA which include those impacts which are not routinely measured in money terms.”

Source: UK Magenta Book (“recommended central government guidance on evaluation that sets out best practice for departments to follow”), Chapter 2 “Identifying the right evaluation for the policy” (see p.24 of the pdf: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/220542/magenta_book_combined.pdf )

The UK Magenta Book does not directly provide guidance on economic evaluations, but refers to the Green Book where users can find general guidance on this approach, and particularly about CBA: https://www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in-central-governent