EPATEE topical case study : Linkage between M&V tools (data collection) and evaluation (complementary analysis)
Summary – Key ideas
The objective of the case study is to analyse the linkage between data collection related to monitoring of energy savings and evaluation on energy efficiency policies. In an ideal scenario, this topic has to be considered as early as possible in the policy cycle, when preparing the policy and planning the related monitoring and evaluation activities (particularly the data collection).
One of the main issues regarding the linkage between regular data collection through monitoring and evaluation is the bridge to combine the two. Namely data verification as to how the data is being collected and what kind of results it shows, and how to subsequently present the data as a reliable evidence base to the decision makers. This can be an issue for evaluators, especially in terms of how to ensure data reliability and to present verified data to policy makers who will then decide on future implementation of evaluated energy efficiency policies.
From the examples shown in this topical case study, it can be relevant to focus on:
- the data collected from ex-post analysis of energy savings,
- the question of data being collected through intertwined systems and/or platforms that deal with monitoring, verification and evaluation of data, where data is collected through a common methodology and verified through automatized checks and carefully specified search filters,
- to conduct a thorough final analysis with an assessment (or at least an approximate consideration) of net energy savings, exploring to what extent the baseline used to estimate the energy savings reflects a counterfactual of what would have happened in the absence of the policy measure (see more details in the EPATEE case study about Evaluating net energy savings: Voswinkel et al., 2018).
This final analysis may serve as a convincing point for the policy makers to continue, adapt (or cancel) the energy efficiency policies and programmes. This establishes evaluation as a necessary and crucial step alongside measurement, monitoring and verification of energy savings in order to finally have a successful long-term policy, taking advantages of the evidence base from monitoring and evaluation for continuous improvement and fine-tuning.
Monitoring is indeed commonly used to organise a regular feedback loop and reporting, that provides information to know if the policy is on track to its target(s), to detect changes in trends about actions implemented, costs and other monitored indicators, and to see if changes in policy settings are needed.
Monitoring might have two limitations:
- it is often based on declared or self-reported data from the participants;
- it is difficult to change the type of data collected, as it might require complicated or costly changes in the monitoring tools and processes, and it might increase administrative burden for both, participants and the public authority in charge of the scheme.
The first point is often tackled by verification and control processes. The second point calls for further and specific data collection and analysis that are the core of evaluation works. In some cases, evaluation is also used to complement verification processes or check if they are robust enough.
On the basis of evaluation, policy makers may move forward with the energy efficiency policy even if the targets have not been necessarily achieved, since evaluation can serve as a launching point to correct faults found in the data collected and to achieve those targets in the long-term. Nevertheless, one cannot focus on evaluation as the only tool in achieving a comprehensive outlook of all errors and frauds within the process, but its main goal can be to take notice of that these may exist and can be corrected. Above all, evaluation can provide analyses complementary to monitoring. Particularly to investigate issues raised along monitoring and for which regular data collection is not enough.
Table 1. Pros and cons of linking regular data collection to evaluation.
|– Direct availability of data ready for evaluation
– Transparency on the type of data collected through the implementation of consistent methodologies
– Basis for the future advancement of energy efficiency policies
– Improvement on the existing policies through evaluation of carefully collected data
– Early identification of issues worth to be investigated through ex-post analysis
|– Question of validity of data collected (particularly in case of self-reported data)
– Possible confusion between different administrative levels responsible for various aspects on advancing energy efficiency policies (policy makers, data managers, evaluators, funds…): need for good coordination and communication
|– Data collected on a regular basis might not be enough to evaluate all aspects (and particularly net energy savings)|
Brief description of the examples
The Croatian example focuses on a potential future merge of two online platforms. One focusing on monitoring and verification of energy savings (System for Monitoring, Measuring and Verifying Energy Savings – SMiV), and the other on reporting data on energy consumption in the public sector (Energy Management Information System – ISGE). The advantage in this case would be to automatize data collection and processing of energy savings, and to have both ex-ante and ex-post perspective: SMiV calculates energy savings based on engineering estimates, while ISGE can offer ex-post data from billing analysis. This can however cause additional administrative costs (but also reducing these costs in the long-term) and preliminary confusion among different administrative levels responsible for the management of these tools.
The Austrian example was triggered by the EED and subsequently by the implementation of an EEO (Energy Efficiency Obligation) in Austria. In order to provide obligated parties with a reporting system that minimises their administrative costs, an online database with standardised reporting spreadsheets was created. It is worth noting that the database is also used by companies for the reporting of the energy audit obligation according to EED Article 8 and by public authorities reporting energy savings from alternative measures (for EED Article 7). The database is adapted regularly according to experiences and needs of both the National Energy Efficiency Monitoring Agency and front-end users of the database.
The Finnish example provides insights to the long-running monitoring system for voluntary Energy Efficiency Agreements and the Energy Audit Programme. The system underwent a major overhaul ten years ago when a monitoring system with a common web-based database for the two policy measures was established. This entailed multiple benefits in terms of data collection, processing and utilization both by administrators and rapporteurs. There is one designated webmaster (Motiva Oy) who has been assigned to maintain and develop the database and who is able to process the raw data from the database for various ex-post and ex-ante reporting needs.
The French example deals with data collection in the framework of the white certificates scheme. It gives an overview of data collected by public bodies to frame and monitor the scheme, and of processes implementation.