EPATEE topical case study : Comparing Estimated versus Measured Energy Savings
Summary
This topical case study compares two approaches to evaluate energy savings: detailed engineering estimate and evaluations based on measurement, which could be a direct measurement (often within the industrial sector) or a billing analyses (often within the residential sector). In essence this case study looks at the difference between ‘estimated’ versus ‘actual measured’ energy consumption for the baseline and the situation after the energy efficiency actions have been implemented. When reliable measured or metered data of energy consumption can be collected for all participants and that external factors can be controlled, measurement-based approaches will likely provide more reliable results. But in practice, these conditions are far from being commonly met. This topical case study therefore discusses the pros and cons of both approaches and the cautions to take when comparing their results. This topic is a general issue, relevant for all sectors and types of policy measures.
We clarify first the different types of data and methods used in energy savings evaluations and considered in this case study. Among the different evaluation methods distinguished in EPATEE, the ones relevant for this case study are the five bottom-up methods: direct measurement (method 1), billing analysis (method 2), deemed estimate of unitary energy savings (method 3), mixed deemed and ex-post estimate (method 4), and detailed engineering estimates (method 5).
We then review when to use each approach (‘estimation-based’ and ‘measurement-based’) according to literature. This neither is a black-and-white situation: methods are often used in combination with each other, very much depending on data availability. Larger programs tend to need more data and more types of data for evaluation purposes; especially when factors as the rebound effect are investigated.
A synthesis concludes the literature review by explaining what main factors can cause differences between estimated and measured energy savings, and how they can be analysed and taken into account. References are provided for readers looking for examples or more details.
Finally, two practical examples illustrate the topic further. For the Better Energy Homes scheme in Ireland, both estimation-based (engineering calculations) and measurement-based (billing analysis) approaches have been used. The engineering calculations are used to monitor and report energy savings on a regular basis, providing a continuous feedback on the results of the scheme. The estimation-based approach makes possible to calculate savings based on the data that can easily be collected from each participant (e.g., type of actions installed, type of buildings). They represent ‘theoretical’ energy savings as they are based on standard assumptions (e.g., about energy behaviours or performance of the actions). Then billing analysis was used for an ex-post evaluation that aimed at investigating the energy savings actually achieved by the participants, and to what extent energy savings could be attributed to the scheme (using a control group). This required tremendous efforts in collecting billing data. The difference between the two approaches is considered as the ‘loss of savings’ (or shortfall1) due to e.g. a rebound effect or poorly installed equipment. The evaluators acknowledged that the difference in the results from the engineering estimates and the billing analysis could not be disaggregated into the potential influences of the various factors that might affect these results. In practice this means that billing analysis does not always deliver perfect results or perfect understanding of the results, unless data collection was planned early enough. This would indeed require to collect many complementary data2. The evaluators also mentioned that despite all their efforts put in data collection and rigorous data analysis, there were still some bias that could create uncertainties in the results from the billing analysis.
The second example is about a renovation programme in Lithuania. Difficulties in collecting metered data were also encountered, restricting the opportunities to form a sample to compare estimated and measured energy savings. While for Ireland the data were collected from the gas network operator, for Lithuania the data were collected from a district heating company. In both cases the estimated energy savings were based on simplified engineering calculations as defined for the respective national Energy Performance Certificates. In both cases, the measured savings were about 36% lower than the estimated savings.