03 – Evaluating legislative/normative measures on energy efficiency in non-residential buildings using engineering savings estimates
This specific guide can be applied to evaluate the savings due to legislative/normative measures, such as energy performance standards, in the non-residential buildings sector using the engineering estimates method. It includes guidance and explanations specific to this combination of policy measure type, sector and method. As well as links to general guidance and explanations, that can also apply to this combination.
1. USE OF THE GUIDE – AUDIENCE, OBJECTIVES AND FOCUS
The primary audience for this guide concerns energy efficiency programme designers, implementers or supervisors, and evaluators looking for guidance on the evaluation process of energy savings in the scope of this tool.
Although the application of the tool will generally concern the (sub)national level, account will be taken of issues at EU level when relevant (e.g. the specific format of saving figures for the EED).
This guide is not about the preceding step in the evaluation process, the choice of the method. About this previous step in the evaluation process, see the guidance provided here. However, after presenting the capabilities and limitations of the tool at hand, the user will be offered alternatives for the method within this tool (see section 6).
The objective of this guide is to provide:
- Information on the scope of the tool that enables the user to decide whether this tool is suited to his/her needs, and whether complementary or additional method(s) could be needed or useful (section 2);
- Guidance about specifying the evaluation objectives and requirements (section 3);
- Guidance about key methodological choices to calculate energy savings (section 4);
- Guidance about the inputs (data requirements) and outputs of the method (energy savings metrics) (section 5);
- Possible alternative methods (with pros and cons) (section 6)
- Background about evaluation results other than energy savings (section 7);
- Relevant examples, case studies and/or good practices (section 8);
- Relevant references for further reading (section 9).
The specific guide is intended for assessing realised (ex-post) energy savings. However, earlier (ex-ante) evaluations of expected savings are taken into account if available (see section 4).
The focus of the tool is on impact evaluation, i.e. determining the energy savings, but not on how this has been reached through a step by step process with intermediate results (process evaluation).
Readers looking for the basic and general principles of energy efficiency evaluation may find the following link useful.
2. SCOPE OF THE TOOL – POLICY, SECTOR and METHOD
2.1 About legislative / normative measures
Normative legislative measures in residential buildings in practice come down to efficiency standards. For standards, a distinction can be made between specific standards on for example insulation (e.g. the heat resistance of double glazing) or boilers (minimum conversion efficiency standards) and system-wide energy performance standards (e.g. maximum gas consumption for new dwellings).
The policy measure type described in this guide is restricted to system-wide standards. These standards are associated with energy performance certificates and coefficients. Allocating an energy performance certificate or coefficient is no guarantee that the actual energy consumption will be in line with the certificate. To get to know the actual energy consumption, an evaluation by measuring or billing analysis is necessary.
Another aspect that has a large influence on energy consumption in this case is the behaviour of the occupants and the occupancy rate of dwellings. When it is not possible to gather individual data by performing interviews, the monitoring and evaluation should be done for a group of (comparable) buildings, where individual differences no longer play a role.
More information and examples on the different subtypes residing under the main type (legislative / normative measures) can be found here and here.
More detailed information on the evaluation of legislative / normative measures can be found here.
2.2 Evaluation for a combination of policy measure types
This specific guidance addresses legislative / normative measures. When legislative / normative measures are combined with other policy measure types it is assumed that the overall savings are mainly resulting from the policy measure at hand. However, the evaluation concerns the combined savings effect of both policy measures.
The tool is not capable of attributing part of the (overall) calculated savings to each of the policy measures (see also double counting in the section on Gross to Net savings).
2.3 Evaluation when combined with energy taxes
The calculated savings effect for legislative / normative measures will overlap with that of the energy tax. The tool is not capable of attributing part of the (overall) calculated savings to either the policy measures at hand or the energy tax. For dealing with this overlap, see section on Gross to Net savings.
2.4 About non-residential buildings
Information on (sub)sectors defined in the Toolbox can be found here, chapter 2, p.17
Most non-residential buildings concern offices, schools, hotels, shops, etc. in various parts of the services sectors, but office buildings in industrial sectors are also part of non-residential buildings.
The buildings sector, both non-residential (in this specific guidance tool) and residential buildings are characterized by high dependency of energy consumption from insulation and other energy efficiency or energy generation technologies. Evaluation has to distinguish between new buildings that can be planned including energy efficient technologies from the start, and renovation of existing buildings where new technologies have to be adapted to the building’s given infrastructure. Due to the fundamental differences between implementing savings actions in existing and new buildings, evaluation methods can differ. Where necessary this tool will distinguish the two cases.
2.5 About engineering estimates
Information about the various evaluation methods can be found here, table 1 and 2. This source also covers the combination of the method at hand with other methods, which will be dealt with below.
Engineering estimates for energy savings employ a detailed analysis. This can be done using calibrated simulation or more or less complex modelling of an individual unit (e.g. an energy balance of an individual building) or an individual company (each unit is a participant of the company)
2.6 Complementary methods to determine total savings
Complementary methods are methods that are required, in addition to the primary selected method, to calculate energy savings.
The method at hand is meant to calculate unitary savings for one building. These unitary savings should be multiplied by the number of actions or participants in order to have the calculated total savings. The number of actions or participants can be obtained in various ways. See the link here, table 2 and 3 for more information. The number of actions is sometimes directly available from the monitoring of policy measures, such as in case of a subsidy scheme. Possible complementary methods are direct monitoring of the actions or participants, statistics or sales and market data.
The unitary savings calculated with the method at hand may vary due to the non-uniform nature of the saving action. In that case the total savings are calculated by summing up the unitary savings for all saving actions (instead of multiplication average unitary savings with number of actions).
2.7 Additional methods to increase reliability of the results
An additional method can be applied on top of engineering estimates to improve the reliability of the evaluation results and/or the cost-effectiveness of the evaluation approach.
The engineering estimates method is very suitable for calculating the savings for a complex energy using system like a non-residential building or specific industrial processes. Engineering calculations include assumptions (e.g. about behaviors) that can create uncertainties in the results. It can therefore be useful to check the engineering estimates with measurement (direct measurement or billing analysis) as additional method. The combination can increase the reliability of the savings figures in a cost-effective way. For possible combinations with an additional method see chapter 6 here.
3. EVALUATION OBJECTIVES and REQUIREMENTS
3.1 Meeting evaluation goals and ambition
The table shows whether this tool can be used to report on general evaluation goals or criteria. See also this document.
|General types of evaluation goals or criteria||Level of ambition||Remarks|
|Calculation of realized energy savings from saving actions||Fair (renovated buildings)/Good (new building)||Depending on monitoring quality|
|Calculation of energy savings attributed to the policy measure(s)||Low (renovated buildings)/Fair (new buildings)||Due to difficulties with additionality|
|Cost-effectiveness of saving action (for end-users)||Fair||Depending on quality of saving calculations|
|Cost-effectiveness of policy (government spending)||Low||Due to difficulties with additionality. Government spending is nearly zero|
|CO2-emission reduction from saving actions||Fair (renovated buildings)/Good (new building)||Depending on quality of saving calculations|
|CO2-emission reduction attributed to the policy measure(s)||Low (renovated buildings)/Fair (new buildings)||Due to difficulties with additionality|
 For new buildings the effect of standards as policy measure is quite reliable given a well-defined baseline
For more information on verification of actual energy savings and attribution/baseline/corrections, see section 4, and for cost-effectiveness and emission reduction see section 7.
Further evaluation results can be included like CO2-emissions savings or macroeconomic effects.
3.2 Reporting expectations
- Gross and Net savings, these are the savings that are calculated by the method of engineering estimates (gross) or the adjusted savings directly attributed to the policy measure (net). See further in section 4 of this guide under the heading of Calculating Gross and Net savings.
- Yearly or cumulative savings, these energy savings can be ranked as cumulative over a specific period or on a yearly basis.
3.3 Time frame for evaluation
The length of the period under evaluation is dependent on the active period of the policy measure, the need to monitor developments before the implementation of savings actions (in case of methods based on before/after saving actions), and the time needed to present (reliable enough) results or impacts that fit into the decision making process. In some cases, the periodicity of evaluation can be set by law. Because legislative action applies to everyone, it is not possible to conduct analyses using treatment and comparison groups. Therefore, it is important for establishing a baseline for additionality (see section “calculation baselines”) before the implementation of the policy.
The planning of evaluation activities concerns regular monitoring of energy consumption and factors that define consumption, intermediate check of (ex-ante) estimated (unitary) savings through measuring or surveys, intermediate evaluations to improve the policy implementation and the final evaluation and reporting. If the engineering estimate method is not combined with physical ex-post measurements for verification, the timeframe for evaluation is relatively low. It can already be performed after the finalizing the planning phase for the energy efficiency action. See also planning of evaluation in the link here.
3.4 Expertise needed for chosen method
Knowledge about behaviour of people, designing interviews and interviewing skills, as well as analytical and statistical knowledge for comparisons is needed for a sound evaluation.
4. KEY METHODOLOGICAL CHOICES FOR CALCULATION OF ENERGY SAVINGS
General principles of calculating realized savings using different methods can be found here and here
This section deals with key methodological choices to be considered when calculating energy savings: consistency between ex-ante and ex-post evaluation, baseline, normalization and adjustment factors. These choices are important to document when reporting energy savings, to ensure the transparency of the results.
The method uses models or simulations to calculate expected energy savings from a savings action. Depending on the choice of model, it is capable to do so either for unitary savings, for example for one buildings, or on more complex units, either smaller, e.g. one floor of a building, or larger, e.g. building block. The modelling approach is more sophisticated than the unitary savings approach; however results can be more precise. The methodological choices laid out in the following sections can either be part of model calibration or employed for later adjustment of results.
4.1 Matching method with earlier ex-ante evaluation
From the viewpoint of methodological consistency and data availability using the same method in the ex-ante evaluation and in this tool on ex-post evaluation might be an obvious choice.
The engineering estimates methods for savings calculation is indeed applicable for ex-ante evaluation and ex-post evaluation. For operational reasons, the unit of analysis in the ex-ante evaluation will then likely be on a larger scale with lower granularity (i.e. heating system instead of each part of the heating system separately) data and results.
The engineering estimates method ex-post can be combined with other methods for ex-ante evaluation, such as deemed savings or stock modeling, depending on the evaluation objectives, timeframe and data available for the situation after implementing the actions.
Here a combination is particularly possible with stock modelling. Therefore an analysis of a sample can be scaled up based on the given non-residential building stock. See chapter 7, calculation approaches in this document.
In practice, ex-ante and ex-post evaluations are applied consecutively. The ex-post evaluation builds on an ex-ante evaluation that makes use of data coming from previous ex-post evaluation or studies (e.g. about previous periods of the same policy measure, or about the same types of energy saving actions as the ones promoted by the new policy measure). These previous ex-post studies could have used another type of method as well.
4.2 Calculation baselines
Energy savings are defined in general as the difference between the actual situation and a reference situation without the saving actions (and without the policy measures that influence these saving actions). In case of saving actions the reference situation can be defined using various calculation baselines: Before/after, With/without, Trend, Target/control group and Minimum efficiency standards; see further here.
Energy savings resulting from saving actions due to policy measures for normative legislative measures using engineering estimates method in the non-residential buildings sector can be calculated from the following options:
- For new buildings the “before/after” option is not valid, but the “with/without” option can be applied. In case the building should comply with a new, more stringent standard, the option “(previous) standard” is also possible.
- For renovated buildings the option “before/after” is the most obvious choice, also in the case that the renovated building should comply with a standard.
See also here
Depending on the requirements in section 3 on evaluation objectives and requirements, preference should be given to minimum efficiency standards as baseline or if not available to before/after (renovation) or with/without (new). In some legal frameworks, for example Article 7 energy savings reporting in the European Energy Efficiency Directive (EED), the choice of these baselines is obligatory.
See further information in this document.
4.3 Normalization factors
The calculation with minimum efficiency standards of comparison with trends provides a change in energy consumption that should be corrected for influences on energy consumption other than the saving actions. These so-called normalization factors can be weather (with effect on consumption), and changes in energy using activities, such as production (industry), occupation rate (buildings) or car usage (transport).
Normalization factors can also comprise Performance gap and the Pre-bound effect (see [link to doc on normalization and adjustment factors, gross and net savings, etc. here.
For the specific guidance at hand the normalization factors can concern weather, rebound effect and occupation rate. See table 1 in this link and here.
Normalization factor Weather concerns a correction of energy consumption data for differences in outdoor temperature during the heating season or cooling season. The baseline calculations are executed with energy consumption data that have been corrected for yearly deviations from the long-term mean outdoor temperature (e.g. actual versus mean heating/cooling degree days).
Energy consumption can furthermore be corrected for occupation rate of buildings (with effect on heating/lighting), when this rate is different for the baseline situation and the situation after the saving actions.
See also here.
4.4 Adjustment factors
Adjustment factors define which part of the calculated energy savings can be attributed to a policy measure or meets the definition of savings specified in the evaluation objectives or reporting requirements (see next section on “Calculating Gross and net savings).
Adjustment factors can concern the Free rider effect, the Spill-over/multiplier effect, Additionality and Non-compliance, see here.
Additionality and non-compliance are connected to unitary savings, while free riders and spill-over work through the number of actions (see next section on “Calculating Gross and Net savings).
Double counting might be relevant in case of another policy focusing on the same saving actions as evaluated with the specific guidance tool at hand. Double counting can only be accounted for at a higher level than individual specific guidance tools; see Distinction of energy efficiency improvement measures by type of appropriate evaluation method, here and here.
For legislative actions, the adjustment factors free rider and spill-over effect are not relevant because energy efficiency action is obligatory. However, when a legislative action is accompanied by financial support programmes, free rider and spill-over effects have to be calculated.
For correction of the gross unitary savings, the mentioned additionality and non-compliance factors can be applied ; see table 1 in this link.
Alternatively, for correction of the gross number of saving actions, free rider and spill-over effects are applied as the adjustment factors.
Further information can be found in chapter 8 of this document.
For free riders a distinction must be made between saving actions due to the policy measure and actions, which would have been taken anyway. The engineering estimates method does not provide directly this information, thus other ways must be found. This can be a survey among participants to the policy measure about their motivation, or application of Randomized Controlled Trials (RCT) or Quasi-experimental designs (see further in this topical case study. However, RCTs and Quasi-experimental designs require the existence of a treatment and comparison group. As mentioned above, this is not possible for normative legislative measures as long as they are not combined with other measures like financial support.
In case of top-down methods, adjustment factors concern autonomous savings (or technological progress) and price-induced energy efficiency progress.
4.5 Calculating Gross and Net energy savings
Gross savings concern the calculated savings from saving actions using a chosen baseline and normalization factors. Net savings concern the savings attributed to policy measures or to a stakeholder (e.g. an energy company with an obligation to realise savings at their customers).
When calculating the savings a distinction must be made for most specific guides to the unitary savings and number of actions; see here.
The gross unitary savings can be calculated using the baseline(s) minimum standards or, if not possible, before/after (renovation) or with/without (new building). A correction for relevant normalization factors (weather and building occupancy) is then applied.
The gross number of actions can be determined with the complementary method stock modelling. (see chapter 2).
Total gross savings are equal to gross unitary savings times gross number of actions.
Net unitary savings can be determined from gross unitary savings applying the relevant adjustment (see section on adjustment factors).
Alternatively, net number of actions are determined from the gross number applying the relevant adjustment factors (see section on adjustment factors).
Net total savings are equal to net unitary savings times number of actions or unitary savings times net number of actions; see here and here.
Gross savings can be corrected for non-compliance due to inappropriate implementation of saving actions. To this end data should be available of sample-wise checks on the implementation.
The savings should be corrected for the Double counting effect, i.e. the overlap between the savings due to normative legislative action and savings due to other policy measures. The overlap in the calculated savings of both policy measures cannot be processed at the level of an advanced guide, but must be corrected at the level of savings due to overall policy portfolios. For addressing double counting see the previous two links.
Finally, see section 8 on concrete examples
5. INPUT AND OUTPUT
5.1 Main data requirements, sources and collection techniques
Data requirements specified in the table below correspond to the calculation of energy savings, when using the baseline option minimum standards, before/after or with/without is considered using the engineering estimate method.
|Calculation subject||Data requirements||Possible data sources and collection technics|
|Energy consumption||Energy consumption measurements
Delineation of energy consumption of equipment affected by the savings action.
|Energy consultants, smart meters, utilities, surveyed consumption data with plausibility check.|
|Normalization factors affecting energy consumption||Temperature data, occupancy data||Free and commercial databases for weather
Surveyed occupancy data
|Adjustment factors for 1. unitary savings or 2. number of actions||1. Additionality or non-compliance optional, 2. not necessary (if not combined with subsidies)||1.Non-compliance from inspection and additionality from surveys|
|Primary energy factors applied (for conversion from final to primary savings)||Energy source saved
|National or international publications like EED Guidance notes.|
|Number of actions||Number of actions||Data from licenses for new or renovated buildings|
Data issues when evaluating net energy savings
The main good practice to ensure the feasibility and reliability of the evaluation of net energy savings is to think about the method to be used when designing (or revising) the normative legislative measure.
Experience indeed shows that unless the data collection has been planned in advance, it will be very costly, time-consuming or even impossible to collect the data required to apply most of the methods that can be used to evaluate net energy savings. Which makes that in practice, using surveys will remain the only option possible (or considered feasible).
The main challenges when using surveys are:
- to achieve a high answer rate, in order to limit sampling bias;
- to use question phrasing that can limit the risk of bias in the answers
For more details about the evaluation of net energy savings, see the topical case study.
For possible other methods with different data demands, see the section on alternatives for the chosen method
5.2 Energy savings in final terms or in primary terms
Energy savings can be expressed in final terms or in primary terms. See definitions about primary and final energy here.
Final energy savings are based on a reduction of energy consumption at the end-user premises and savings for all energy carriers are added up. For primary energy savings account is taken of the conversion losses when providing the energy carriers to end-users. E.g. for electricity 2-3 times the amount delivered to the end-user is used as input in power production. Therefore, saving one unit of electricity saves 2-3 unit of fuel in power production. The energy savings in primary energy terms provides savings that represent the reduction in primary energy consumption (before conversion in energy carriers for end-users), and before possible transport and distribution of energy that can generate further energy losses.
Engineering estimates can calculate savings in final terms. It can also calculate savings in primary terms provided that savings at end-users are calculated for each energy carrier apart, and primary factors are available to convert the savings in final terms to savings in primary terms.
5.3 Energy savings over time
Implemented saving actions in a year lead to savings over a number of consecutive years (e.g. a more efficient boiler can save gas over its lifetime of about 15 years, insulation over up to 30 years and more efficient computers up to 5 years). Energy savings can be calculated in different metrics in terms of time reference, for example: year-to-year, annual, cumulated annual, cumulative. See the definitions here.
The calculated yearly savings concern the savings of all new saving actions in that year. In this approach only data for the savings in the chosen year are needed.
Adding up the yearly savings over a period, provided that earlier saving actions are still delivering savings, leads to cumulative savings. For the cumulative savings data are needed for the whole period.
Another cumulative approach, to be applied for the EE directive, is to multiply the (new) savings in a year with the number of years up to a target year and add this result with that for all other years up to the target year. This cumulative approach stimulates early saving actions, as these count more times to the target than later actions.
Finally, savings from a saving action can be discounted and summed up over the lifetime of the action See link here.
Engineering estimates can provide yearly savings of new saving actions in that year. It can also provide cumulative savings, provided that data are available over a period.
Cumulative savings according to the Energy Efficiency Directive can be provided when the lifetimes of savings actions are known. Finally, the tool can provide discounted cumulative savings when discount factors (e.g. economic discounting or technical performance discounting) have been defined for yearly savings over time.
6. ALTERNATIVE FOR CHOSEN METHOD
6.1 Alternatives for the chosen method
Often other savings calculation methods can be applied as well, although they will all have pros and cons regarding various aspects dealt with in preceding sections.
The table below presents the pros and cons of the method for evaluating normative legislative measures in the non-residential buildings sector, and for commonly used alternative methods for the same combination of policy measure and sector.
|Type of method||Pros||Cons|
|Engineering Estimates (Method at hand)||Detailed estimate
Relatively simple and cost-effective
Calculation already after planning phase
|Real savings can deviate from estimate
Participants may deviate from the planned action (not relevant as far as legislative measure is not used in combination with other measures)
|Measurement||Most exact savings calculation||Expensive and time-consuming
Analysis only after physical implementation of savings action
|Unitary energy savings||Simplest method for rather detailed estimate
Very simple and cost-effective
Calculation already after planning phase
|Rather imprecise due to performance differences in a system
May neglect systemic influences on savings
Participants may deviate from the planned action (see above)
7. ADDITIONAL EVALUATION RESULTS
7.1 Calculating avoided CO2 emissions
Avoided CO2 emissions can be evaluated from the energy savings by applying emission factors. Four key aspects are to be taken into account when choosing the emission factor(s):
- Emission factors vary according to the energy source, so the data about energy savings need to be available per energy type.
- Emission factors for a given type of energy can vary over time (especially for electricity).
- Emission factors can take into account:
- Direct emission factors: that take into account the emissions generated when producing the energy used;
- Lifecycle emission factors: which take into account all the emissions generated from the extraction of the energy resources up to the dismantling of the energy production plant (i.e. power plant, refinery…).
Due to the differences that the choice of emission factor(s) can induce, it is important to document what emission factor(s) has(have) been used.
The reduction in CO2-emissions can only be calculated when savings are calculated per relevant energy carrier and a specific emission factor is available for each energy carrier.
The avoided emission of other greenhouse gases due to energy savings are not taken into account here, as these emissions (and more specifically their reductions) are generally negligible compared to CO2 (apart from policy measures targeting the agriculture sector).
IPCC (Intergovernmental Panel on Climate Change) provides a detailed database of peer-reviewed emission factors.
7.2 Calculating other co-benefits
Possible co-benefits from saving energy concern:
- Additional employment
- Reduction of energy poverty
- Other emission reductions (NOx, SO2, fine particles, etc.)
- Better indoor climate
- Reduced dependency on energy import
The following co-benefits are regarded as most relevant and/or feasible to calculate in conjunction to this tool (see table)
The calculation of the co-benefits is explained in the references.
|Type of co-benefits||Why it can be relevant (and for whom)||References where more details can be found|
|Employment||Investment in energy efficiency, especially in the building sector, generates local employment||https://combi-project.eu/|
7.3 Other aspects of importance
For the following aspect an evaluation can be executed (see table).
|Type of aspects||Why it is evaluated||References where more details can be found|
|Indoor air quality||Air quality can decrease due to a poor ventilation system||Evaluation of the tightening of the energy performance coefficient of new dwellings see here|
8. CONCRETE EXAMPLES
A concrete example for normative legislative measures in the non-residential buildings sector could not be found. However, many parallels are present with residential buildings. One comprehensive study has been performed by Joanne Wade and Nick Eyre (2015) for the UK Energy Research Center. “Energy Efficiency Evaluation: The evidence for real energy savings from energy efficiency programmes in the household sector”. Section 4.1 treats minimum efficiency standards for buildings.
9. FURTHER READING
General guidance on evaluations
- Baumgartner, R. (2017). Chapter 12: Survey Design and Implementation for Estimating Gross Savings Cross-Cutting Protocol. The Uniform Methods Project: Methods for Determining Energy-Efficiency Savings for Specific Measures. Prepared for NREL (National Renewable Energy Laboratory), September 2017. http://www.nrel.gov/docs/fy17osti/68568.pdf
- Hoffman, I., Schiller, S., Todd, A., Billingsley, M., Goldman, C., Schwartz, L., 2015. Energy Savings Lifetimes and Persistence: Practices, Issues and Data. Technical Brief, Lawrence Berkeley National Laboratory, May 2015. https://emp.lbl.gov/publications/energy-savings-lifetimes-and
- Eichhammer et al., 2008. Distinction of energy efficiency improvement measures by type of appropriate evaluation method. Final Report on Evaluation and Monitoring for the EU Directive on Energy End-Use Efficiency and Energy Services. EMEEES Project report.
Relevant case studies
- EPATEE Case Study “[Nordic Countries (Iceland, Finland, Norway, Sweden, Denmark) Nordsyn (Market Surveillance of Ecodesign and Energy Labelling) and the Effect Project