02 – Evaluating Minimum standards on efficiency in Residential building using Deemed savings

The primary audience for this guide concerns energy efficiency programme designers, implementers or supervisors, and evaluators looking for guidance on the evaluation process of energy savings in the scope of this tool.

The objective of this guide is to provide:

• Information enabling users to decide whether this tool is suited to his/her needs, and whether complementary or additional method(s) are needed or useful (section 2);
• Guidance about specifying the evaluation objectives and requirements (section 3);
• Guidance about key methodological choices to calculate energy savings (section 4);
• Guidance about the inputs (data requirements) and outputs of the method (energy savings metrics) (section 5);
• Possible alternative methods (with pros and cons) (section 6)
• Background about evaluation results other than energy savings (section 7);
• Relevant examples, case studies and/or good practices (section 8);
• Relevant references for further reading (section 9).

The guide is intended for assessing realised (ex-post) energy savings. However, account is taken of earlier (ex-ante) evaluations of expected savings, if available (see section 4).

The focus of the tool is on impact evaluation, i.e. determining the energy savings, but not on how this has been reached through a step by step process with intermediate results (process evaluation).

Readers looking for the basic and general principles of energy efficiency evaluation may find the following link useful.

2.1 About minimum efficiency standards

For minimum efficiency standards (sometimes referred to as Minimum Energy Performance Standards), a distinction can be made between standards for equipment (e.g. the heat resistance of double glazing, minimum conversion efficiency standards) and system-wide standards (e.g. maximum gas consumption for new dwellings).

More information and examples on the different subtypes residing under the main type (legislative / normative measures) can be found here and here.

The policy measure type described in this guide is restricted to system-wide standards. These standards are usually based on energy performance coefficients butt they do not guarantee that the actual energy consumption of an individual dwelling will be in line with the standard. This may be due to the influence of the behaviour of the occupants and the occupancy rate of dwellings. To assess the effective energy consumption of a dwelling, an evaluation, e.g. by measuring or billing analysis, is necessary.

2.2 Evaluation for a combination of policy measure types

When a minimum efficiency standards is combined with other types of policy measures , it is assumed that the overall savings are mainly resulting from the policy measure at hand. However, the evaluation concerns the combined savings effect of both policy measures (this guide does not allow attributing part of the (overall) calculated savings to different policy measures (see also Double counting in the section on Gross to Net savings).

2.3 About residential buildings

Information on (sub)sectors defined in the Toolbox can be found here, chapter 2, p.17

Other, equivalent descriptions of the sector are “dwellings”, “houses” and “apartments”.

In addition, it is important to take into account rebound, spill-over and free-riders effects when focusing on the residential sector. See sub-section on “Normalization factors” in section 4.

2.4 Evaluation for cross-sector saving actions

No cross-sector saving actions are applicable as the actions concern only residential dwellings.

2.5 About deemed savings

Deemed savings are based on a set of pre-determined savings values for efficiency measures that are calculated on the basis of data and assumptions from various sources available beforehand (e.g. previous studies, manufacturers data, national statistics, expert estimates, etc.).

Information about the various evaluation methods can be found here, table 1 and 2.

This source also covers the combination of the method at hand with other methods, which will be dealt with below.

2.6 Additional methods to increase reliability of the results

As deemed savings are based on data which reliability may be low, direct measurements on a sample group can be an additional method to check the evaluation of deemed savings.

Engineering estimations (see also Advanced Guidance # 16, 19 and 30) can also be applied for new dwellings to assess deemed savings where there is no “reference” situation.

Billing analysis before/after can show changes in overall energy consumption after renovation, but due to other influencing factors it is less suitable to check the deemed savings. The method is not applicable to new dwellings because no billing is possible before. For possible combinations with an additional method, see chapter 6 here.

3.1 Meeting evaluation goals and ambition

The table shows whether this tool can be used to report on general evaluation goals or criteria. See also this document.

General types of evaluation goals or criteria	Level of ambition	Remarks
Calculation of achieved energy savings from saving actions	Good	Depending on quality of deemed savings
Calculation of energy savings attributed to a policy measure(s)	Fair	Reliability lower due to rebound effect
Cost-effectiveness of saving action (for end-users)	Good	Investments known and savings estimated
Cost-effectiveness of policy (government spending)	Excellent	Large savings without subsidies, only effort on (update) deemed savings values regularly
GHG- and pollutants emission reduction from saving actions	Good	See section 7
GHG- and pollutants emission reduction attributed to the policy measure(s)	Fair	See energy savings

For more information on verification of actual energy savings and attribution/baseline/corrections, see section 4, and for cost-effectiveness and emission reduction see section 7.

3.2 Reporting expectations

• Gross and Net savings, these are the savings that are calculated by the method of engineering estimates (gross) or the adjusted savings directly attributed to the policy measure (net). See further in section 4 of this guide under the heading of Calculating Gross and Net savings.
• Yearly or cumulative savings, these energy savings can be ranked as cumulative over a specific period or on a yearly basis.
• Size of rebound effect.

3.3 Time frame for evaluation

The length of the period under evaluation is dependent on the active period of the policy measure, the need to monitor developments before the implementation of savings actions (in case of methods based on before/after saving actions), and the time needed to present (reliable enough) results or impacts that fit into the decision making process. In some cases, the periodicity of evaluation can be set by law.

Note: it is important to ensure comparable situations during the evaluation period (e.g. take into account changes in the number of persons living in the evaluated dwellings, changes in behaviours).

The planning of evaluation activities concerns regular monitoring of energy consumption and factors that define consumption, intermediate check of (ex-ante) estimated (unitary) savings through measuring or surveys, intermediate evaluations to improve the policy implementation and the final evaluation and reporting. See also planning of evaluation in the link here.

3.4 Expertise needed for chosen method

As to the rebound effect knowledge about behavior of inhabitants, designing interviews and interviewing skills are needed for a proper evaluation.

3.5 Boundaries for the evaluation

The tool concerns both new dwellings as well as renovated dwellings. According to the European EED for renovation of dwellings, a financial threshold exists for applying a minimum efficiency standard.

Boundaries for the evaluation will be linked to the evaluation objectives. The scope could be limited to certain group(s) of dwellings or alternatively cover the total stock of dwellings in a country.

This section deals with key methodological choices to be considered when calculating energy savings: consistency between ex-ante and ex-post evaluation, baseline, normalization and adjustment factors. These choices are important to document when reporting energy savings to ensure the transparency of the results.

General principles of calculating realized savings using different methods can be found here and here

The deemed savings method estimates the energy saved per dwelling due to saving actions to comply with a minimum efficiency standard. If the number of dwellings complying with the standard is known, the total savings can be calculated.

4.1 Matching method with earlier ex-ante evaluation

For ex-ante evaluation only a few methods are usually considered, namely deemed savings, engineering estimate and stock modeling (see table 4 in this link).

From the viewpoint of methodological consistency and data availability, using the same method in the ex-ante evaluation and in ex-post evaluation might be an relevant choice.

The ex-post method deemed savings can be combined with the different methods engineering estimate and stock modelling used for the ex-ante evaluation, depending on the evaluation objectives, timeframe and data available for the situation after implementing the actions. The engineering estimate method is generally not applied for large numbers of dwellings with uniform saving actions. The stock modelling method, including complementary method deemed savings, can be combined the ex-post method deemed savings. For possible combinations of methods applied ex-ante and ex-post, see chapter 7 (different methods) here.

In practice, ex-ante and ex-post evaluations are applied consecutively. The ex-post evaluation builds on an ex-ante evaluation that makes use of data coming from previous ex-post evaluation or studies (e.g. about previous periods of the same policy measure, or about the same types of energy saving actions as the ones promoted by the new policy measure). These previous ex-post studies could have used another type of method as well.

4.2 Calculation baselines

Energy savings are defined in general as the difference between the actual situation and a reference situation without the saving actions (and without the policy measures that influence these saving actions). In case of saving actions, the reference situation can be defined using various calculation baselines: Before/after, With/without, Trend, Target/control group and minimum efficiency standards; see further here.

Total savings are found by multiplying the unitary savings per action by the number of actions. The chosen baseline can define the total savings through the unitary savings, but also through the number of actions.

For new dwellings, the applicable baseline is “With/without” (saving actions to comply with minimum efficiency standards). For renovated dwellings also the baseline “Before/after” can be used. See table 1 in this document.

The definition of a baseline from “Target/control” group is not possible for new dwellings because they are theoretically supposed to comply with minimum efficiency standards.

4.3 Normalization factors

The calculation from a baseline situation assessed with and without the measure must take into account influences of other actions (excluding the studied saving actions). These so-called normalization factors can be performance gap, climate/weather (with effect on consumption), the rebound effect and changes in energy using activities such as occupation rate. All normalization factors affect total savings through the unitary savings.

For new dwellings, the normalization factors can concern only the performance gap, for renovated dwellings also the rebound effect (See table 1 in this link and here).

The performance gap concerns differences in operating conditions or defects from installation at the cost of the expected (deemed) savings due to the standard.

For renovated dwellings, the rebound effect could result from the extra energy consumption of higher thermostat setting after saving actions on dwellings that lower the energy bill. In addition, the rebound effect can be observed in new dwellings when the occupants deliberately consume more energy than they used to consume in their previous apartment because they know they make savings to the better insulation of their new flat.

Changes in activity (e.g. occupation rate of the dwelling) are not assumed for saving actions to comply with the standard. The normalization factor Weather is relevant when applying the deemed savings method.

4.4 Adjustment factors

Adjustment factors allow taking into account effects not directly caused by the studied policy measure or meeting the definition of savings specified in the evaluation objectives or reporting requirements (see next section on “Calculating Gross and net savings).

Adjustment factors can concern the Free rider effect, the Spill-over/multiplier effect, Additionality and Non-compliance, see here.

Additionality and non-compliance are related to unitary savings, while free riders and spill-over are related to the number of actions impacted by the policy measure.

Double counting might be relevant in case of another policy focusing on the same saving actions as evaluated with the specific guidance tool at hand. Double counting can only be accounted for at a higher level than individual specific guidance tools; see Distinction of energy efficiency improvement measures by type of appropriate evaluation method, here and here.

Non-compliancy with minimum efficiency standards means that not all saving actions are implemented, or that the actions are not implemented in a way allowing to achieve the expected savings. This is especially an issue for renovated dwellings where starting situations may be very different. Sample wise check of energy consumption of new and renovated is needed to determine non-compliancy.

Free riders are not an issue because minimum efficiency standards are obligatory where free riders cannot profit from a policy but would have realised savings actions anyway.

4.5 Calculating Gross and Net energy savings

Gross savings concern the calculated savings from saving actions using a chosen baseline and normalization factors. Net savings concern the savings attributed to policy measures or to a stakeholder (e.g. an energy company with an obligation to realise savings at their customers).

When calculating the savings a distinction must be made for most specific guides to the unitary savings and number of actions; see here.

The gross unitary savings can be calculated using the With/without baseline (or also the Before/after baseline for renovated dwellings with a minimum efficiency standard). The unitary savings per dwelling assessed from the deemed savings, must be then adjusted according to the performance gap (difference between the estimated or expected energy performance that is calculated based on the historical data of the same building and the actual performance) and by taking into account the rebound effect for renovated dwellings. The number of actions is the number of new dwellings and renovated dwellings (complying with the standard).

Net unitary savings can be determined from gross unitary savings by applying the adjustment factor related to non-compliancy (see section on adjustment factors). The net number of actions is equal to the gross number (because there is no adjustment factor free rider).

In case applicable, the energy savings should be corrected for the double counting effect, i.e. the overlap between the savings due to the financial incentive measure and savings due to other policy measures. The overlap in the calculated savings of both policy measures cannot be processed at the level of this specific guide but must be corrected at the level of savings due to overall policy portfolios. For more explanations on double counting see links here and here.

5.1 Main data requirements, sources and collection techniques

Data requirements specified in the table below correspond to the calculation of energy savings, when using the baseline option With/Without or Control Group.

Calculation subject	Data requirements	Possible data sources and methods
Energy consumption (in case of additional method billing analysis)	Energy consumption of (part of) new/renovated dwellings according to standards	Energy bills “after” and, for renovated dwellings bills “before”
Normalization factors	Performance gap, rebound effect (only for renovated dwellings according to standard)	Performance gap: checking technical implementation Rebound effects: interview/survey among the users of studied dwellings
Adjustment factors	Climate/weather Non-compliancy	Use climate correction factor to remove the part of the energy consumption due to climate variations Checking for correct energy standard implementation
Number of actions	Number of (new or renovated) dwellings according to the energy standard	Building permits or building completion statistics
Primary energy factors applied (from final to primary savings)	Fuel input and electricity output of power production	Statistics on electricity conversion Datasheet from equipment manufacturers to know the energy efficiency of installed equipment for energy generation and energy consumption (heating & cooling systems…)

Data issues when evaluating net energy savings

The main good practice to ensure the feasibility and reliability of the evaluation of net energy savings is to think about the method to be used when designing (or revising) the minimum efficiency standards.

In this tool with the deemed savings method the deemed savings should be based on earlier ex-ante values that were checked ex-post with actual energy consumption data. For net savings data should be gathered to determine the performance gap and non-compliance. The data collection through surveys must be planned in advance. For more details about the evaluation of net energy savings, see the topical case study.

Data issues with the additional method

For the additional method billing analysis energy consumption data have to be gathered “after” (and for renovated dwellings also “before”).

For possible other methods with different data demands see the section on alternatives for the chosen method.

5.2 Energy savings in final terms or in primary terms

Energy savings can be expressed in final terms or in primary terms. See definitions about primary and final energy here.

The deemed savings method can calculate savings in final terms. It can also calculate savings in primary terms provided that savings at end-users are calculated for each energy carrier apart, and primary factors are available to convert the savings in final terms to savings in primary terms.

5.3 Energy savings over time

Implemented saving actions in a year lead to savings over a number of consecutive years (e.g. a more efficient boiler can save gas over its lifetime of about 15 years, insulation over up to 30 years and more efficient computers up to 5 years). Energy savings can be calculated in different metrics in terms of time reference, for example: year-to-year, annual, cumulated annual, cumulative. See the definitions here.

For the cumulative savings, data are needed for the whole considered period.

Another cumulative approach consists on multiplying the (new) savings for a year with the number of years up to a target year. This cumulative approach may cause biases due to the progressive implementation of the measure.

Finally, savings from a saving action can be discounted and summed up over the lifetime of the action See link here.

The deemed savings method can provide yearly savings.

6.1 Alternatives for the chosen method

The table below presents the pros and cons of the methods usually used for evaluating minimum efficiency standards in residential buildings, and for commonly used alternative methods for the same combination of policy measure and sector.

Type of method	Pros	Cons
Method at hand (deemed savings)	Relatively easy, no real energy consumption data needed	Less precise
Engineering estimates	More accurate, both ex-ante and ex-post	Expensive, needs experts (e.g. thermal engineering)
Measurement in combination with surveys	Very precise, only energy use related to dwelling	Takes time, expensive
Billing analysis in combination with surveys	Based on actual energy data	Not applicable for new dwellings, only for renovated dwellings with standard Expensive and time consuming Required expertise in big data analysis and statistics

7.1 Calculating avoided GHG and pollutants emissions

Avoided CO2 emissions can be evaluated from the energy savings by applying emission factors. Four key aspects are to be taken into account when choosing the emission factor(s):

1. Emission factors vary according to the energy type, so the data about energy savings need to be available per energy type.
2. Emission factors for a given type of energy can vary over time (especially for electricity).
3. Emission factors can take into account:
   1. Direct emission factors: that take into account the emissions generated when producing the energy used;
   2. Lifecycle emission factors: which take into account all the emissions generated from the extraction of the energy resources up to the dismantling of the energy plant.

Due to the differences that the choice of emission factor(s) can induce, it is important to document what emission factor(s) has(have) been used.

For savings concerning one fuel only (e.g. for gas boilers, the reduction in CO2-emissions can be calculated from the savings with an emission factor for the fuel / energy carrier at stake. For residential buildings, that would mainly be gas and/or electricity.

For savings concerning electricity only (e.g. appliances), the reduction in CO2-emissions can be calculated from the savings with an emission factor for electricity that takes into account the different inputs of power production. The actual factor to be applied can vary, depending on saving action(s) and sector, year of implementation, policy considerations, etcetera (see example here).

The avoided emission of other greenhouse gases due to energy savings are not taken into account here, as these emissions (and more specifically their reductions) are generally negligible compared to CO2 (apart from policy measures targeting the agriculture sector).

IPCC (Intergovernmental Panel on Climate Change) provides a detailed database of peer-reviewed emission factors.

7.2 Calculating cost-effectiveness

Cost-effectiveness is the ratio between costs to achieve energy savings and the amount of savings and possibly other benefits.

A distinction can be made according to the point of view adopted to assess cost-effectiveness:

• Cost-effectiveness for the end-user or participant
• Cost-effectiveness for society at large
• Cost-effectiveness for the party that takes responsibility for saving targets (government or actor with an Energy Efficiency Obligation)

See link to report on knowledge base

The calculation of cost-effectiveness for end-users demands requires, next to the savings data, data on investments made, subsidies on investments, interest rates, lifetimes of the saving actions, energy prices (including taxes) per type of end-user and discount factors per type of end-user. The calculation is based on the additional costs of a dwelling that conforms to the efficiency standard for the home owner and the related energy savings results.

For cost-effectiveness from a societal viewpoint no account is taken of subsidies and taxes, energy prices concern world market price, and a lower value of the discount factors is valid.

7.3 Calculating other co-benefits

Possible co-benefits from saving energy concern and GHG and pollutants reductions:

• Extra employment
• Reduction of energy poverty
• Comfort increasing
• Health improvement
• Reduced dependency on (insecure) energy import

The following co-benefits are regarded as most relevant and/or feasible to calculate in conjunction to this guide (see table). The methods to use for calculation of co-benefits are detailed at the given web links.

Type of co-benefits	Why it can be relevant (and for whom)	References where more details can be found
Increased comfort	Can increase support for savings measures	Evaluation of the tightening of the energy performance coefficient of new dwellings, See  https://repository.tudelft.nl/view/tno/uuid:8837756f-64b0-48a6-b18c-317b4f2cd012
Reduced energy poverty	Relevant for low-income households

7.4 Other aspects of importance

An evaluation can be executed for the following aspects (see table).

Type of aspects	Why it is evaluated	References where more details can be found
Indoor air quality	Worse if ventilation with heat recovery is not working well Moisture in bad insulated dwelling may cause breath diseases	Evaluation of the tightening of the energy performance coefficient of new dwellings, see https://repository.tudelft.nl/view/tno/uuid:8837756f-64b0-48a6-b18c-317b4f2cd012

A comprehensive study has been performed by Joanne Wade and Nick Eyre (2015) for the UK Energy Research Center, “Energy Efficiency Evaluation: The evidence for real energy savings from energy efficiency programmes in the household sector”. Section 4.1 treats minimum efficiency standards for buildings. The report can be accessed on the following website: www.ukerc.ac.uk/programmes/technology-and-policy-assessment/energy-efficiency-evaluation.html

A study of the impact of main French policies for energy efficiency of dwellings in 2012 and 2050 from prospective modelling of energy demand for heating based on energy performance diagnosis led by Louis-Gaëtan GIRAUDET et al. The study is accessible (in French) at https://www.ademe.fr/evaluation-prospective-politiques-reduction-demande-denergie-chauffage-residentiel

Please also see section 9 for additional suggested reading material covering general guidance on evaluations, ex-post evaluation of a residential energy efficiency policy measure, and a publication covering energy savings calculation methods under Article 7 of the Energy Efficiency Directive, including deemed savings.

General guidance on evaluations

Eichhammer et al., 2008. Distinction of energy efficiency improvement measures by type of appropriate evaluation method. Final Report on Evaluation and Monitoring for the EU Directive on Energy End-Use Efficiency and Energy Services. EMEEES Project report. https://www.epatee-lib.eu/media/docs/EMEEES_WP3_Report_Final.pdf

Vreuls, H. et al., 2005. Evaluating Energy Efficiency Policy measures & DSM programmes. Volume II Country reports and case examples used for the evaluation guide book. Prepared under the IEA Implementing Agreement on Demand-Side Management Technologies and Programmes. http://www.ieadsm.org/wp/files/Exco%20File%20Library/Key%20Publications/Volume%202%20total.pdf

About deemed savings method

Labanca, N. Bertoldi, P., 2016. Energy Savings Calculation Methods under Article 7 of the Energy Efficiency Directive. Prepared for DG Energy by the JRC, January 2016. http://publications.jrc.ec.europa.eu/repository/bitstream/JRC99698/report%20on%20eed%20art%207%20-%20publishable.pdf

About ex-post evaluation of a residential energy efficiency policy measure

Rogan, F., Ó gallachóir, 2011. Ex-post evaluation of a residential energy efficiency policy measure using empirical data. A paper that presents an ex-post evaluation of residential building regulations for new dwellings in Ireland in terms of the energy savings that can be attributed to the policy. https://www.eceee.org/library/conference_proceedings/eceee_Summer_Studies/2011/7-monitoring-and-evaluation160/ex-post-evaluation-of-a-residential-energy-efficiency-policy-measure-using-empirical-data/

Frontier Economics, 2017. The impact of Minimum Energy Efficiency Standards in the private rented sector. A report for Citizens Advice. July 2017. https://www.citizensadvice.org.uk/Global/CitizensAdvice/Energy/Frontier%20Economics%20for%20Citizens%20Advice%20-%20The%20Impact%20of%20Minimum%20Energy%20Efficiency%20Standards%20in%20the%20Private%20Rented%20Sector.pdf