05 – Evaluating legislative/normative measures on energy efficiency of Household appliances using stock modelling

The primary audience for this specific guide concerns energy efficiency programme designers, implementers or supervisors, and evaluators looking for guidance on the evaluation process of energy savings in the scope of this tool.

Although the application of the specific guide will generally concern the (sub)national level, account will be taken of issues at EU level when relevant (e.g. the specific format of saving figures for the EED).

This guide is not about the preceding step in the evaluation process, the choice of the method. About this previous step in the evaluation process, see the guidance provided here. However, after presenting the capabilities and limitations of the tool at hand, the user will be offered alternatives for the method within this tool (see section 6).

The objective of this guide is to provide:

• Information on the scope of the tool that enables the user to decide whether this tool is suited to his/her needs, and whether complementary or additional method(s) could be needed or useful (section 2);
• Guidance about specifying the evaluation objectives and requirements (section 3);
• Guidance about key methodological choices to calculate energy savings (section 4);
• Guidance about the inputs (data requirements) and outputs of the method (energy savings metrics) (section 5);
• Possible alternative methods (with pros and cons) (section 6)
• Background about evaluation results other than energy savings (section 7);
• Relevant examples, case studies and/or good practices (section 8);
• Relevant references for further reading (section 9).

The specific guide is intended for assessing realised (ex-post) energy savings. However, account is taken of earlier (ex-ante) evaluations of expected savings, if available (see section 4).

The focus of the tool is on impact evaluation, i.e. determining the energy savings, but not on how this has been reached through a step by step process with intermediate results (process evaluation).

Readers looking for the basic and general principles of energy efficiency evaluation may find the following link useful.

2.1 About legislative / normative measures

Normative legislative measures in practice come down to efficiency standards. For standards) a distinction can be made between specific standards on for example insulation (e.g. the heat resistance of double glazing) or boilers (minimum conversion efficiency standards) and system-wide energy performance standards (e.g. maximum gas consumption for new dwellings).

The policy measure type described in this guide concerns energy performance standards on appliances. A required maximum energy consumption has to be met to sell the appliance on the market. Requesting an energy performance is no guarantee that the actual energy consumption will be in line with the required maximum consumption . To get to know the actual energy consumption, an evaluation by sample wise measuring is necessary.

More information and examples on the different subtypes residing under the main type (legislative / normative measures) can be found here and here.

More detailed information on the evaluation of legislative / normative measures can be found here.

2.2 Evaluation for a combination of policy measure types

This specific guide addresses legislative / normative measures, but sometimes a combination of policy measure types is implemented, e.g. subsidies for very efficient appliances. With two or more policy measures stimulating the same saving actions it is difficult to ascribe part of the overall savings to each policy measure type.

A practical approach for combined policy measure types is to perform the evaluation for the set of policy measures as such. The combination is characterised by the policy measure which is seen as most important as to realised savings. E.g. in case of maximum consumption standards and subsidies, the last one is seen as decisive for the savings, because the standards only prevent buying the least efficient appliance while the subsidy defines the buying of (much) more efficient appliances. However, this approach is valid when the subsidy is sufficiently high to cover the gap between the higher price for the more efficient appliance and benefits from energy savings.

When legislative / normative measures are combined with other policy measure types it is assumed that the overall savings are mainly resulting from the policy measure at hand. However, the evaluation concerns the combined savings effect of both policy measures.

The tool is not capable of attributing part of the (overall) calculated savings to each of the policy measures (see also double counting in the section on Gross to Net savings).

2.3 Evaluation when combined with energy taxes

The calculated savings effect for legislative / normative measures will overlap with that of the energy tax. The specific guide is not capable of attributing part of the (overall) calculated savings to either the policy measures at hand or the energy tax. For dealing with this overlap, see section on Gross to Net savings.

2.4 About appliances in households

Information on (sub)sectors defined in the Toolbox can be found here, chapter 2, p.17

The subsector considered here only concerns appliances in households, mostly considering energy consumption in the form of electricity, and explicitly excludes buildings.

2.5 About the stock modelling method

Information about the various evaluation methods can be found here table 1 and 2. This source also covers the combination of the method at hand with other methods, which will be dealt with below.

Stock modelling is based on stock and market statistics for specific appliance types, such as washing machines, refrigerators, televisions. It describes the yearly composition of the stock in terms of more or less efficient versions of the appliance. Through the yearly uptake of more efficient new appliances (and removal of old less efficient appliances) the average efficiency can increase, leading to energy savings. The minimum efficiency standard for an appliance type prevents buying the least efficient appliances and thus directs the composition of the stock to a higher average efficiency.

Surveys on diffusion and uptake of more energy-efficient appliances are needed to fill the stock model for past and future years. Complementary surveys among participants are necessary to give information on (changing) usage behaviour of appliances for reliable estimates on actual energy performance.

The focus of the method is not on unitary energy savings, as is the case for most other methods, but on the number of saving actions, i.e. the number of new appliances with a specific efficiency level from the stock model. For the calculation of unitary savings see next section.

2.6 Complementary methods to determine total savings

Complementary methods are methods that are required, in addition to the primary selected method, to calculate energy savings.

The method at hand concerns the determination of the number of actions or participants. In order to provide the total savings, the unitary savings per action or participant should also be calculated. These concern the difference in energy consumption between the replaced appliance and the new appliance. Due to the various old and new efficiency classes and replacement possibilities this concerns a set of differences/savings. This set can be constructed through the deemed savings method where each efficiency class is characterized by a consumption level. For more information about methods to calculate unitary savings, see the link here.

2.7 Additional methods to increase reliability of the results

An additional method can be applied on top of measurement to improve the reliability of the evaluation results and/or the cost-effectiveness of the evaluation approach.

For higher reliability of results, direct measurements can be employed for a subsample of participants. Results can then be compared to the estimates from the stock model with additional survey and adjusted accordingly.

For possible combinations with an additional method see chapter 6 see chapter 6 here.

3.1 Meeting evaluation goals and ambition

The table shows whether this tool can be used to report on general evaluation goals or criteria. See also this publication.

General types of evaluation goals or criteria	Level of ambition	Remarks
Calculation of realized energy savings from saving actions	Fair	Depends on the set of differences in electricity use between old and new appliance
Calculation of energy savings attributed to the policy measure(s)	Low	Autonomous choices for more efficient appliances can limit the policy effect
Cost-effectiveness of saving action (for end-users)	Fair	Stock models can give reliable information on possible energy cost savings for the end-user. Due to different usage behaviour though, breaking it down to the single consumer is difficult.
Cost-effectiveness of policy (government spending)	Fair	There is no government spending at the saving action level for legislative policies, but the policy effect is uncertain (see savings)
CO2-emission reduction from saving actions	Fair	Depends on the way the (avoided) electricity consumption is produced. The savings decrease with an increasing share of renewable electricity.
CO2-emission reduction attributed to the policy measure(s)	Low	Less reliable (see savings)

For more information on verification of actual energy savings and attribution/baseline/corrections, see section 4, and for cost-effectiveness and emission reduction see section 7.

3.2 Reporting expectations

The method will make it possible to report (net) savings of normative legislative measures for household appliances. However, the described method of stock modelling is associated with high uncertainty about usage behaviour.

3.3 Time frame for evaluation

The length of the period under evaluation is dependent on the active period of the policy measure, the need to monitor developments before the implementation of savings actions (in case of methods based on before/after saving actions), and the time needed to present (reliable enough) results or impacts that fit into the decision making process. In some cases, the periodicity of evaluation can be set by law.

In case of stock modeling the life times of appliances define for which historic years the composition of the stock as to more or less efficient appliances must be available. For unitary savings, the set of differences in energy consumption between old and new appliances, there should be data on consumption per efficiency class over the same period.

The planning of evaluation activities concerns regular monitoring of energy consumption and factors that define consumption, intermediate check of (ex-ante) estimated (unitary) savings through measuring or surveys, intermediate evaluations to strengthen the appliance standards and the final evaluation and reporting. Stock models become more reliable with time during the active time of the policy measure. The longer the measure has been running, the more accurate changes in market data can be observed. (see also to planning of evaluation : See also planning of evaluation in the link here.

3.4 Expertise needed for chosen method

Knowledge about factors that define stock changes, as well as analytical and statistical knowledge for comparisons is needed for a sound

General principles of calculating realized savings using different methods can be found here and here

This section deals with key methodological choices to be considered when calculating energy savings: consistency between ex-ante and ex-post evaluation, baseline, normalization and adjustment factors. These choices are important to document when reporting energy savings, to ensure the transparency of the results.

The stock modelling method is applied using market penetration data of new appliances, theoretical energy savings when replacing old appliances by new ones, and behavioral data to estimate real energy savings.

4.1 Matching method with earlier ex-ante evaluation

From the viewpoint of methodological consistency and data availability using the same method in the ex-ante evaluation and in this tool on ex-post evaluation might be an obvious choice. As ex-ante evaluation methods include stock modeling (for number of actions) and deemed savings (for unitary savings) the same approach is possible for both ex-ante and ex-post evaluation.

A different method than the one(s) used for the ex-ante evaluation can be applied for the ex-post evaluation, depending on the evaluation objectives, timeframe and data available for the situation after implementing the actions. For possible combinations of methods applied ex-ante and ex-post, see chapter 7, approaches in this approaches in this document.

If stock modelling does not provide an acceptable combination with the earlier applied ex-ante method it might be useful to select another method (see examples of alternatives in section 6).

In practice, ex-ante and ex-post evaluations are applied consecutively. The ex-ante evaluation builds on an ex-post evaluation that makes use of data coming from previous ex-post evaluation or studies (e.g. about previous periods of the same policy measure, or about the same types of energy saving actions as the ones promoted by the new policy measure). These previous ex-post studies could have used another type of method as well.

4.2 Calculation baselines

Energy savings are defined in general as the difference between the actual situation and a reference situation without the saving actions (and without the policy measures that influence these saving actions). In case of saving actions the reference situation can be defined using various calculation baselines: Before/after, With/without, Trend, Target/control group and Minimum efficiency standards; see further here.

Energy savings resulting from savings actions due to minimum energy efficiency standards using stock modelling can be calculated using the with/without baseline option. The baseline concerns a calculation with the stock model without the standard that prevents buying the least efficient appliances. With the standard these least efficient appliances are replaced by more efficient new appliances, resulting in extra savings.

See also here

4.3 Normalization factors

The calculation with any of the methods listed under “Calculation Baselines” provides a change in energy consumption that should be corrected for influences on energy consumption other than the savings actions. These so-called normalization factors can be weather (with effect on consumption), the rebound effect and changes in energy using activities, such as production (industry), occupation rate (buildings) or car usage (transport).

For the specific guidance at hand the normalization factors can concern weather, rebound effect and occupation rate. See table 1 in this link.

Energy consumption can be corrected for the rebound effect, e.g. the extra energy consumption of appliances due to more intensive use after saving actions that lower the energy bill.

See also here and here.

4.4 Adjustment factors

Adjustment factors define which part of the calculated energy savings can be attributed to a policy measure or meets the definition of savings specified in the evaluation objectives or reporting requirements (see next section on “Calculating Gross and net savings).

Adjustment factors can concern the Free rider effect, the Spill-over/multiplier effect, Additionality, Non-compliance and Double counting, etc., see here.

Additionality and non-compliance are connected to unitary savings, while free riders and spill-over work through the number of actions (see next section on “Calculating Gross and net savings).

Double counting might be relevant in case of another policy focusing on the same saving actions as evaluated with the specific guidance tool at hand. Double counting can only be accounted for at a higher level than individual specific guidance tools (see Distinction of energy efficiency improvement measures by type of appropriate evaluation method, here and here.

For legislative actions, the adjustment factors free rider and spill-over effect are not relevant because energy efficiency action is obligatory. Only when a standard is accompanied by financial support programmes free rider and spill-over effects have to be calculated.

For correction of the gross unitary savings, the mentioned additionality and non-compliance factors can be applied; see table 1 in this link.

Further information can be found in chapter 8 of this document.

4.5 Calculating Gross and Net energy savings

Gross savings concern the calculated savings from saving actions using a chosen baseline and normalization factors. Net savings concern the savings attributed to policy measures or to a stakeholder (e.g. an energy company with an obligation to realise savings at their customers).

When calculating the savings a distinction must be made for most specific guides to the unitary savings and number of actions; see here.

The gross total savings can be calculated using the baseline(s) with/without and correcting for relevant normalization factors as mentioned in the section “normalization factors”.

Net total savings can be determined from gross savings applying the relevant adjustment (see section on adjustment factors).

See here and here.

The savings should be corrected for the Double counting effect, i.e. the overlap between the savings due to normative legislative measures and savings due to other policy measures. The overlap in the calculated savings of both policy measures cannot be processed at the level of an PSMC but must be corrected at the level of savings due to overall policy portfolios. For addressing double counting see the previous two links.

Finally, see also the section on Concrete examples.

5.1 Main data requirements, sources and collection techniques

Data requirements specified in the table below correspond to the calculation of energy savings, when using the baseline option with/without.

Calculation subject	Data requirements	Possible data sources and collection technics
Energy consumption (yearly)	Stock composition for appliance type per energy efficiency class Yearly sales data for an appliance type per energy efficiency class Average lifetime for appliance type Usage data	Technology producers or resellers Statistics offices Consumption panels Surveys for usage data
Normalization factors affecting energy consumption	Rebound effect Information on preferences in energy usage	Literature research about rebound effects for specific technologies Surveyed data on energy use
Adjustment factors	Non-compliance	Check on actual versus assumed energy consumption for efficiency classes
Primary energy factors applied (for conversion from final to primary savings)	Conversion efficiency in power production, type of power generation and Energy source saved (Employed technology)	National or international statistics on power production and fuel use.

Data issues when evaluating net energy savings

The main good practice to ensure the feasibility and reliability of the evaluation of net energy savings is to think about the method to be used when designing (or revising) the normative legislative measure.

For determining non-compliance a check is needed on the actual energy consumption of appliances per efficiency class.

For more details about the evaluation of net energy savings, see the topical case study .

For possible other methods with different data demands, see the section on alternatives for the chosen method.

5.2 Energy savings in final terms or in primary terms

Energy savings can be expressed in final terms or in primary terms. See definitions about primary and final energy here.

Final energy savings are based on a reduction of energy consumption at the end-user premises and savings for all energy carriers are added up. For primary energy savings account is taken of the conversion losses when providing the energy carriers to end-users. E.g. for electricity 2-3 times the amount delivered to the end-user is used as input in power production. Therefore, saving one unit of electricity saves 2-3 unit of fuel in power production. The energy savings in primary energy terms provides savings that represent the reduction in primary energy consumption (before conversion in energy carriers for end-users), and before possible transport and distribution of energy that can generate further energy losses.

Stock modelling can calculate savings in final terms. It can also calculate savings in primary terms provided that primary factors are available to convert the savings in final terms to savings in primary terms.

5.3 Energy savings over time

Implemented saving actions in a year lead to savings over a number of consecutive years. E.g. a more efficient boiler can save gas over its lifetime of about 15 years, insulation over up to 60 years and more efficient computers up to 5 years. Energy savings can be calculated in different metrics in terms of time reference, for example: year-to-year, annual, cumulated annual, cumulative. See the See the definitions here.

The calculated yearly savings concern the savings of all new saving actions in that year. In this approach only data for the savings in the chosen year are needed.

Adding up the yearly savings over a period, provided that earlier saving actions are still delivering savings, leads to cumulative savings. For the cumulative savings data are needed for the whole period.

Another cumulative approach, to be applied for the EE directive, is to multiply the (new) savings in a year with the number of years up to a target year and sum this result with that for all other years up to the target year. This cumulative approach stimulates early saving actions, as these counts more times to the target than later actions.

Finally, savings from a saving action can be discounted and summed up over the lifetime of the action See link here.

Stock modelling can provide yearly savings of new saving actions in that year. It can also provide cumulative savings provided that data are available over a period.

Cumulative savings according to the Energy Efficiency Directive can be provided when the lifetimes of savings actions are known. Finally, the specific guide can provide discounted cumulative savings when discount factors (e.g. economic discounting or technical performance discounting) have been defined for yearly savings over time.

6.1 Alternatives for the chosen method

Often other savings calculation methods can be applied as well, although they will all have pros and cons regarding various aspects dealt with in preceding sections.

The table below presents the pros and cons of the method for evaluating normative legislative measures in appliances in households, and for commonly used alternative methods for the same combination of policy measure and sector.

Type of method	Pros	Cons
Stock Modelling	Very simple and cost-effective (if stock/sales data available) Calculation at any time during activity of policy measure	Imprecise May neglect systemic influences on savings Participants may deviate from the planned action (see above)
Direct Measurement (for samples)	Most exact savings calculation	Expensive and time-consuming Difficult Analysis only after physical implementation of savings action
Deemed Savings	Relatively simple and cost-effective Calculation already after planning phase	Real savings can deviate from deemed savings. Participants may deviate from the planned action in actual implementation leading to savings different than the ones calculated using deemed savings (not relevant as far as legislative measure is not used in combination with other measures)

7.1 Calculating avoided CO2 emissions

Avoided CO2 emissions can be evaluated from the energy savings by applying emission factors. Four key aspects are to be taken into account when choosing the emission factor(s):

1. Emission factors vary according to the energy type, so the data about energy savings need to be available per energy type.
2. Emission factors for a given type of energy can vary over time (especially for electricity).
3. Emission factors can take into account:
   1. Direct emission factors: that take into account the emissions generated when producing the energy used;
   2. Lifecycle emission factors: which take into account all the emissions generated from the extraction of the energy resources up to the dismantling of the energy plant.

Due to the differences that the choice of emission factor(s) can induce, it is important to document what emission factor(s) has(have) been used.

The reduction in CO2-emissions can only be calculated when savings are calculated per relevant energy carrier and a specific emission factor is available for each energy carrier.

This can be a challenge for energy net importers where different amounts of electricity are imported from different countries, and emission factor of the electricity in the grid vary significantly over time.

The avoided emission of other greenhouse gases due to energy savings are not taken into account here, as these emissions (and more specifically their reductions) are generally negligible compared to CO2 (apart from policy measures targeting the agriculture sector).

IPCC (Intergovernmental Panel on Climate Change) provides a detailed database of peer-reviewed emission factors.

7.2 Calculating other co-benefits

Possible co-benefits from saving energy concern:

• Extra employment
• Reduction of energy poverty
• Other emission reductions (NOx, SO2, fine particles, etc.)
• Better indoor climate
• Reduced dependency on (insecure) energy import

However, for minimum efficiency standard appliances none of co-benefits are relevant.

7.3 Other aspects of importance

For the following aspect an evaluation can be executed (see table).

Type of aspects	Why it is evaluated	References where more details can be found
Indoor air quality	It can worsen if the  appliances emit too much (e.g. heat, CO2, nitrogen oxides, dust)	Evaluation of emissions of appliances http://www.ukerc.ac.uk/publications/energy-efficiency-report.html

One comprehensive study that among others treats household appliances has been performed by Joanne Wade and Nick Eyre (2015) for the UK Energy Research Center. “Energy Efficiency Evaluation: The evidence for real energy savings from energy efficiency programmes in the household sector”. Section 4.3 appliance market transformation activities.

General guidance on evaluations

Baumgartner, R. (2017). Chapter 12: Survey Design and Implementation for Estimating Gross Savings Cross-Cutting Protocol. The Uniform Methods Project: Methods for Determining Energy-Efficiency Savings for Specific Measures. Prepared for NREL (National Renewable Energy Laboratory), September 2017. http://www.nrel.gov/docs/fy17osti/68568.pdf

Hoffman, I., Schiller, S., Todd, A., Billingsley, M., Goldman, C., Schwartz, L., 2015. Energy Savings Lifetimes and Persistence: Practices, Issues and Data. Technical Brief, Lawrence Berkeley National Laboratory, May 2015. https://emp.lbl.gov/publications/energy-savings-lifetimes-and

Eichhammer et al., 2008. Distinction of energy efficiency improvement measures by type of appropriate evaluation method. Final Report on Evaluation and Monitoring for the EU Directive on Energy End-Use Efficiency and Energy Services. EMEEES Project report. https://ec.europa.eu/energy/intelligent/projects/sites/iee-projects/files/projects/documents/emeees_project_report_en.pdf

Relevant studies on stock modelling

Elektrische apparatuur in Nederlandse huishoudens, Hoofdrapport (definitief), Van Holsteijn en Kemna BV, Delft, 8 december 2008

Ecodesign impact accounting, Status report for European Commission, VHK, June 2016 (ex-ante analysis with stock modeling for all appliances)

Relevant case studies

EPATEE Case Study “[Nordic Countries (Iceland, Finland, Norway, Sweden, Denmark) Nordsyn (Market Surveillance of Ecodesign and Energy Labelling) and the Effect Project

EPATEE Case Study [Denmark] Energy Companies’ Energy Savings Efforts

EPATEE Topical Case Study: Comparing Estimated versus Measured Energy Savings