23 – Evaluating Information/Education measures on energy efficient appliances using direct measurements

The primary audience for this guide is energy efficiency programme designers, implementers or supervisors, and evaluators looking for guidance on the evaluation process of energy savings in the scope of this specific guide.

Although the application of the specific guide will generally concern the (sub) national level, account will be taken of issues at EU level when relevant (e.g. the specific format of saving figures for the EED).

This guide is not about the preceding step in the evaluation process, the choice of the method. About this previous step in the evaluation process (the choice of the method), see the guidance provided here. However, after presenting the capabilities and limitations of the guide at hand, the user will be offered alternatives for the method within this guide (see section 6).

The objective of this guide is to provide:

• Information on the scope of the guide that enables the user to decide whether this guide is suited to his/her needs, and whether complementary or additional method(s) could be needed or useful (section 2);
• Guidance about specifying the evaluation objectives and requirements (section 3);
• Guidance about key methodological choices to calculate energy savings (section 4);
• Guidance about the inputs (data requirements) and outputs of the method (energy savings metrics) (section 5);
• Possible alternative methods (with pros and cons) (section 6)
• Background about evaluation results other than energy savings (section 7);
• Relevant examples, case studies and/or good practices (section 8);
• Relevant references for further reading (section 9).

The guide is intended for assessing realised (ex-post) energy savings. However, account is taken of earlier (ex-ante) evaluations of expected savings, if available (see section 4).

The focus of the guide is on impact evaluation, i.e. determining the energy savings, but not on how this has been reached through a step by step process with intermediate results (process evaluation).

Readers looking for the basic and general principles of energy efficiency evaluation may find the following link useful.

2.1 About Informational and Educational measures

More information and examples on the different subtypes residing under the main type Information/Education can be found here and here.

Informational and educational measure include energy billing, information campaigns, voluntary energy audits, regional or local information centres, voluntary labelling. The focus of this specific guide is restricted to focused information (advice) through information centres (households) or voluntary audits (companies).

Information centers provide households with additional information necessary to make informed decisions when buying new appliances. For this purpose, the provided advice is classified as to different appliances, so that the savings per appliance type can be calculated.

More detailed information on the evaluation of informational and educational measures can be found in this link.

2.2 Evaluation for a combination of policy measure types

When informational or educational measures are combined with other policy measures types it is assumed that the overall savings are mainly resulting from the policy measure at hand. However, the evaluation concerns the combined savings effect of both policy measures.

The guide is not capable of attributing part of the (overall) calculated savings to each of the policy measures (see also Double counting in the section on Gross to Net savings).

2.3 Evaluation when combined with energy taxes

The calculated savings effect for informational and educational measures will overlap with that of the energy tax. The guide is not capable of attributing part of the (overall) calculated savings to either the policy measures at hand or the energy tax. For dealing with this overlap, see section on Gross to Net savings.

2.4 About appliances in households

Information on (sub) sectors defined in the Toolbox can be found here, chapter 2, p.17.
The subsector considered here only concerns appliances in households and explicitly excludes electricity using systems related to buildings.

2.5 About the method direct measurement

Information about the various evaluation methods can be found in this link, table 1 and 2. This source also covers the combination of the method at hand with other methods, which will be dealt with below.

Direct measurement methods determine the energy consumption of one or more types of household appliances that are targeted by the policy measure information/education by in-situ measurement.

2.6 Complementary methods to determine total savings

Complementary methods are methods that are required, in addition to the primary selected method, to calculate energy savings.

The method at hand is meant to calculate unitary savings for one single appliance. These unitary savings should be multiplied by the number of actions or participants in order to have the calculated total savings. The number of actions or participants can be obtained in various ways. See this link, table 2 and 3.

The number of actions is sometimes directly available from the monitoring of the policy measure, such as the number of provided advice on specific efficient appliances through info-centers or internet. This numbers should be corrected for the follow-up rate from surveys.

Other possible sources for estimating the number of actions for this type of policy is deriving it from sales statistics provided by manufacturers or whole sellers. Surveys among the affected consumer groups can also be used to estimate the effect of focused information.

2.7 Additional methods to increase reliability of the results

An additional method can be applied on top of direct measurement to improve the reliability of the evaluation results and/or the cost-effectiveness of the evaluation approach.

For possible combinations with an additional method see chapter 6 in this link.

To check the results of direct measurement and reduce the effort for the evaluation billing analysis may be used, provided that the focused information on targeted appliances concerns a non-negligible part of total electricity consumption.

3.1 Meeting evaluation goals and ambition

The table shows whether this specific guide can be used to report on general evaluation goals or criteria. See also this document.

General types of evaluation goals or criteria	Level of ambition	Remarks
Calculation of realized energy savings from saving actions	Medium	Information on number of households buying efficient appliances due to the policy difficult to assess
Calculation of energy savings attributed to the policy measure(s)	Low	Additional information on follow-up rate of focused advices
Cost-effectiveness of saving action (for end-users)	High	Free advice and almost same price for more efficient appliances
Cost-effectiveness of policy (government spending)	Low	Info-centres are a rather costly activity given the often low follow-up of provided advice
CO2-emission reduction from saving actions	High	See energy savings
CO2-emission reduction attributed to the policy measure(s)	High	See energy savings

For more information on verification of actual energy savings and attribution/baseline/corrections see section 4, and for cost-effectiveness and emission reduction see section 7.
3.2 Reporting expectations
The method will make it possible to report (net) savings of labelling of household appliances. However, the described method of direct measurement is associated with a very high effort.

3.3 Time frame for evaluation

The length of the period under evaluation is dependent on the active period of the policy measure, the need to monitor developments before the implementation of savings actions (in case of methods based on before/after saving actions), and the time needed to present (reliable enough) results or impacts that fit into the decision making process. In some cases, the periodicity of evaluation can be set by law.

The planning of evaluation activities concerns regular monitoring of energy consumption and factors that define consumption, intermediate check of (ex-ante) estimated (unitary) savings through measuring or surveys, intermediate evaluations to improve the policy implementation and the final evaluation and reporting. Direct measurements can only be performed after physical implementation of the energy efficiency action.

3.4 Expertise needed for chosen method

Knowledge about the behaviour of households as to actually using the advice on efficient appliances (follow-up rate for advice), designing interviews and interviewing skills are needed for a proper evaluation.

General principles of calculating realized savings using different methods can be found here and here.

This section deals with key methodological choices to be considered when calculating energy savings: consistency between ex-ante and ex-post evaluation, baseline, normalization and adjustment factors. These choices are important to document when reporting energy savings, to ensure the transparency of the results.

The method measurement is applied using in-situ measurements of consumption of one or more appliances directly in households (or in the additional method by analyzing the energy bill of the household affected). It directly compares households that are targeted by the policy measure information/education and other households (see baseline choices).

4.1 Matching method with earlier ex-ante evaluation

From the viewpoint of methodological consistency and data availability using the same method in the ex-ante evaluation and in this specific guide on ex-post evaluation might be an obvious choice. However, for ex-ante evaluation only a few methods are usually considered, namely deemed savings, engineering estimate and stock modeling. The method direct measurement is not suited for ex-ante evaluations.

A different method than the one(s) used for the ex-ante evaluation can be applied for the ex-post evaluation, depending on the evaluation objectives, timeframe and data available for the situation after implementing the actions. For possible combinations of methods applied ex-ante and ex-post, see chapter 7, calculation approaches in this link.

If the measurement method does not provide an acceptable combination with the earlier applied ex-ante method it might be useful to select another method (see examples of alternatives in section 6).

In practice, ex-ante and ex-post evaluations are applied consecutively. The ex-post evaluation may also build on an earlier ex-post evaluation that makes use of measurement data for appliances coming from previous ex-post evaluation or studies (e.g. about previous periods of the same policy measure, or about the same types of energy saving actions as the ones promoted by the new policy measure). These previous ex-post studies could have used another type of method as well.

4.2 Calculation baselines

Energy savings are defined in general as the difference between the actual situation and a reference situation without the saving actions (and without the policy measures that influence these saving actions). In case of saving actions the reference situation can be defined using various calculation baselines. Before/after, With/without, Trend, Target/control group and Minimum efficiency standard. For further background, see here.

Energy savings in this specific guide can be calculated by applying the baseline option target/control group, where the control group contains households that have not received information on efficient appliances. Another baseline option could be Before/after but it is not possible to measure “before” as that household is only known after providing advice on efficient appliances.

4.3 Normalization factors

The calculation with a target/control group baseline provides a difference in energy consumption that should be corrected for influences on energy consumption other than the saving actions. These so-called normalization factors can be weather (with effect on consumption), the rebound effect and changes in energy using activities, such as production (industry), occupation rate (buildings) or car usage (transport).

Normalization is normally not needed as the factors will be the same for the Target group and the Control group. However, this should be checked and possibly the factors should be applied nevertheless. For example if the time series available for both groups are limited (which could thus mean that the energy savings would be calculated for a particular period, not necessarily representative of typical conditions). For this specific guide the normalization factors can concern the rebound effect, e.g. more intense use of efficient appliances they lead to a lower bill (see table 1 in this link).

4.4 Adjustment factors

Adjustment factors define which part of the calculated energy savings can be attributed to a policy measure or meets the definition of savings specified in the evaluation objectives or reporting requirements (see next section on “Calculating Gross and net savings).

Adjustment factors can concern the Free rider effect, the Spill-over/multiplier effect, Additionality and Non-compliance. See here.

Additionality and non-compliance are connected to unitary savings, while free riders and spill-over work through the number of actions (see next section on “Calculating Gross and net savings).

In case of another policy focusing on the same saving actions as evaluated with the guide at hand the adjustment factor Double counting might be relevant. If the other policy is not covered in the guide at hand double counting can only be accounted for at a higher level than individual guides.

(See Distinction of energy efficiency improvement measures by type of appropriate evaluation see the link here and here).

For correction of the gross unitary savings the factor non-compliance can be applied. See this link.

For correction of the gross number of saving actions, free riders can possibly be applied as adjustment factor.

For free riders a distinction must be made between saving actions due to the policy measure and actions which would have been taken anyway. The method measurement does not provide directly this information, thus other ways must be found, such as a survey among participants to the policy measure about their motivation, or application of Randomized Controlled Trial (RCT) or Quasi-experimental design (see further in this topical case study).

4.5 Calculating Gross and Net energy savings

Gross savings concern the calculated savings from saving actions using a chosen baseline and normalization factors. Net savings concern the savings attributed to policy measures or to a stakeholder (e.g. an energy company with an obligation to realise savings at their customers).

When calculating the savings a distinction must be made between the unitary savings and number of actions (see this link).

The gross unitary savings can be calculated using the baseline Target/control group (see section on baselines) and correcting, if needed, for the rebound-effect (see section on normalization factors).

The gross number of actions is determined with the complementary method (see chapter 2).

Total gross savings are equal to gross unitary savings, times gross number of actions.

The net unitary savings follow from the gross unitary savings after correction for non-compliance (see adjustment factors). The net number of actions follows from the gross number after correction for free riders. Total net savings are equal to net unitary savings, times net number of actions. See here and here.

The savings should be corrected for the Double counting effect, i.e. the overlap between the savings due to informational/educational measure and savings due to other policy measures. The overlap in the calculated savings of both policy measures cannot be processed at the level of a PSMC but must be corrected at the level of savings due to overall policy portfolios.

For addressing double counting see here and here.

See section on Concrete examples.

5.1 Main data requirements and data sources and collection technics

Data requirements specified in the table below correspond to the calculation of energy savings, when using the baseline option target/control group.

Calculation subject	Data requirements	Possible data sources and collection technics
Energy consumption /unitary savings	Electricity consumption per appliance	In-situ measurement (or possibly energy bills), smart metering
Normalization factors affecting energy consumption	Rebound effect (possibly)	Survey on changes in behavior
Adjustment factors	Free Rider   Non-compliance	Survey on reasons for purchase of new appliance Check on actual savings of advised efficient appliances
Number of actions	Number of provided advice on appliances (per type) and actually  replaced	Monitoring of activity of information centers and follow-up rate of advice from sales statistics and surveys in households
Target/control group	Data about participants might be needed to ensure representativeness	Monitoring of sample for participants and non-participants

Data issues when evaluating net energy savings

The main good practice to ensure the feasibility and reliability of the evaluation of net energy savings is to think about the method to be used when designing (or revising) the educational or informational measure.

Experience indeed shows that unless the data collection has been planned in advance, it will be very costly, time-consuming or even impossible to collect the data required to apply most of the methods that can be used to evaluate net energy savings. Which makes that in practice, using surveys will remain the only option possible (or considered feasible).

The main challenges when using surveys are:

• to achieve a high answer rate, in order to limit sampling bias;
• to use question phrasing that can limit the risk of bias in the answers;
• participants (especially in households) may not be aware of topics that are interesting for evaluators.

For more details about the evaluation of net energy savings, see the topical case study.

For possible other methods with different data demands see the section on alternatives for the chosen method.

5.2 Energy savings in final terms or in primary terms

Energy savings can be expressed in final terms or in primary terms. See definitions about primary and final energy here.

The measurement method can calculate savings in final terms. It can also calculate savings in primary terms provided that savings at end-users are calculated for each energy carrier apart, and primary factors are available to convert the savings in final terms to savings in primary terms.

5.3 Energy savings over time

Implemented saving actions in a year lead to savings over a number of consecutive years. E.g. a more efficient boiler can save gas over its lifetime of about 15 years, insulation over up to 60 years and more efficient computers up to 5 years. Energy savings can be calculated in different metrics in terms of time reference, for example: year-to-year, annual, cumulated annual, cumulative. See the definitions here.

The calculated yearly savings concern the savings of all new saving actions in that year. In this approach only data for the savings in the chosen year are needed. Adding up the yearly savings over a period, provided that earlier saving actions are still delivering savings, leads to cumulative savings. For the cumulative savings data are needed for the whole period. Another cumulative approach, to be applied for the EE directive, is to multiply the (new) savings in a year with the number of years up to a target year and sum this result with that for all other years up to the target year. This cumulative approach stimulates early saving actions, as this count more times to the target than later actions.

Finally, savings from a saving action can be discounted and summed up over the lifetime of the action See link here.

The measurement method (and alternative billing analysis) can provide yearly savings of new saving actions in that year. It can also provide cumulative savings provided that data are available over a period.

6.1 Alternatives for the chosen method

Often other savings calculation methods can be applied as well, although they will all have pros and cons regarding various aspects dealt with in preceding sections.

The table below presents the pros and cons of the method for evaluating informational and educational measures for household appliances, and for commonly used alternative methods for the same combination of policy measure and sector.

Type of method	Pros	Cons
Measurement [and billing analysis] for unitary savings of replacing appliances	Precise determination of savings per appliance	Expensive and time consuming
Deemed savings for unitary savings of replacing appliances	Less expensive and time consuming	Savings depend on the old and new appliance, which both can differ, set of deemed savings needed. Adjustment and normalization of savings is harder

7.1 Calculating avoided CO2 emissions

Avoided CO2 emissions can be evaluated from the energy savings by applying emission factors. Four key aspects are to be taken into account when choosing the emission factor(s):
1) Emission factors vary according to the energy type, so the data about energy savings need to be available per energy type.
2) Emission factors for a given type of energy can vary over time (especially for electricity).
3) Emission factors can take into account:
a. Direct emission factors: that take into account the emissions generated when producing the energy used;
b. Lifecycle emission factors: that takes into account all the emissions generated from the extraction of the energy resources up to the dismantling of the energy plant.
Due to the differences that the choice of emission factor(s) can induce, it is important to document what emission factor(s) has (have) been used.
The reduction in CO2-emissions can be calculated from the savings with an emission factor for electricity that takes into account the different inputs of power production. The actual factor to be applied can vary, depending on saving action(s) and sector, year of implementation, policy considerations, and etcetera, see the document in this link as example
The avoided emission of other greenhouse gasses due to energy savings are not taken into account here, as these emissions (and more specifically their reductions) are generally negligible compared to CO2 (apart from policy measures targeting the agriculture sector).
IPCC (Intergovernmental Panel on Climate Change) provides a detailed database of peer-reviewed emission factors.

7.2 Calculating cost-effectiveness

Cost-effectiveness is the ratio between costs to achieve energy savings and the amount of savings and possibly other benefits.

A distinction can be made according to the point of view adopted to assess cost-effectiveness:

• Cost-effectiveness for the end-user or participant
• Cost-effectiveness for society at large
• Cost-effectiveness for the party that takes responsibility for saving targets (government or actor with an Energy Efficiency Obligation)

For more details about the different perspectives, see for example (Breitschopf et al., 2018).

The calculation of cost-effectiveness for end-users demands, next to the savings, data on investments made, subsidies on investments, interest rates, lifetimes of the saving actions, energy prices (including taxes) per type of end-user and discount factors per type of end-user. The calculation is based on additional cost of appliances, which are equipped with an energy efficiency label.

For cost-effectiveness from a societal viewpoint no account is taken of subsidies and taxes, energy prices concern world market price, and a lower value of the discount factors is valid. The calculation may be based on the cost for setting up the measure and the combined energy savings of all households affected.

7.3 Calculating other Co-benefits

Possible co-benefits from saving energy concern:

• Extra employment
• Reduction of energy poverty
• Other emission reductions (NOx, SO2, fine particles, etc.)
• Better indoor climate
• Reduced dependency on (insecure) energy import

The following co-benefits are regarded as most relevant and/or feasible to calculate in conjunction to this specific guide (see table)

The calculation of the co-benefits is explained in the references.

Type of co-benefits	Why it can be relevant (and for whom)	References where more details can be found
Alleviation of energy poverty	Low-income households	Energy Efficiency for Low Income Households (European Commission 2016)

Voluntary Labelling of Refrigerators and AC units using direct measurement supported by surveys (Thailand)

Vine, E.; du Pont, P.; Waide, P.: Evaluating the impact of appliance efficiency labeling programs and standards: process, impact, and market transformation evaluations
https://www.sciencedirect.com/science/article/pii/S0360544201000536

General guidance on evaluations

• Baumgartner, R. (2017). Chapter 12: Survey Design and Implementation for Estimating Gross Savings Cross-Cutting Protocol. The Uniform Methods Project: Methods for Determining Energy-Efficiency Savings for Specific Measures. Prepared for NREL (National Renewable Energy Laboratory), September 2017. http://www.nrel.gov/docs/fy17osti/68568.pdf
• Hoffman, I., Schiller, S., Todd, A., Billingsley, M., Goldman, C., Schwartz, L., 2015. Energy Savings Lifetimes and Persistence: Practices, Issues and Data. Technical Brief, Lawrence Berkeley National Laboratory, May 2015. https://emp.lbl.gov/publications/energy-savings-lifetimes-and
• Eichhammer et al., 2008. Distinction of energy efficiency improvement measures by type of appropriate evaluation method. Final Report on Evaluation and Monitoring for the EU Directive on Energy End-Use Efficiency and Energy Services. EMEEES Project report. https://www.epatee-lib.eu/media/docs/EMEEES_WP3_Report_Final.pdf

Relevant case studies

• None available