29 – Evaluating market based instruments on energy efficiency in the residential sector using econometric methods

The primary audience for this guide is public authorities, obligated parties and other stakeholders involved in EEOs, as well as evaluators looking for guidance on the evaluation process of energy savings in the scope of this guide.

Although the application of the guide will generally concern the national level, account will be taken of issues at EU level when relevant (e.g. the specific format of saving figures for the EED and more particularly its’ Article 7).

This guide is not about the preceding step in the evaluation process, the choice of the method, but to present the case of one particular situation (combination of policy measure, sector and method). About this previous step in the evaluation process, see the general guidance about integrating evaluation into the policy cycle. However, after presenting the capabilities and limitations of the guide at hand, the user will be offered alternatives for the method within this guide (see section 6).

The objective of this guide is to provide:

• Information on the scope of the guide that enables the user to decide whether this guide is suited to his/her needs, and whether complementary or additional method(s) could be needed or useful (section 2);
• Guidance about specifying the evaluation objectives and requirements (section 3);
• Guidance about key methodological choices to calculate energy savings (section 4);
• Guidance about the inputs (data requirements) and outputs of the method (energy savings metrics) (section 5);
• Possible alternative methods (with pros and cons) (section 6)
• Background about evaluation results other than energy savings (section 7);
• Relevant examples, case studies and/or good practices (section 8);
• Relevant references for further reading (section 9).

The guide is intended for assessing realised (ex-post) energy savings. However, account is taken of earlier (ex-ante) evaluations of expected savings, if available (see section 4).

The focus of the guide is on impact evaluation, i.e. determining the energy savings, but not on how this has been reached through a step by step process with intermediate results (process evaluation).

Readers looking for the basic and general principles of energy efficiency evaluation may find the following link about general guidance useful.

2.1 About market based instruments

EEOs are a type of market-based instruments (general category used in the EPATEE Toolbox and Knowledge Base, as well as in the MURE database). Market-based instruments are policies that set targets in terms of outcomes (e.g. energy savings) to be delivered by market actors, without prescribing the delivery mechanisms and types of actions to achieve these targets (IEA, 2017).

EEOs are regulatory mechanisms setting energy savings targets, which can be expressed in energy or CO2, savings that must be achieved by obligated parties (either energy suppliers or energy distributors). EEOs include rules about what types of actions can be eligible, how energy savings shall be calculated, how obligated parties shall demonstrate their role in stimulating actions (materiality issue), if third parties can be involved in the scheme, if approved energy savings can be traded (cf. white certificates schemes), penalties in case of non-achievement, etc. (Bertoldi et al. 2015).

The main specificity of market-based instruments (here EEOs) is that, within the framework set by the policy, the market actors are free to choose their strategy (flexibility of delivery mechanisms). The underlying assumption is that this approach helps to minimize or optimize the costs of energy savings. In practice, the cost-efficiency of an EEO scheme will depend, among other factors, on the efficiency of its administration, and particularly the way that the energy savings are monitored and verified (Broc 2017).

EEOs can also have other goals than least-cost savings, such as alleviating energy poverty or favouring the market penetration of innovative actions or any actions favored by authorities for ad hoc reasons. These goals can influence the evaluation objectives.

In terms of evaluation, one specificity of EEOs is that most of the activities of the schemes are implemented by market actors. This can create difficulties in accessing or collecting data, particularly about costs (Rosenow and Bayer, 2017). Public authorities can enforce requirements about what data shall be provided or stored by the obligated parties. A key design issue is thus to find the right balance between these requirements and minimizing the administration costs (for the obligated parties, as well as for the public authorities making the controls and verifications).

More information and examples on the different subtypes residing under the main type market based instruments can be found in: in the EPATEE Toolbox and Knowledge Base, as well as in the MURE database). The focus of this guide is restricted to Energy Efficiency Obligations (EEO), but is useful as well to evaluate White Certificate Systems (WCS), which can be seen as an extension of the EEO.
In the evaluation of broad schemes like EEO (Energy Efficiency Obligation) or WCS (White Certificates Schemes), different methods can be applied for various (sub) sectors or types of projects. Therefore, this guide can be combined with other guides to evaluate the savings of the scheme.

Article 7 of the Energy Efficiency Directive (EED) requires EU Member States to introduce energy efficiency obligation schemes. EEOS require energy companies to save 1.5% of their final energy sales on an annual basis with EE projects. As an alternative to EEOSs, Article 7 provides Member States the option to introduce alternative policy measures to EEOS as long as these measures deliver equivalent energy savings.

More detailed information on the evaluation of market based instruments can be found in this link.

2.2 Evaluation for a combination of policy measure types

Depending on the national context, there can be overlaps between the EEO and other policy measures. The EEO can include rules to avoid certain overlaps, for example by specifying that actions receiving a public subsidy cannot be eligible to the EEO (e.g. in Austria or Italy), or that only actions with a performance higher than minimum requirements set in current regulations are eligible (rule used for most EEOs).

At the opposite, the EEO can allow actions that received a support or incentive from other policy measures. For example, when the financial barriers are supposed to be too high to be overcome by incentives provided by market actors only (e.g. for renovation works in dwellings, as in France and Ireland).

When market based instruments are combined with other policy measures types it is assumed that the overall savings are mainly resulting from the policy measure at hand. However, the evaluation concerns the combined savings effect of both policy measures.

The guide at hand is not capable of attributing part of the (overall) calculated savings to each of the policy measures (see also Double counting in the section on Gross to Net savings).

For attributing part of the (overall) calculated savings to each of the policy measures see section on Concrete examples.

2.3 Evaluation when combined with energy taxes

The calculated savings effect for market based instruments will overlap with that of the energy tax. The guide is not capable of attributing part of the (overall) calculated savings to either the policy measures at hand or the energy tax. For dealing with this overlap see section on Gross to Net savings.

Separating the effects of the EEOs and carbon/energy taxes goes beyond the scope of this guidance.

It should be noted that EEOs can have an impact on energy prices, depending on the cost recovery mechanisms, i.e. how the obligated parties can recover the costs they have incurred to meet their energy efficiency targets. When analysing the interactions between EEOs and carbon/energy taxes, a first issue is therefore to assess the impact of the EEOs on energy prices (Giraudet and Finon, 2015). However, in most cases, this impact remains small compared to the share that energy taxes represent in energy prices. Nevertheless, this impact might be high compared to the margin of the energy suppliers, which is important when the supplier is selling forward energy at fixed price and shall anticipate the future impact of EEOs on the energy price.

2.4 About the residential sector

EEOs can be transversal (i.e. actions are eligible in several or all sectors, like for most EEOs in Europe), or restricted to a particular sector (e.g. residential sector, like in UK). This guidance is focused on actions done in the residential sector.

Most EEOs make the difference between action types that can be standardised and thus evaluated with deemed savings (case dealt with in this guidance), and action types that are more specific and require case-by-case energy savings calculations

Actions done in the residential sector are usually easier to standardise, hence the focus of this guidance. Nevertheless, most of this guidance can also be applied to standardised actions defined for other sectors

Information on (sub) sectors defined in the Toolbox can be found in the EPATEE terminology, chapter 2, p.17.

2.5 About econometric analysis

Information about the various evaluation methods can be found here, table 1 and 2. This source also covers the combination of the method at hand with other methods, which will be dealt with below.

The econometric method is a top-down simulation with econometric models. An econometric model specifies the statistical relationship that is believed to hold between the various economic quantities pertaining to a particular economic phenomenon, in this particular case energy savings. In the case of the econometric method, a detailed econometric model with access to an update database is required and is also a prerequisite for the use of this method, as setting up such a detailed economic model is a significant effort. However, in several countries such a model and database will be available. The model provides a regression analysis of energy use over time.

Regression analysis can attribute changes in energy consumption to all kind of explaining factors, in this case the uptake of different retrofit actions and their effects on gas consumption, dependent on a set of dwelling features, income and tenure. The results incorporate also the effects of higher energy prices and stimulating policies (including free riders and take-back, thus the net savings). But these effects have not been separated from total savings because they were not taken into account in the regression analysis. None of the additional methods can address these effects because there is not a control group with different prices and policy. As for total savings billing analysis is not useful because it cannot attribute the savings to different retrofit actions or policy effects. The measurement method can be applied (on a subset of dwellings) at disaggregated level to check the savings per type of retrofit action, but cannot separate the policy effect. The same is true for engineering estimates, but this method is more costly. Deemed savings that have been checked with other methods can be used as a check on savings per retrofit action. But the unit consumption method is only useful for equipment and not for insulation measures. As for total savings methods the (sub) sector intensity method provides neither the savings per retrofit action, nor the policy effects.

2.6 Complementary methods to determine total savings

This guide does not concern complementary methods.

2.7 Additional methods to increase reliability of the results

An additional method can be applied on top of econometric analysis to improve the reliability of the evaluation results and/or the cost-effectiveness of the evaluation approach.

In regression analysis the combined effect of a set of explaining variables is analysed. This implies that leaving out, or adding, an explaining variable may change the values of regression factors for all other explaining variables. Moreover, the results are dependent on the quality of the yearly data for all variables applied.

Regression analysis focusing on aggregated energy consumption is a top-down “black box” approach which can best be checked with bottom-up approaches focusing on the different saving actions influenced by energy taxes. This concern the methods deemed savings, engineering estimates, unit consumption or (sample-wise) measurement.

Type of method	Short description	Objective
Deemed savings	Deemed savings are the result of standardised calculations where most of the parameters are defined beforehand, based on reference values (e.g. national statistics on the building stock) or results from previous studies on the same type of action.	To verify if the average values and assumption used in the econometric models are representative of the participants or actions reported
Direct measurement	Direct measurement can be done on a specific parameter (e.g. sensors to monitor duration of use), or on the energy consumed by a given equipment or process (e.g. sub-metering). For more details, see e.g. (Mort 2017)	To verify if the average values and assumption used in the econometric models reflect actual conditions (e.g. energy performance, duration of use, indoor temperature)
Billing analysis	Calculation of energy savings from metered data of energy bills. For more details, see Specific Guidance 29	To assess actual energy savings (as can be experienced by the end-users)
Engineering estimates	Calculation of energy savings, based on engineering model or formula, and using data specific to the cases evaluated (e.g. using of Energy Performance Certificates) For more details, see Specific Guidance 30	To test the robustness of the results of the econometric models

For possible combinations with an additional method see also chapter 6 in the document here.

3.1 Meeting evaluation goals and ambition

The table shows whether this guide (econometric methods) can be used to report on general evaluation goals or criteria. See also (Broc et al., 2009).

General types of evaluation goals or criteria	Level of ambition	Remarks
Calculation of realized energy savings from saving actions	Limited	Regression analysis does not focus on specific saving actions
Calculation of energy savings attributed to the policy measure(s)	Low	Regression analysis may be used for distinction of autonomous and price driven effects from energy savings (top-down)
Cost-effectiveness of saving action (for end-users)	N/A	Method does not provide insights on cost-effectiveness
Cost-effectiveness of policy (from a society’s perspective)	N/A	Method does not provide insights on cost-effectiveness
CO2-emission reduction from saving actions	Limited	Regression analysis does not focus on specific saving actions
CO2-emission reduction attributed to the policy measure(s)	Low	See energy savings

For more information on verification of actual energy savings and attribution/baseline/corrections, see section 4, and for cost-effectiveness and emission reduction see section 7.

3.2 Reporting expectations

The rules of the EEO include the definition of the unit used to count the results to be reported by the obligated parties. This unit includes several criteria for which various options are possible.

Criteria	Common options	Remarks
Nature of the objective	Energy savings CO2 savings Bill savings	Choice mostly depending on the primary objective of the EEO
Duration for which the results are counted	Annual (or first-year) Lifetime cumulated Cumulative over the obligation period	Choice depending on the EEO objectives and on the diversity of actions possible. A “lifetime cumulated” unit can be chosen to value long-lifetime actions, Possibly applying a discount factor (for more details see Energy Savings over time in section 5). Defining saving lifetimes requires assumptions that can add uncertainties to the deemed savings.
Energy basis (if nature = energy savings)	Primary energy savings Final energy savings	It is a political choice, depending on the intention of the objectives of the public authorities. The two biggest EEOS in Europe are for one (French) based on final energy savings, and the other one (Italian) on primary energy savings. It shall be mentioned that to include cogeneration in the scope of the actions, primary energy savings shall be taken into account. EED article 7 requires to report to the European Commission final energy savings. However it allows countries to count primary energy savings at national level.
Energy unit (if nature = energy savings)	Multiple of Joule (e.g. PJ), toe (e.g. ktoe) or kWh (e.g. TWh)	Choice usually depending on the energy unit commonly used in the country (e.g. for the national energy balance)
Evaluation perspective	Gross / Additional / Net	For more explanations, see in section 4 Calculating Gross and Net energy savings.

Possible reporting options:

• Net savings
• Yearly savings
• Effect of other variables than energy taxes, e.g. energy using activities or another policy measure.

3.3 Time frame for evaluation

The length of the period under evaluation is dependent on the active period of the policy measure, the need to monitor developments before the implementation of savings actions (in case of methods based on before/after saving actions), and the time needed to present (reliable enough) results or impacts that fit into the decision making process. In some cases, the periodicity of evaluation can be set by law.

For regression analysis, the time series for analysis should be as long as possible, covering changes in most of the explaining variables and/or energy consumption. Preferably, the length of the period before the introduction of the policy should be as long as that of the active period of the measure (without major changes in this period).

The planning of evaluation activities concerns regular monitoring of energy consumption and factors that define consumption, intermediate check of (ex-ante) estimated (unitary) savings through measuring or surveys, intermediate evaluations to improve the policy implementation and the final evaluation and reporting. See also the EPATEE guidance about how to plan and prepare evaluations.

3.4 Expertise needed for chosen method

The actual estimation and interpretation requires expertise in econometric and statistical analysis (theory and practice) including the testing of statistical parameters to assure quality and robustness of the results.

In order to set up a reliable regression analysis expertise is needed on the possibilities and limitations: the choice of relevant influencing variables, possible regression formulas (including lagged variables or deterministic variables) and the availability/quality of time series for all variables.

General principles of calculating savings using different methods can be found in (Broc et al., 2009) and in (Eichhammer et al., 2008).

This section deals with key methodological choices to be considered when calculating energy savings: consistency between ex-ante and ex-post evaluation, baseline, normalization and adjustment factors. These choices are important to document when reporting energy savings, to ensure the transparency of the results.

Contrary to most other methods used in the guidance this method does not show the composition of the savings, e.g. amount of insulation actions times the unitary savings per action, including the investments made, etcetera. However, given an application per category, insight in the contribution per group of end-users will be provided.

This method provides by nature the net savings due to the market-based instrument, thus no corrections of gross savings are needed. However, if other policy measures focus on the group of end-users affected by the market-based instrument, a correction should be made for Double counting (the contribution of other policy measures, see paragraph on Gross to Net savings).

4.1 Matching method with earlier ex-ante evaluation

From the viewpoint of methodological consistency and data availability using the same method in the ex-ante evaluation and in the ex-post evaluation is recommended. However, regression analysis builds on an extensive set of observed values over a considerable period. Using regression analysis for the ex-ante evaluation is only possible if information is available from comprehensive energy scenarios. Therefore, regression analysis is most often not an option for an ex-ante evaluation.

Ex-post regression analysis could be matched with ex-ante methods such as deemed savings, engineering estimate or elasticity analysis (see chapter 7 in this document).

The deemed savings and engineering estimate methods concern the ex-ante calculation of individual savings for a given action. A complementary method providing number of actions (see section 2) is needed to calculate the total savings that can be compared with the regression results. However, given these ex-ante bottom-up methods, applying them ex-post as well seems more appropriate than regression analysis.

The elasticity analysis method provides directly ex-ante total savings (see Specific Guidance 22 ), which can be compared with ex-post regression results. Given an evaluation using ex-ante elasticity analysis, the ex-post regression analysis can be seen as a more elaborate evaluation version (see section 2). Regression analysis provides the opportunity to calculate the effect of a measure while accounting for the effect of other drivers and policy measures.

It is important here to distinguish ex-post verifications done as part of controls, and ex-post verifications done as part of an evaluation.

Controls are commonly decided on a risk-based approach, in order to focus the means of control where the risks are higher or more critical. Therefore, results from controls are not meant to be representative.

Evaluation methodologies are usually designed to look at the whole scheme, taking into account representativeness, risks of sample bias, etc. Depending on the evaluation objectives, evaluation results can thus be expected to be representative.

If the econometric method does not provide an acceptable combination with the earlier applied ex-ante method it might be useful to select another method (see examples of alternatives in section 6).

In practice, ex-ante and ex-post evaluations are applied consecutively. The ex-post evaluation builds on an ex-ante evaluation that makes use of data coming from previous ex-post evaluation or studies (e.g. about previous periods of the same policy measure, or about the same types of energy saving actions as the ones promoted by the new policy measure). These previous ex-post studies could have used another type of method as well

4.2 Calculation baselines

Energy savings are defined in general as the difference between the actual situation and a reference situation without the saving actions (and without the policy measures that influence these saving actions). In case of saving actions the reference situation can be defined using various calculation baselines. For further background see the link to this document.

In the case of the present guide, no baseline is applicable because the calculated savings from saving actions due to energy taxes are a direct result of the regression analysis, where the introduction of the measure is one of the explaining variables. The introduction of the tax in a year resembles a “Before/after” baseline but because of an analysis over the whole period and for all other explaining variables no separate results before and after the introduction moment are supplied.

4.3 Normalization factors

The calculation with no baseline provides a change in energy consumption that should be corrected for influences on energy consumption other than the saving actions. These so-called normalization factors can be weather (with effect on consumption), the rebound effect and changes in energy using activities, such as production (industry), occupation rate (buildings) or car usage (transport).

For more details about prebound and rebound effects, see for example the topical case study about the comparison between estimated and measured energy savings (Sipma et al., 2019), or the webinar #4 of EPATEE.

The rebound effect is already accounted for in observed energy consumption and the change in activity is processed in the time series for activity as explaining variable. The only normalization factor weather can be processed through correction of energy consumption data for yearly deviations from long term weather (mean outdoor temperature during the heating season or mean number of hot summer days for cooling). But this correction can also be made part of the regression analysis, using time series for weather as explaining variables.

4.4 Adjustment factors

Adjustment factors define which part of the calculated energy savings can be attributed to a policy measure or meets the definition of savings specified in the evaluation objectives or reporting requirements (see next section on “Calculating Gross and net savings).

For a general introduction about adjustment factors, see table 1 here.

The only applicable adjustment factor for this guide is Double counting (see section on Gross and Net savings). The other factors are connected to either the unitary savings or the number of actions, but these quantities are not covered in the regression analysis method.

5.1 Main data requirements and data sources and collection technics

Data requirements specified in the table below correspond to the calculation of energy savings.

Calculation subject	Data requirements	Possible data sources and collection technics
Energy consumption (end-use or group  of end-users)	Yearly values per type of energy carrier, corrected for variations in heating degree days	EU and national statistics
Energy price per sector (per group of end-users)	Mean prices per type of energy carrier delivered to end-users	EU and national statistics
Relevant activity levels as to energy consumption	Yearly activity level	EU and national statistics or other sources
Other discrete variables (e.g. other policy measures)	Introduction year of policy measure (value 0 > 1)	Policy measure overview

Data issues when evaluating energy savings

The main good practice to ensure the feasibility and reliability of the evaluation of energy savings is to think about the method to be used when designing (or revising) the market-based instrument.

Experience indeed shows that unless the data collection has been planned in advance, it will be very costly, time-consuming or even impossible to collect the data required to apply most of the methods that can be used to evaluate net energy savings. Which makes that in practice, using surveys will remain the only option possible (or considered feasible).

For more details about the evaluation of net energy savings, see the dedicated topical case study (Voswinkel et al., 2018).

For possible other methods with different data demands see the section on alternatives for the chosen method.

5.2 Energy savings in final terms or in primary terms

Energy savings can be expressed in final terms or in primary terms See definitions about primary and final energy in the EPATEE terminology.

Regression analysis can calculate savings in final terms. It can also calculate savings in primary terms provided that savings at end-users are calculated for each energy carrier apart, and primary factors are available to convert the savings in final terms to savings in primary terms.

5.3 Energy savings over time

Regression analysis cannot provide yearly or cumulative savings of new saving actions in a certain year as it cannot separate single saving actions (each with their own lifetime).

6.1 Alternatives for the chosen method

Often other savings calculation methods can be applied as well, although they will all have pros and cons regarding various aspects dealt with in preceding sections.

The table below presents the pros and cons of the method for evaluating market-based instruments in residential building, and for commonly used alternative methods for the same combination of policy measure and sector.

Type of method	Pros	Cons
Deemed savings	·        Provide visibility to stakeholders ·        No delay in getting results from the monitoring system ·        Low running cost (once the catalogue is defined)   Calculations directly related to the energy efficiency improvements due to the energy saving actions	·        Use limited to action types that can be described in a standardised way ·        Do not reflect the energy savings achieved for a given situation, but an average result for a population of actions ·        Can require significant preliminary efforts (if many action types to be included in the catalogue) ·        Quality depending on the data available to define deemed savings ·        Possible gaps between deemed savings and actual savings (see section 4)   Additional method needed to evaluate ex-post the additionality of the savings (see section 4)
Engineering calculations (see also the related Specific Guidance 30)	·        Can be used for almost all action types ·        Can enable to automatize energy savings calculations (through standardised formula for simple cases) ·        Can reflect the energy savings achieved for a given situation (specific calculations) ·        Limited delay in getting the results (calculations can be done before the actions are installed)	·        Require to collect data for each case (so can be costly if data collected only for this purpose and for large numbers of actions / projects) ·        Possible gaps between engineering estimates and measured savings (see the corresponding topical case study (Sipma et al., 2019)) ·        Additional method needed to evaluate ex-post the additionality of the savings (see section 4)
Direct measurements	·        Provide data about actual energy consumption (for the baseline and/or for the situation with energy saving actions) or about actual values for key parameters (e.g. power, duration of use) ·        Can be used to assess performance gaps	·        Can be costly if measurements only done for this purpose and for large numbers of actions ·        If sampling is used, attention should be paid to avoid sampling bias (if data are to be extrapolated) ·        Additional method needed to evaluate ex-post the additionality of the savings (see section 4) ·        Delay in installing the actions (if used to verify the baseline, then time needed to make the measurements, unless data are already available) ·        Delay in getting the results (if used to verify the situation with energy saving actions, then time needed to make measurements after the actions are installed + time to analyse the data)
Billing analysis (see also the related Specific Guidance 8)	·        Provide data about actual energy consumption / energy savings ·        Can be used to evaluate ex-post net or additional savings (if a control or comparison group can be found)	·        Can only be used for ex-post evaluation ·        Frequent difficulties to collect billing data (unless data collection carefully planned and prepared in advance, e.g. collecting participants’ approval when actions are installed) ·        Difficulties to get representative samples (cf. sampling bias + data losses along the evaluation process) ·        Delays in getting the result (at least one year to get the consumption after installing actions + time to process and analyse data) ·        Difficulties to find relevant control or comparison groups (when assessing net or additional savings)

7.1 Calculating avoided CO2 emissions

Avoided CO2 emissions can be evaluated from the energy savings by applying emission factors. Four key aspects are to be taken into account when choosing the emission factor(s):

1. Emission factors vary according to the energy type, so the data about energy savings need to be available per energy type.
2. Emission factors for a given type of energy can vary over time (especially for electricity).
3. Emission factors can take into account:
   1. Direct emission factors: that take into account the emissions generated when producing the energy used;
   2. Lifecycle emission factors: that take into account all the emissions generated from the extraction of the energy resources up to the dismantling of the energy plant.

Due to the differences that the choice of emission factor(s) can induce, it is important to document what emission factor(s) has(have) been used.

The reduction in CO2-emissions can only be calculated when savings are calculated per relevant energy carrier and a specific emission factor is available for each energy carrier

The avoided emission of other greenhouse gasses due to energy savings are not taken into account here, as these emissions (and more specifically their reductions) are generally negligible compared to CO2 (apart from policy measures targeting the agriculture sector).

When needed, IPCC (Intergovernmental Panel on Climate Change) provides a detailed database of peer-reviewed emission factors.

7.2 Calculating cost-effectiveness

Cost-effectiveness is the ratio between costs to achieve energy savings and the amount of savings and possibly other benefits.

A distinction can be made according to the point of view adopted to assess cost-effectiveness:

• Cost-effectiveness for the end-user or participant (e.g. payback time)
• Cost-effectiveness from the obligated parties’ point of view (e.g. least cost of target achievement)
• Cost-effectiveness for society at large (e.g. social net present value)
• Cost-effectiveness from the point of view of the public authority (e.g. comparing different types of policy measures)

For more details about the different perspectives, see for example (Breitschopf et al., 2018).

In the case of an EEO scheme, the calculation of cost-effectiveness requires to collect other data on top of the ones used to calculate energy savings, as summarized in the table below. The costs and benefits listed in this table apply when considering the point of view of categories (i.e. all participants together, all obligated parties together, etc.). When making a cost-benefit analysis from an individual point of view (i.e. one participant, one obligated party), the selection of costs and benefits can be different according to the own view and practice of this participant or obligated party.

Point of view	Costs	Benefits
Participants	·         Part of the investments paid by the participants	·         Financial aids received from obligated parties (or other intermediaries) ·         Gross energy savings
Obligated parties	·         Costs to achieve their targets* ·         Losses in revenues (due to the additional energy savings)	·         Costs recovered on network tariffs (if energy distributors) or energy prices (if energy suppliers) ·         Costs of energy production (or distribution or purchase) avoided due to the additional energy savings
Public authorities	·         Administration costs ·         Losses in tax revenues (due to additional energy savings)	·         Increases in tax revenues (due to additional investments made in energy efficiency actions)
Society	·         Part of the investments paid by the participants (for additional actions only) ·         Costs of the obligated parties ·         Administration costs for the public authorities	·         Additional energy savings

*: when possible, these costs can be disaggregated in sub-categories, especially to differentiate administration costs (e.g. costs of reporting to the public authority), communication & marketing costs (e.g. communication campaigns) and costs related to the technical or financial support provided to final customers (e.g. energy audits, grants). It can also be useful to identify costs related to quality processes and monitoring.

NOTE: the table above does not deal with non-energy impacts. Depending on the context and objectives of the EEO scheme, non-energy benefits can be larger than the benefits from energy savings. When assessing the cost-effectiveness of an EEO scheme from the society’s point of view, it is therefore recommended to consider if it is relevant to include non-energy impacts in the scope of analysis.

Experience has shown that the fact that EEO scheme involves private actors makes it often difficult to collect cost data, and particularly homogeneous cost data. When obligated parties report cost data, it is indeed common that they use different sub-categories of costs, or only their “total” costs (with a different scope of “total” costs from one obligated party to the other).

When possible, it can therefore be useful to consult with obligated parties to define clear categories of costs.

When the obligated parties are energy distributors, they usually have to report their total costs to the energy regulatory authority, so that they are allowed to recover their costs on network tariffs. This provides information about total costs for obligated parties. But it does not mean that costs per sub-categories will be available.

When the obligated parties are energy suppliers, there is usually no direct obligation for them to report cost data. This can be asked by the public authorities on a voluntary basis. Or it can be required by including a specific provision in the law enforcing the scheme (if enforced by law). However there could be limitations due to other legal provisions about confidentiality of strategic data for private actors.

Depending on the indicator(s) used to assess cost-effectiveness, it can be needed to use discount factors (e.g. when the indicator is Net Present Values). In that case, it is important to document the use of discount factors, and if possible to make a sensitivity analysis (testing several values or ranges of discount factors). As this can affect significantly the results.

Likewise, the calculations of cost-effectiveness indicators will usually require to consider scenarios of energy prices over given periods. The assumptions about trends in energy prices should be documented. Whenever possible, it is recommended to make a sensitivity analysis (testing several scenarios of energy prices).

For more discussions about cost-benefit analysis of EEO schemes in Europe, see for example (Rosenow and Bayer 2017).

For the policy measure covered in this guide no cost-effectiveness can be calculated because it concerns a top-down approach without data on saving actions with their (cost) savings, investments, etc.

7.3 Calculating other Co-benefits

Possible co-benefits from saving energy concern:

• Extra employment
• Reduction of energy poverty
• Other emission reductions (NOx, SO2, fine particles, etc.)
• Better indoor climate
• Reduced dependency on (insecure) energy import

For a general background about non-energy impacts, see the corresponding EPATEE general guidance.

For the same reasons as mentioned for cost-effectiveness most of the other co-benefits cannot be calculated for this guide. Only reduced import dependency can be determined from the savings, provided that the effect of taxation is calculated per energy carrier type.

For the case of EEO scheme in the residential sector, a special attention is frequently paid to the impacts on energy poverty. This is indeed now required by the amended Energy Efficiency Directive (EU2018 (2002)).

Indeed for most EEO schemes, the costs incurred by the obligated parties will be recovered, partly or fully, either on network tariffs (if energy distributors) or energy prices (if energy suppliers). So at the end, EEO schemes can have an impact on energy prices, thereby increasing the risks of energy poverty.

However, the impact of an EEO scheme on energy prices should be compared to existing levels of taxations on energy prices, as well as to current trends in wholesale energy prices. If the impact of the EEO scheme is small compared to these, therefore it also means that its possible negative impact on energy poverty is also limited.

At the opposite, some EEO schemes include specific objectives or provisions to tackle energy poverty. For example, this is the priority objective and single scope of the new period of the UK EEO scheme (since December 2018).

In case of a transversal EEO scheme (covering several sectors), the evaluation objectives can also include assessing the possible cross-sectoral effects in terms of cross-subsidizing. Such effects can for example happen when it is more cost-effective for obligated parties to achieve their targets in a given sector, whereas costs will be recovered on energy prices in all sectors.

I.G. Hamilton, A.J. Summerfield, D. Shipworth, J.P. Steadman, T. Oreszczyn, R.J. Lowe
Energy efficiency uptake and energy Savings in English houses: a cohort study
Energy and Buildings, 118 (2016), pp. 259-276

General guidance on evaluations

• Baumgartner, R. (2017). Chapter 12: Survey Design and Implementation for Estimating Gross Savings Cross-Cutting Protocol. The Uniform Methods Project: Methods for Determining Energy-Efficiency Savings for Specific Measures. Prepared for NREL (National Renewable Energy Laboratory), September 2017. http://www.nrel.gov/docs/fy17osti/68568.pdf
• Hoffman, I., Schiller, S., Todd, A., Billingsley, M., Goldman, C., Schwartz, L., 2015. Energy Savings Lifetimes and Persistence: Practices, Issues and Data. Technical Brief, Lawrence Berkeley National Laboratory, May 2015. https://emp.lbl.gov/publications/energy-savings-lifetimes-and
• Eichhammer et al., 2008. Distinction of energy efficiency improvement measures by type of appropriate evaluation method. Final Report on Evaluation and Monitoring for the EU Directive on Energy End-Use Efficiency and Energy Services. EMEEES Project report.

Relevant case studies

• EPATEE Case Study “[UNITED KINGDOM] Supplier Obligations”