09 – Evaluating financial incentives for energy efficiency in residential buildings using deemed savings

The primary audience for this guide is energy efficiency programme designers, implementers or supervisors, and evaluators looking for guidance on the evaluation process of energy savings in the scope of this guide.

Although the application of the guide will generally concern the (sub)national level, account will be taken of issues at EU level when relevant (e.g. the specific format of saving figures for the EED, and more particularly its article 7).

This guide is not about the preceding step in the evaluation process, the choice of the method. About this previous step in the evaluation process, see the guidance provided here. However, after presenting the capabilities and limitations of the guide at hand, the user will be offered alternatives for the method within this guide (see section 6).

The objective of this guide is to provide:

• Information on the scope of the guide that enables the user to decide whether this guide is suited to his/her needs, and whether complementary or additional method(s) could be needed or useful (section 2);
• Guidance about specifying the evaluation objectives and requirements (section 3);
• Guidance about key methodological choices to calculate energy savings (section 4);
• Guidance about the inputs (data requirements) and outputs of the method (energy savings metrics) (section 5);
• Possible alternative methods (with pros and cons) (section 6)
• Background about evaluation results other than energy savings (section 7);
• Relevant examples, case studies and/or good practices (section 8);
• Relevant references for further reading (section 9).

The guide is intended for assessing realised (ex-post) energy savings. However, account is taken of earlier (ex-ante) evaluations of expected savings, if available (see section 4).

The focus of the guide is on impact evaluation, i.e. determining the energy savings, but not on how this has been reached through a step by step process with intermediate results (process evaluation).

Readers looking for the basic and general principles of energy efficiency evaluation may find the following link useful.

2.1 About financial incentives

Financial incentives can take various forms: direct non-refundable aids (grants, subsidies, tax credits), loans with low interest rates, financial guarantees on loans, etc.

More information and examples on the different subtypes residing under the main type (financial incentives) can be found here and here.

Analysis of financial incentives for energy efficiency in buildings can for example be found in (Federici et al., 2018; Maki et al., 2016; Olubunmi et al., 2016; Studer and Rieder, 2019).

The focus of this guidance is restricted to the most common types of financial incentives offered to households or other owners of residential buildings: direct non-refundable aids (grants and subsidies) and loans with low interest rates. However, most of this guidance can also be applicable to the evaluation of schemes using other types of financial incentives (e.g. financial guarantees, third-party financing).

The incentives can be defined as a fixed amount (lump sum) or as a percentage of the investment costs. The incentive amount or rate can depend on the type of action or on the level of energy performance achieved (the higher the performance, the higher the incentive). The base to calculate the financial incentives may make a distinction between labour costs and costs for materials or products. The scope of costs taken into account is particularly important when assessing the cost-effectiveness of the incentive scheme (see section 7).

Limiting the risks of inflation effect is often critical in the design of the incentives (which can therefore be a subject of evaluation).

The incentives can also depend on the income level or other criteria (age, occupation, etc). related to the dwelling owner or occupant. This is especially the case when the objectives of the incentive scheme include alleviating energy poverty. When the design of the scheme considers income levels or social criteria, the evaluation objectives often include the analysis or assessment of distributional effects or effects on energy poverty.

Schemes offering financial incentives often represent significant budget commitments for the state, regional authorities or other public bodies that decide about funding the schemes. These schemes are often implemented by a mandated body (also named “entrusted party” or “implementing public authority” in the EED). This managing body usually monitors the scheme (e.g. number of participants, amounts of funding provided, number and types of actions funded) and reports to the government or other public institutions (e.g. Parliament, Court of Auditors) about the results of the scheme. Cost-effectiveness indicators can thus be given a high importance in the reporting needs.

More detailed information on the evaluation of financial incentives can be found here.

2.2 Evaluation for a combination of policy measure types

When financial incentives are combined with other types of policy measures (e.g. tailored energy advice, training programmes for building professionals), it is assumed here that the overall savings due to energy efficiency improvements in buildings are mainly resulting from the financial incentives. However, in practice, the evaluation often encompasses the combined savings effect of all policy measures promoting the implementation of the types of actions in the scope of the evaluation.

This guidance is not capable of attributing part of the (overall) calculated savings to each of the policy measures (see also Double counting in the section on Gross to Net savings).

For discussions about policy mix or comparisons of different types of policy measures, see for example (Boza-Kiss et al., 2013; Murphy, 2014).

2.3 Evaluation when combined with energy taxes

The calculated savings effect for financial incentives will overlap with that of the energy tax. The energy tax indeed increases the cost-effectiveness of the energy saving actions from the point of view of the investors (most often the building owners). Therefore, both policy measures, the financial incentives and the energy tax, improve the economic conditions for the investments in energy saving actions. The financial incentives will most often be equivalent to reducing the investment cost. While the energy tax will increase the economic benefit from the actions (by increasing the economic value of the energy savings), thereby reducing the payback time.

This guidance is not capable of attributing part of the (overall) calculated savings to either the incentive scheme or the energy tax.

Distinguishing the effects of financial incentives and energy taxes can be investigated with modelling of investment behaviours, by simulating scenarios with different policy packages. Calibrating this type of model usually requires data about the number of actions implemented in the country (or other area considered for the evaluation).

This type of analysis can be complemented with surveys or modelling about the willingness to pay for energy saving actions (see for example, Collins and Curtis, 2018). Such studies can be helpful for the design of the financial incentives (e.g. identifying the level of incentive that can trigger actions) and to provide a basis to evaluate the additionality of the financial incentives (or the free-rider effects) (see also section 4).

2.4 About existing residential buildings

This guidance is focused on energy saving actions for improving the energy efficiency of existing dwellings. However, most of it is also applicable to the case of new buildings. The main difference lies in the types of baseline that can be used. When using deemed savings for new buildings, the choice of baseline is usually equivalent to the option “minimum energy performance standards” (see Calculation baselines in section 4).

This guidance covers both, individual houses and multifamily buildings. In practice, the incentive schemes can be focused on specific segments of the dwellings stock. For example, to tailor the incentive scheme in order to tackle specific barriers, such as decision processes in co-owned buildings (or condominiums) or split incentives in the case of the rental sector. This guidance does not consider possible differences in incentives according to the types of buildings. This can however affect the way to analyse additionality or free-rider effects, when evaluating additional or net energy savings (see also section 4).

Moreover, when a scheme covers different types of buildings, it is recommended to define different deemed savings for each main type of buildings. Energy savings from the same action types can indeed vary significantly depending on whether they are installed in individual houses or multifamily buildings.

In terms of end-uses, this guidance deals with the thermal end-uses, i.e. HVAC (heating, ventilation and air conditioning), and domestic hot water. Energy saving actions in the scope of this guidance thus includes actions improving the energy performance of the building envelope (e.g., wall insulation, roof insulation, windows replacement), heating and cooling systems (e.g., boiler replacement, hydraulic balancing of the heat distribution system), ventilation systems (e.g. heat recovery ventilation) or the system for domestic hot water.

The guidance does not deal with actions on electrical end-uses, such as appliances and lighting. About savings from financial incentives for appliances, see Specific Guidance 14 .

Financial incentives for the energy renovation of residential buildings can be to encourage single actions (e.g. wall/roof/floor insulation, boiler replacement) or deep renovations (i.e. aiming at improving the global energy performance of the buildings). In most cases, the incentives are conditional upon minimum energy performance requirements. The incentive scheme may also require other conditions, such as quality label or installation by a qualified building professional.

In practice, the use of deemed savings is preferred when evaluating policies or programmes promoting single actions or predefined packages of actions (see for example, the EPATEE case study about Better Energy Homes in Ireland: Broc, 2017). Deemed savings can also be used for schemes promoting overall improvements of energy performance, as a default option or when rough estimates are enough according to the evaluation objectives (see for example, the EPATEE case study about the subsidy scheme for energy savings in social housing of Amsterdam: Meulen and Menkveld, 2018; Niessink, 2018). See section 6 for a comparison with alternative methods.

Information on (sub)sectors defined in the Toolbox can be found here, chapter 2, p.17.

2.5 Evaluation for cross-sector saving actions

Most of this guidance can also apply in principle to schemes promoting the renovation of tertiary buildings (service sector). One major difference is that tertiary buildings can be much more diverse than residential buildings, as it encompasses many sub-sectors (e.g., offices, hospitals, stores, schools).

Deemed savings are therefore less commonly used for tertiary buildings. Examples of deemed savings for tertiary buildings can be found in the French white certificates scheme.

At the opposite, it is more common for tertiary buildings than for residential buildings to have their HVAC systems operated with BAS (Building Automation System) that can provide detailed data of energy consumption together with data for other relevant variables such as temperature setpoint or patterns of use of the buildings (e.g. working days vs. week-ends). For more guidance about cases where data from BAS can be used, see for example (Romberger, J. 2017). And more generally for more guidance about cases when data can be collected through an energy management system, see for example (Stewart, 2017).

2.6 About deemed savings

The Energy Efficiency Directive (2018(2002), Annex V(1.a)) defines “deemed savings” as “results of previous independently monitored energy improvements in similar installations”. They are also often called ex-ante or stipulated savings.

In practice, deemed savings are the result of standardised calculations where energy savings ratios (e.g. savings per action or savings per m²) are defined beforehand. The units can be for example the number of actions implemented the areas of insulation materials installed, etc.

Deemed savings are thus unitary savings. The number of actions (or related units) needs to be obtained by another method (see next section).

The standardised energy savings ratios can be defined in various ways. Typical approaches can be:

• Defining a standardised calculation formula for each action type covered by the incentive scheme, and then setting values for each parameter according to reference values (e.g. national statistics on the building stock) or results from previous studies on the same type of action.
• Defining a typical building (including average size, typical shape, etc.) for each category of buildings covered by the incentive scheme, and the calculating energy savings for the different eligible action types as simulated for these typical buildings with a building energy model (for example the official model used to issue Energy Performance Certificates). (see for example, the EPATEE case study about Better Energy Homes in Ireland: Broc, 2017)

Another approach can be to define energy savings ratios per given changes in energy class (according to the classes used for Energy Performance Certificates), as illustrated in the EPATEE case study about the subsidy scheme for energy savings in social housing of Amsterdam (Meulen and Menkveld, 2018). However, when data about Energy Performance Certificates (EPC) are available, it is preferred to use the data specific to each EPC, rather than average energy savings ratios, as the latter option increases the uncertainties (as discussed in Niessink, 2018 about the same case study).

Deemed savings are associated to standardised types of actions. This usually includes detailed specifications of the action types to ensure that deemed savings are used to count savings from actions that can be considered similar.

A set of deemed savings can also be defined for the same action type, to take into account variations due to parameters that can easily be monitored. For example, different values for deemed savings of insulation of walls can be defined according to climate zones, building types (individual houses / multifamily buildings), etc. In that case, the monitoring of these parameters should be ensured, in addition to the number of action (for example by collecting these data through the application files for the incentives).

As the main objective of using deemed savings is to keep monitoring and evaluation requirements as simple as possible, the number of data to monitor is usually limited to a few parameters that can be documented easily (e.g. building type) or is needed to be registered anyway (e.g. location of the building). However, this does not mean that the calculation made to define the deemed savings is simplistic. Depending on the data available from general references and other studies, and on the means and time available, deemed savings can be defined from simple experts’ estimates up to sophisticated calculations including statistical analysis (for example about the building stock or previous market trends).

Due to the way they are defined, deemed savings do not reflect the energy savings that are achieved for a given situation, but an average result for a typical situation or a large population of similar actions. Whenever possible, it is recommended to test and compare different approaches to define the deemed savings, to assess the robustness of the energy savings ratios and their sensitivity to key parameters. Sensitivity analysis can for example be used to select the most relevant parameters to differentiate deemed savings.

Likewise, it is recommended to update the deemed savings on a regular basis, for example to take into account new data available (e.g. updated national statistics), changes in regulations (especially when used to set the baseline), results from ex-post studies, etc.

Information about the various evaluation methods can be found here, table 1 and 2. This source also covers the combination of the method at hand with other methods, which will be dealt with below.

2.7 Complementary methods to determine total savings

Complementary methods are methods that are required, in addition to the primary selected method (here deemed savings), to calculate total energy savings.

Deemed savings are meant to calculate unitary savings. In case of actions in the residential sector, the unit can be a participant (household), an equipment (e.g. boiler), a dwelling, a m² of area insulated, etc.

These unitary savings then need to be multiplied by the number of actions or participants in order to have the calculated total savings. The number of actions or participants can be obtained in various ways. See this link, table 2 and 3.

In the case of an incentive scheme, the number of actions is usually directly available from the monitoring system, as the participants need to submit an application file, or at least to provide a minimum set of information, in order to receive the financial incentive.

In most cases, the evaluation of the scheme will make use of a set of deemed savings, defined according to action type, and possibly further differentiated according to key parameters (e.g. dwelling type, climate zone, type of heating system). In this case, the data needed to calculate the total savings go beyond the number of actions or participants, and include the variables used to differentiate the deemed savings. It is therefore recommended to ensure as much as possible that the monitoring system will enable a systematic collection of these data.

This guidance does not deal with monitoring systems. For more details about the links between monitoring and evaluation, see the topical EPATEE case study about the links between monitoring and evaluation (Maric et al., 2018).

When it is not possible to collect the data along the implementation of the scheme (e.g. if requiring data from participants could decrease participation rate or significantly increase administration costs), then an alternative is to collect the data by surveying samples of participants. Particular care should then be taken in the sampling method, so that the results can be extrapolated from the samples to the whole participants. For more details about surveys, see for example (Baumgartner, 2017). And about sampling, see for example (Khawaja et al., 2017).

2.8 Additional methods to increase reliability of the results

An additional method can be applied on top of using deemed savings to improve the reliability of the evaluation results and/or the cost-effectiveness of the evaluation approach.

Deemed savings are the result of a standardised assessment, based on data available from general references (e.g. national statistics, analysis on a typical case) or studies about actions previously implemented. Therefore, they are not reflecting the possible particularities of the actions installed in the participants’ dwellings.

This means that deemed savings do not necessarily reflect the actual energy savings (i.e. as could be experienced by a given participant). Unless further analysis is done, the related uncertainty is unknown.

Depending on the evaluation objectives, other methods can be used to investigate the reliability of the deemed savings (e.g. verifying the relevance of the data or assumptions used) or assess actual energy savings, as for example illustrated in the table below.

Type of method	Short description	Objective
Survey of participants	Surveys can be used to collect data complementary to the ones collected by the monitoring system (e.g. about characteristics of the households, buildings, etc.) For more details, see for example (Baumgartner, 2017)	To verify if the average values and assumption used in the deemed savings are representative of the participants or actions reported
Direct measurement	Direct measurement can be done on a specific parameter (e.g. sensors to monitor duration of use), or on the energy consumed by a given equipment or process (e.g. sub-metering). For more details, see e.g. (Mort, 2017)	To verify if the average values and assumption used in the deemed savings reflect actual conditions (e.g. energy performance, duration of use, indoor temperature)
Billing analysis	Calculation of energy savings from metered data of energy bills. For more details, see Specific Guidance 8	To assess actual energy savings (as can be experienced by the end-users)
Engineering estimates	Calculation of energy savings, based on engineering model or formula, and using data specific to the cases evaluated. For more details, see Specific Guidance 30	To test the robustness of the deemed savings

As the number of actions or participants monitored for an incentive scheme is usually very large, this could be costly to apply these additional methods to the whole actions or participants. Therefore, a common practice is to apply these methods on samples of actions or participants. As sampling can create bias and uncertainties in the results, a particular attention should be paid to limit sampling bias and obtain samples as representative as possible. For more details about sampling, see for example (Khawaja et al., 2017).

Combining different methods can be a cost-effective approach to increase the reliability of the savings figures and to get a better understanding of the impacts of the energy saving actions and the incentive scheme.

For possible combinations with an additional method see also chapter 6 here.

3.1 Meeting evaluation goals and ambition

Typical objectives of using deemed savings can be:

• Getting savings estimates on an almost on-going basis (no time-lag);
• Keeping the monitoring and evaluation system simple, and minimizing related running cost;
• Providing visibility to market actors.

At the opposite, deemed savings are not the most appropriate method in case the primary evaluation objective is to assess actual energy savings.

The table shows whether this guide can be used to report on general evaluation goals or criteria. See also this document.

General types of evaluation goals or criteria	Level of ambition	Remarks
Calculation of realized energy savings from saving actions	Low to medium	Depending on the quality of the deemed savings. In any case, complementing deemed savings with a method to verify actual energy savings is recommended for this objective (see above Additional methods to increase reliability of the results in section 2) Deemed savings should not be used to estimate savings for only one or a small number of actions. Deemed savings should be used to estimate savings for a large number of actions or participants.
Calculation of energy savings attributed to the policy measure(s)	Low to medium	Additionality criteria or default adjustment factors can be used to make that deemed savings correspond to additional or net savings. It is recommended for this objective to complement deemed savings with further ex-post analysis (see below Calculating Gross and Net energy savings in section 4)
Cost-effectiveness of saving action (for end-users)	Low	Deemed savings do not necessarily reflect energy savings as experienced by a given end-user. They represent average values. So they should only be used to estimate cost-effectiveness indicators on average. When the objective is to get more detailed and accurate results about cost-effectiveness from the end-users’ perspective, it is recommended to consider further ex-post analysis and using other methods (e.g. billing analysis or engineering estimates, see section 6).

3.2 Reporting expectations

Deemed savings are mostly used to evaluate energy savings at the policy level. As explained above in section 2 (see About deemed savings), deemed savings are not appropriate to provide the result specific to a given participant or action. Such specific reporting needs are more related to M&V than to policy evaluation. For more details about this, see for example the option C of the IPMVP (EVO, 2016), or (Jacobson, 2017).

This guidance deals with the evaluation of energy savings from an incentive scheme, i.e. a policy measure. In this case, deemed savings are used to provide results for a group of participants, and whenever possible for the whole participants.

Deemed savings can be defined in a variety of energy savings (or even CO2 savings) metrics. The choice of the metric(s) will mostly depend on the evaluation objectives and the data available to define the deemed savings.

The table below discusses possible options of metrics.

Criteria	Common options	Remarks
Nature of the objective	Energy savings CO2 savings Bill savings	Choice mostly depending on the primary objective of the incentive scheme. The results can be expressed in several metrics when needed. (for more details about CO2 savings see section 7)
Duration for which the results are counted	Annual (or first-year) Lifetime cumulated Cumulative over the obligation period	Choice depending on the objectives of the incentive scheme, and on the diversity of actions possible (cf. heterogeneity or not in action lifetimes). A “lifetime cumulated” unit can be chosen to value long-lifetime actions that are often the priority when dealing with energy efficiency in buildings. However, this requires to estimate energy savings over long durations, thereby increasing uncertainties (for more details, see Energy savings over time in section 5).
Energy basis (if nature = energy savings)	Primary energy savings Final energy savings	Choice depending on the objectives and scope of the incentive scheme. For example, primary energy savings can be chosen if the scope includes actions dealing with energy switching (change in the energy source), or if the priority is on electricity savings. Final energy savings can be chosen for example if the focus is on reducing end-users’ energy bills (e.g. to alleviate energy poverty). About specific data needs related to assessing final or primary energy savings, see section 5.
Energy unit (if nature = energy savings)	PJ / ktoe / TWh / …	Choice usually depending on the energy unit commonly used in the country (e.g. for the national energy balance), or on the objectives or scope of the incentive scheme. For example, ktoe can be used if the priority is to save fossil fuels. TWh can be used if the priority is on electricity savings.
Evaluation perspective	Gross / Additional / Net	For more explanations, see Calculating Gross and Net energy savings in section 4.

Deemed savings can be first expressed in the metric chosen according to the main policy objective, and then converted in other metrics for other reporting purposes (e.g. reporting to European Commission for the article 7 of the EED), or for communicating to different audiences (e.g. bill savings might be more explicit when communicating to the general public).

Whatever the energy savings metrics used, it is important to keep the documentation of the deemed savings, as well as the distribution of the total energy savings according to the different types of actions (and other key parameters, when relevant). These details will indeed be needed to convert the results from one metric to the other. Guidance and examples of template to document deemed savings can be found in the standard ISO 50046.

3.3 Time frame for evaluation

The period under evaluation depends on the evaluation objectives and on the current context of the incentive scheme. If the financial incentives have been changed (e.g. changes in incentive rates, in the list of eligible actions) at different points in time, then it can be interesting to evaluate separately each period between changes. The period under evaluation can also be defined according to the budget years of the scheme, for example when results need to be reported to the Parliament or to the Court of Auditors.

One of the main advantages of deemed savings is that it enables to assess energy savings as soon as the data about the numbers (or other units) of actions per action type are available from the monitoring system. This is one of the reasons why deemed savings are often chosen when the objective is to evaluate results on a regular basis (e.g. each year) or to get a quick feedback loop to detect if adaptations are needed.

It should be noted however that the definition of a set (or catalogue) of deemed savings requires time, as well as updates. The time and resources needed to develop (or update) a catalogue of deemed savings depend on the evaluation objectives (about the accuracy of the results), the number and diversity of the action types to be included in the catalogue, data availability, whether a consultation is included in the process, the level of consensus (or disagreement) about the data and calculation formula, etc.

Depending on the cases, developing a set of deemed savings can take from a few days up to several months.

The update of deemed savings can be done on a regular or case-by-case basis. It is usually made to take into account changes in the context that can affect either the definition of the baseline or of the energy efficiency action. For example, update of the regulations setting minimum energy performance requirements, revised national statistics, changes in market average, availability of a new product or solution with higher energy performance.

The use of deemed savings can be complemented by other methods to verify their reliability, and possibly update them (see Additional methods to increase reliability of the results in section 2). These additional methods have different timeframes. It is thus recommended to consider how the combination of evaluation activities should be planned. Especially to ensure the feasibility of the corresponding data collection and to optimize the use of resources (time and budget). (See also planning of evaluation in the link here).

3.4 Expertise needed for chosen method

The definition of deemed savings requires expertise about the following:

• Expertise about the energy efficiency actions eligible to the incentive scheme (e.g. about building components (building envelope) and HVAC (Heating, Ventilation and Air Conditioning) systems): this is needed to know what action types can be defined in a standardised way (and how), and to select the essential parameters to describe the action types and their energy saving effect.
• Expertise about data or trends in energy consumption in dwellings and related energy efficiency markets: this is needed to define the baseline.

When the monitoring system cannot provide all the data needed for the evaluation, then expertise about survey and sampling methods can be required to prepare the complementary data collection.

General principles of calculating realized savings using different methods can be found here and here.

This section deals with key methodological choices to be considered when calculating energy savings: consistency between ex-ante and ex-post evaluation, baseline, normalization and adjustment factors. These choices are important to document when reporting energy savings, to ensure the transparency of the results.

4.1 Matching method with earlier ex-ante evaluation

From the viewpoint of methodological consistency and data availability using the same method in the ex-ante evaluation and in the ex-post evaluation is recommended.

In the case of an incentive scheme, the ex-ante evaluation can take the form of an impact assessment of the scheme, including assumptions about participation rates. An impact assessment can indeed be required by law for any scheme committing public budget (possibly depending on a threshold of budget per year). More generally, an impact assessment is commonly used to size the scheme, and particularly to define quantitative objectives and estimate the budget needed. Whenever possible, the impact assessment can include several scenarios to compare impacts according to distinct alternatives (e.g. in terms of incentive rates, list of eligible actions, eligibility criteria for participants, etc.).

The ex-post evaluation will then aim at assessing if the objectives have been met, if differences occurred in the mix of action types installed by the participants (compared to the expectations), etc.

The same deemed savings can be used for both, the ex-ante and the ex-post evaluation. Respectively to set the objectives and to verify their achievement. When it is not the case, it is important to use the same principles when defining the baseline(s).

If deemed savings do not provide an acceptable combination with the method applied earlier for the ex-ante evaluation, it might be useful to select another method (see examples of alternative methods in section 6).

In practice, ex-ante and ex-post evaluations are applied consecutively. The ex-post evaluation builds on an ex-ante evaluation that made use of data coming from previous ex-post evaluation or studies (e.g. about previous periods of the same incentive scheme, or about the same types of energy saving actions as the ones promoted by a new policy measure). These previous ex-post studies could have used another type of method as well.

As discussed in section 2 (see Additional methods to increase reliability of the results), additional method(s) can be used to verify ex-post the energy savings initially estimated with deemed savings. In that case, the results from these ex-post verifications can be used to improve the deemed savings for future ex-ante or ex-post evaluations. When doing so, the representativeness of the results from ex-post verifications should be analysed to see if these results can be used for updating the deemed savings. It is important here to distinguish ex-post verifications done as part of controls (e.g. to mitigate risks of frauds), and ex-post verifications done as part of an evaluation.

Controls are commonly decided on a risk-based approach, in order to focus the efforts of control where the risks are higher or more critical. Therefore, results from controls are not meant to be representative. At the opposite, evaluation methodologies are usually designed to look at the whole scheme, taking into account representativeness, risks of sample bias, etc. Depending on the evaluation objectives, evaluation results can thus be expected to be representative. This is why data from verifications done as part of a M&V system might not be usable for an ex-post evaluation (or under certain limitations).

For a general discussion about possible combinations of methods applied ex-ante and ex-post, see chapter 7, in this document.

4.2 Calculation baselines

Gross energy savings are defined in general as the difference between the situation including the implementation of energy saving actions and a reference situation without the saving actions. This reference situation can be defined using various calculation baselines. For further background see further here.

When evaluating the energy savings from an incentive scheme, the baseline can also be defined to represent what the situation would have been in the absence of the incentive scheme. This is in particular linked to the concept of additionality, as defined in the amended Energy Efficiency Directive (EU2018 (2002)):

“To determine the savings that can be claimed as additional, Member States shall have regard to how energy use and demand would evolve in the absence of the policy measure in question by taking into account at least the following factors: energy consumption trends, changes in consumer behaviour, technological progress and changes caused by other measures implemented at Union and national level” (Energy Efficiency Directive 2018(2002), Annex V(2) point (a)).

Depending on the baseline option chosen, the energy savings can therefore mean to be gross or additional savings. Then gross energy savings estimated with deemed savings can also serve as basis to evaluate net energy savings, when combined with further analysis (see below Calculating Gross and Net energy savings).

The table below specifies what type of energy savings can be obtained according to the baseline option.

Moreover, when using deemed savings, it should be reminded that the calculated energy savings do not reflect the energy savings that are achieved for a given situation, but an average result for a typical situation or a large population of similar actions. This also means that the baseline is not defined according to a particular participant or building, but according to reference values (e.g. national statistics, modelling of typical dwellings assumed to be representative of the dwelling stock).

Baseline option	Explanation	Type of calculated energy savings
Stock average	This baseline corresponds to a before/after comparison. When using deemed savings, the before situation is not defined specifically as the before energy consumption of each participant or dwelling, but as average (or typical) before energy consumption per main type of dwelling (possibly also differentiating according to key parameters such as climate zone, type of heating system, etc.).	Gross energy savings   Additional or net energy savings can then be calculated by applying adjustment factors (see below).
Market average	This baseline corresponds to a with/without comparison, where it is assumed that the market average represents the counterfactual (what the participants would have done in the absence of the incentive scheme).   Usually the market average first provides the baseline level of energy performance for the type of product or equipment installed by the participants. This parameter is then combined with other data (e.g. reference values for dwelling characteristics) to calculate the baseline energy consumption.	Additional energy savings (i.e. savings additional compared to a business-as-usual scenario represented by the market average)
Trend scenario	This baseline corresponds to a with/without comparison, where the counterfactual is estimated based on extrapolations of previous about the energy saving actions promoted by the incentive scheme trends (e.g. number of renovation works and related performance levels), or about energy consumption of the stock of dwellings targeted by the scheme.   The trend scenario then provides either baseline levels of energy performance per action type, or baseline energy consumption per dwelling type.	Additional energy savings (i.e. savings additional compared to a business-as-usual scenario based on extrapolation of previous trends)
Minimum efficiency standards	This baseline corresponds to a particular type of with/without comparison, commonly used to assess savings additional compared to current regulations.   The minimum efficiency standards usually provide the baseline level of energy performance either for the type of product or equipment installed by the participants, or for the overall energy performance of the dwelling (e.g. in kWh/m².year). This parameter is then combined with other data (e.g. reference values for dwelling characteristics) to calculate the baseline energy consumption.	Additional energy savings (i.e. savings additional compared to current regulations)

The option “market average” is more commonly used when the scheme promotes single actions (or predefined action packages). It cannot be used for schemes promoting deep renovations. In this case, the option “trend scenario” can be an alternative, especially when labels about energy efficiency renovations are in place and can provide a proxy to estimate the trend in deep renovations.

Deemed savings can also be defined directly from energy savings ratios assessed in previous ex-post evaluations using other methods. In that case, the baseline option is the one that was used in the previous ex-post evaluation and can include other types of baseline option (e.g. a control or comparison group).

A particular case can be when a pilot phase has been used to specify the incentive scheme, and that it was possible to make an ex-post evaluation of the energy savings during this pilot phase. As recommended by the E2e initiative (Evidence for Action on Energy Efficiency) in the US, it can then be possible to use a RCT (Randomised Controlled Trials) method to evaluate net energy savings. The RCT method uses a comparison between a control group (not subject to the pilot scheme) and a participants’ group (subject to the pilot scheme), where individuals (e.g. building owners or occupants) are taken randomly in the population and randomly assigned to one of the two groups. The population means here all the persons who will be eligible to the scheme once it will be implemented in full scale.

The RCT method usually calculates net energy savings, as the difference observed between the two groups in their changes in energy consumption (method also named Difference-in-Differences). The baseline is then a combination of before situation and control group.

For more details about the RCT method, see the Specific Guidance 8 .

If this approach has been used to define the deemed savings, then the ex-post evaluation (done after the scheme is implemented on full scale) can be used to investigate the possible differences between the pilot phase and the full-scale phase of the scheme, to analyse if the deemed savings defined from the pilot phase are still relevant to evaluate savings from the full-scale phase.

For a general discussion about calculation baselines, see the link to this document.

4.3 Normalization factors

The calculation of deemed savings is usually made based on normalized (or conventional) conditions of use. The types of energy saving actions considered in this guidance are related to space heating or cooling. This means that the calculation is made taking into account normalized weather conditions (e.g. reference Heating Degree Days, and possibly Cooling Degree Days, as defined in building regulations).

Deemed savings can be the result of a simplified engineering calculations, such as the ones used for Energy Performance Certificates. In that case, the normalized weather conditions are an input data of the calculation (even most commonly directly integrated in the calculation model). When using this approach, another input of the calculation will be normalized assumptions about heating behaviours (e.g. indoor temperature per type of rooms, night and absence mode). This means that the deemed savings will be based on conventional energy consumption.

In that case, it is recommended to consider if it would be relevant to apply factors to take into account prebound and rebound effects:

• prebound effect: lower baseline energy consumption compared to the estimated conventional energy consumption. This can occur for example in very inefficient dwellings, for which a normalized use of space heating would not be affordable for the occupants. This can also occur when the statistics about the building stock do not take into account the fact that a share of the dwellings can have been renovated since their construction.
• rebound effect: higher energy consumption than the estimated conventional energy consumption for the situation after implementing the energy saving action(s). This can occur for example when the energy efficiency improvement makes affordable for the occupants to increase the indoor temperature, heat more rooms, or heat for longer periods.

Factors to take into account prebound effect can for example be derived from comparisons between conventional energy consumption from Energy Performance Certificates and metered energy consumption. Prebound effect can also be considered in the calibration of the calculation model used to define the deemed savings.

Factors to take into account rebound effect can for example be derived from previous surveys, measurement campaigns or the literature.

For more details about prebound and rebound effects, see for example:

• the topical case study about the comparison between estimated and measured energy savings (Sipma et al., 2019);
• the experience sharing webinar about “How and what can we learn from verifying energy savings first estimated with engineering calculations?”;
• examples of studies who investigated these effects (Hong et al., 2009; Milne and Boardman, 2000; Raynaud et al., 2014).

It should be noted that prebound and rebound effects can be analysed with different perspectives according to the main policy objectives. For example, if the main objective is to reduce the absolute energy consumption in dwellings, then rebound effect will be perceived as a negative effect. At the opposite, if the main objective is to alleviate energy poverty, then part or all of the rebound effect can be perceived positively, as an improvement of the thermal comfort for the occupants. Which can then be connected to co-benefits such as improved indoor air quality and sanitary conditions, and thereby reductions in health expenses.

About the energy consumption for the situation after installing the energy saving action(s), another factor might need to be taken into account: possible performance gaps.

Performance gaps correspond to cases where the observed energy performance of the energy saving action(s) installed is lower than the expected energy performance. For example about insulation actions, because the insulation materials have not been installed properly (e.g. leaving thermal bridges) or the insulation materials have lower performance than claimed. About condensing boilers, an example can be when settings and conditions of use (e.g. types of radiators and related temperature for hot water) do not enable the boiler to condense and recover the heat from steam water in flue gas.

Enforcing quality requirements can help to minimize the risks of performance gaps.

Factors to take into account: performance gaps can for example be derived from previous studies on samples of actions.

An example of use of correction factors to take into account rebound effect and performance gaps can be found in the case study about the UK Supplier Obligation; see in this file the explanations about the “in-use factor”).

Deemed savings can also be based on statistics of metered energy consumption. In that case, time series long enough (e.g. 10 years) can be used to ensure that the data reflect usual weather conditions. When only short time series are available, then the weather conditions of the years included in the time series can be compared with normalized weather conditions (e.g. from building regulations). When needed, weather corrections can then be applied (e.g. using a weather factors based on Heating Degree Days, or making regression analysis).

If the deemed savings are based on statistics of metered energy consumption, then possible prebound or rebound effects or performance gaps are directly included in the metered data (no need for further corrections).

4.4 Adjustment factors

Adjustment factors define which part of the calculated energy savings can be attributed to a policy measure or meets the definition of savings specified in the evaluation objectives or reporting requirements (see next section on “Calculating Gross and net savings).

For a general introduction about adjustment factors, see table 1 here.

When using deemed savings, adjustment factors can concern the Free rider effect and the Spill-over/multiplier effect. For related definitions, see this document. Free-rider and spill-over effects can also be encompassed in a single additionality factor.

These adjustment factors are used when the evaluation objective is to assess net energy savings. In that case, they mostly apply when the chosen baseline option is “stock average”, or other similar options used for a before/after comparison.

They can also be applied when another baseline option has been chosen. But then, they should be defined to avoid overlaps with the effects already taken into account in the definition of the baseline. For example, if the baseline option is “minimum efficiency standards”, then the scope of free-rider effect taken into account in the adjustment factor will be focused on the number of actions, and will not deal with the performance level (as this aspect is already assumed to be taken into account in the baseline).

In any case, these effects can only be taken into account in deemed savings by including factors defined from previous studies or surveys. For an example, see the EPATEE case study about Danish EEO scheme.

It should be noted that in most of the available experience (especially in Europe), adjustment factors have been defined to take into account free-rider effect only. Spill-over effects have indeed proven to be more difficult to assess quantitatively.

Adjustment factors can also be applied afterwards, when the ex-post evaluation includes further analysis to assess them specifically for the period under evaluation (e.g. through dedicated surveys of participants and non-participants). Then other methods can be used. For more details, see the EPATEE topical case study about assessing net energy savings (Voswinkel et al., 2018).

In case another policy can overlap with the incentive scheme (promoting the same types of energy saving actions), then it is needed to consider the risks of double counting. Methods to tackle double counting or interaction between policy measures go beyond the scope of this guidance.

4.5 Calculating Gross and Net energy savings

Gross energy savings are energy savings calculated from the point of view of the final consumers, i.e. independently of whether the participants to the incentive scheme would have acted the same or differently in the absence of the scheme.

Usually the baseline energy consumption used to calculate gross energy savings is the energy consumption before the energy saving actions were installed or implemented. When using deemed savings, data specific to each participant is often not available (to keep the monitoring simple). Therefore, the baseline energy consumption for deemed savings can be defined as the average energy consumption in the stock of dwellings.

The calculation of gross energy savings can include the use of normalization factors to ensure that the baseline energy consumption and the energy consumption with the energy saving action are comparable. For the residential sector, this mainly concerns weather conditions (see Normalization factors above).

Depending on the data and approach used to calculate gross savings, it can be relevant to use also correction factors for prebound and rebound effects, as well as performance gaps (see Normalization factors above). These effects indeed affect energy savings from the point of view of the final consumers.

Net energy savings are calculated from the point of view of the public authority or other stakeholder that provides funding to the incentive scheme. Therefore this calculation takes into account effects related to the causality or attribution of the actions or energy savings to the incentive scheme.

In the context of the EU Energy Efficiency Directive, it is also common to speak about additional savings instead of net savings, due to the terminology of the EED article 7. At national level, the concept of additional savings can also be used when the incentive scheme is designed to have effects additional to current regulations, or more generally to trigger savings that are additional compared to a baseline scenario defined to identify the needs in energy efficiency policies.

When using deemed savings, additionality has to be considered in the calculation assumptions, as deemed savings are defined before the actions are implemented. Two main approaches are thus possible:

• Take into account additionality in the definition of the baseline (option presented above in Calculation baselines).
• Apply adjustment factors to gross energy savings (option also presented in Adjustment factors).

Additional methods can then be used to assess net or additional savings ex-post (e.g. when evaluating the latest period of the incentive scheme). Explanations and guidance about methods for this type of ex-post evaluation can be found in the EPATEE topical case study about assessing net energy savings (Voswinkel et al., 2018).

Additional or net savings should also be corrected for double counting, in case of possible overlap between the incentive scheme and other policy measures. The overlap in the calculated savings should be analysed at the level of the overall policy portfolio or sector. For addressing double counting see here and here.

5.1 Main data requirements and data sources and collection technics

Data requirements specified in the two tables below correspond to the calculation of gross energy savings, when using the baseline option [before/after], i.e. a baseline representing the situation before energy saving actions are implemented. The case about using baseline options to calculate additional energy savings is discussed later on (see Data issues when evaluating net energy savings).

The first table deals with the case where deemed savings are the result of simplified engineering calculations that estimate “before” and “after” energy consumption through intermediate parameters. This approach is commonly used for schemes promoting single actions on building envelope or HVAC (Heating, Ventilation and Air Conditioning) systems.

Calculation subject	Data requirements	Possible data sources and collection technics
Definition of segments or building types to represent the dwellings stock	Analysis of the main different segments or building types that need to be distinguished when defining the standardised actions and deemed savings	National statistics about the dwellings stock (e.g. national housing survey, national database of Energy Performance Certificates). History of main construction periods (taking into account main trends in construction practices, and successive building regulations) For more details, see for example: http://www.episcope.eu/building-typology/ http://webtool.building-typology.eu/
Energy performance of the building envelope (heat transfer coefficient: U-values in W/m².K; or thermal resistance: R-values in m².K/W)	Average values representative of the different segments / building types.	National statistics about the dwellings stock (e.g. national housing survey, national database of Energy Performance Certificates). Minimum energy performance requirements set in the successive building regulations, and that can then be used as a proxy for the corresponding construction periods.
Definition of main types to represent the stock of heating systems	Analysis of the shares in the dwelling stocks per type of energy used for space heating. Analysis of the main type(s) of heating systems per type of energy.	National statistics about the dwellings stock (e.g. national housing survey, national database of Energy Performance Certificates). National statistics on energy consumption in the residential sector (e.g. number of customers per energy supplier)
Efficiency of heating systems (per main types of heating systems)	Average values representative of the main types of heating systems (per energy type, and if possible per building type).	National statistics about the dwellings stock (e.g. national housing survey, national database of Energy Performance Certificates). Dedicated surveys or stock modelling.
Energy performance of the actions installed (U-values or R-values for insulation materials, efficiency for heating systems)	Possibility to define deemed savings for different levels of energy performance achieved	Minimum energy performance requirements set for the incentive scheme (eligibility criteria per action type). Average energy performance per class of performance (when deemed savings defined per class of performance).

The second table below deals with the case where deemed savings are directly based on data of energy consumption (without the use of intermediate parameters). The data may be available as unitary energy consumption (e.g., kWh/m².year). This approach is commonly used for schemes promoting deep renovations (i.e. aiming at overall energy performance improvements).

Calculation subject	Data requirements	Possible data sources and collection technics
Baseline energy consumption	Average energy consumption representative of the stock, possibly differentiated per main type of building or per energy class	Previous studies (e.g. surveys, stock modelling) National statistics of energy consumption in the dwellings’ stock
Energy consumption with the energy saving action	Possibility to define deemed savings for different levels of energy performance or energy class	Minimum energy performance requirements set for the incentive scheme (eligibility criteria per action type). Energy consumption per class of performance* (when deemed savings are defined per class of performance).
Dwellings’ area (needed when available data is about unitary energy consumption, e.g. in kWh/m².year)	Either average area per main type of dwelling Or data collected from each participant	National statistics about the dwellings stock (e.g. national housing survey, national database of Energy Performance Certificates). Dedicated surveys Monitoring system (collecting data from participants)

*: an energy class corresponds to a range of energy consumption. In practice, the distribution of energy consumption within an energy class can be asymmetric (for example, most of the products can tend to have an energy consumption close to the maximum in the range). Therefore, the average of the range is not necessarily representative. In case no data is available, a conservative assumption can be made to use the maximum energy consumption of each range to represent the different energy classes.

The third table deals with complementary data that can be needed whatever the approach chosen to define the deemed savings.

Calculation subject	Data requirements	Possible data sources and collection technics
Normalization factor for weather condition	Normalized Heating Degree Days (possibly differentiated per climate zone, when needed)	Heating Degree Days specified in building regulations. Data from the national meteorological office or weather service.
Correction factors (e.g. to take into account prebound and rebound effects, performance gaps)	To be considered if the calculation model could not be calibrated based on statistics of metered energy consumption.	Available studies on these effects. Data from the literature. Default values based one conservative assumptions.

Deemed savings are defined for a unit of action (see About deemed savings in section 2). Therefore, a complementary source of data is needed to obtain the number of actions (or other units of actions): see Complementary methods to determine total savings in section 2.

Data issues when evaluating net energy savings

In the context of the article 7 of the EU Energy Efficiency Directive, it is more common to speak of additional energy savings than net energy savings (see Calculating Gross and Net energy savings in section 4).

When using deemed savings, additionality has to be taken into account in the calculation assumptions, as deemed savings are defined before the actions are implemented. Two main approaches are thus possible:

1. Take into account additionality criteria in the definition of the baseline (option presented above in Calculation baselines).
2. Apply adjustment factors to gross energy savings (option also presented in Adjustment factors).

The first option will lead to use a baseline different from the “before/after” comparison (for more details, see Calculation baselines in section 4). This means the following data requirements:

• for the baseline option “market average”: data about trends or recent market shares per energy class (or similar categories reflecting energy performance levels) to calculate the market average for energy consumption or energy performance characteristics (in case energy consumption is calculated with intermediate parameters).
• for the baseline option “minimum efficiency standards”: data about the current minimum energy performance requirements set in national or European regulations, per action type (and information about the scheduled updates of these regulations).

The corresponding values might need to be updated on a regular basis or case-by-case (e.g. to take into account the most recent years to define the market average, or changes in the current regulations about minimum efficiency standards).

The baseline option for the second option is “before/after” comparison (see above). The data needed for the deemed savings to correspond to net or additional savings is then data to define the adjustment factors. Such data can be obtained from previous studies (e.g. previous surveys or market analysis) or the literature. As far as possible, it is recommended to use data from previous studies on the same policy measure. Literature indeed shows that values for adjustment factors can vary significantly from one policy measure to another, one country from another, etc.

Complementary ex-post evaluations can be used to investigate and update or revise the adjustment factors. For more details about the evaluation of net energy savings, see the corresponding EPATEE topical case study (Voswinkel et al., 2018).

5.2 Energy savings in final terms or in primary terms

Energy savings can be expressed in final terms or in primary terms. See definitions about primary and final energy here.

Usually deemed savings are first calculated in final terms. One advantage of choosing to calculate in final energy is that it enables to compare or define deemed savings (or estimates of energy consumption) with statistics of metered energy consumption (e.g. energy bills).

Deemed savings can also be calculated in primary terms, provided that deemed savings are calculated for each energy carrier apart, and primary factors are available to convert the savings in final terms to savings in primary terms.

In some cases, deemed savings can be directly calculated in primary terms, when the data available for defining the deemed savings are dealing with primary energy consumption. This can be for example the case in the building codes, depending on the country.

Deemed savings can also be calculated as monetary savings, i.e. savings on energy bills (see Reporting expectations in section 3). This requires to define average energy prices per energy carrier, as well as a scenario of energy prices if the savings are calculated in cumulative terms, e.g. over the action lifetime. In this case, a discount rate can also be applied to the calculation (see below Energy savings over time).

For consistency, the metrics should be the same for setting the objectives of the incentive scheme and then evaluating the achievements.

The energy savings results can then be expressed in other metrics for other purposes (e.g. reporting in the context of the EED article 7), provided that the data needed to convert from one metric to the other is available. It is thus important to identify from the outset the needs to express the results in different metrics, so that data used in the energy savings calculations are documented enough to enable future conversions.

In any case, the metric used to calculate the energy savings should be made explicit when presenting the results.

5.3 Energy savings over time

Implemented saving actions in a year lead to savings over a number of consecutive years. E.g. a more efficient boiler can save gas over its lifetime of about 15 years, and insulation over up to 60 years. For savings from behavioural changes (e.g. if the incentive scheme promotes energy display devices with tailored energy advice) might be not much longer than the period of the behavioural intervention (e.g. period over which tailored advice or feedback is provided to the occupants).

Energy savings can be calculated in different metrics in terms of time reference, for example: year-to-year, annual, cumulated annual, cumulative. See the definitions here.

The incentive scheme can count first-year savings only, or up to lifetime-cumulated savings (i.e. savings over the lifetime of the energy saving action).

If only first-year savings are counted and needed for reporting, then no further data is needed, apart from monitoring when the actions are installed (data usually collected as part of the process to pay the incentive).

In other cases, the data about when the actions are installed will need to be complemented with data about the estimated lifetime of the savings.

Examples of lifetime values can be found in the following sources:

• CWA 15693:2007. Saving lifetimes of Energy Efficiency Improvement Measures in bottom-up calculations. CEN Workshop Agreement, April 2007.
• EN 15459:2017. Energy performance of buildings — Economic evaluation procedure for energy systems in buildings. CEN standard, June 2017. (see annex D).
• Ecodesign Impact Accounting – Status Report 2017. Prepared by VHK for the European Commission December 2017. (see annex A, pp.73-76).

On top of cumulating savings over savings lifetime or crediting durations, the savings calculations can also include the application of a discount factor. This discount factor can be used:

• For economic reasons: for example when assessing cost-effectiveness (see also section 7), so that costs and benefits occurring over time can be taken into account according to a given investor’s perspective (i.e. a given preference given to present).
• For technical reasons: for example, to take into account that energy savings can decrease over time (e.g. for behavioural actions).

When using an economic discount factor, it is defined according to the investor’s perspective chosen (and not according to the action type). Discount factors are usually taken from the literature or from reference values (e.g. when considering the public authorities’ perspective). For more discussions about discount factors, see for example the Horizon 2020 projects Briskee and Cheetah.

When using technical discount factors, they might need to be differentiated per action type, as the changes in energy savings over time can depend on the action type. However, there is limited evidence about decrease (or increase) of energy savings over time, especially when dealing with renovations of buildings (for more details about this issue, see for example Hoffman et al., 2015 or McCoy and Kotsch, 2018).

A general default discount factor (i.e. uniform for all action types), encompassing both economic and technical aspects, can also be decided to reflect the risks related to the investment in the energy saving action, as perceived by the final customers or investors.

The choice of the metric depends on the policy or evaluation objectives and reporting needs. For example, when the main objective is to keep the evaluation simple, the choice can be yearly savings, as no further data or assumption is then needed. However, this will favour short-term actions against longer term action. Incentive schemes for energy efficiency in buildings usually aim at promoting actions with long lifetimes, such as wall/roof/floor insulation. Which can therefore support the choice of lifetime-cumulated savings. Lifetime-cumulated savings are also needed when assessing cost-effectiveness (see section 7).

The choice of the metric can also depend on reporting needs, for example in the context of EED article 7. In this framework, a particular type of cumulative savings is needed, counting the savings achieved within the obligation period (currently 2014-2020, and then 2021-2030).

As mentioned above about energy savings in final or primary energy terms, it is important to identify from the outset the needs to express the results in different metrics, so that data used in the energy savings calculations are documented enough to enable future conversions. When dealing with energy savings over time, this mostly means to keep records of the energy savings per main type of action as lifetimes can be different from one type to the other.

6.1 Alternatives for the chosen method

Deemed savings are often chosen when the objective is to keep the evaluation simple, to get a quick feedback loop (e.g. getting results’ estimates without delay) or to provide stakeholders with visibility (see Meeting evaluation goals and ambition in section 3).

Deemed savings are appropriate to evaluate savings from action types that can be described in a standardised way. They are then a cost-effective way to assess savings from large numbers of similar actions (e.g. case of an incentive scheme promoting single actions), without delay. They are less relevant for energy saving projects that would be very specific, thereby requiring case-by-case calculations. This is for example why other methods will usually be preferred when assessing energy savings from a scheme promoting deep renovations.

Engineering calculations are indeed more appropriate in case of large or specific energy saving projects, and can also provide results with limited time lag. Engineering calculations are for example often chosen to evaluate the savings from the incentive scheme when they are used anyway as part of the process of the application for the incentive (e.g. when an Energy Performance Certificate is required as part of the application file). In this case, there is no extra cost of using these data for the evaluation of the incentive scheme. For more details about engineering calculations, see Specific Guidance 15 .

Billing analysis is then commonly chosen when the evaluation objective is more specifically to assess or verify the energy savings actually achieved or to assess the cost-effectiveness or efficiency of the scheme. Billing analysis are particularly used in case of scheme promoting deep renovations. In such cases, the types of actions implemented are not necessarily monitored. This makes that methods based on what types of actions have been implemented (e.g. deemed savings or engineering methods) would be more difficult to use. Also, deep renovations imply that it can be important to take into account the interactions between the different actions (e.g. insulation of building envelope and replacement of heating system). The main drawback of billing analysis is the time lag between the installations of the actions and when the evaluation results can be available (usually from 18 to 30 months). For more details about billing analysis, see Specific Guidance 8 .

Direct measurements can also be required, especially for large projects in the industry (or large commercial buildings) (see e.g. the option of Monitoring and Measurement Plans in the Italian white certificates scheme).

Direct measurements are more rarely used for the evaluation of incentive scheme as considered in this guidance. They can for example be used as part of specific verifications or on-site inspections to complement another method.

The table below presents the pros and cons of these methods commonly used for evaluating energy savings from EEO scheme (see also Additional methods to increase reliability of the results in section 2).

Type of method	Pros	Cons
Deemed savings	·        Provide visibility to stakeholders ·        No delay in getting results from the monitoring system ·        Low running cost (once the set of deemed savings has been defined) ·        Calculations directly related to the energy saving actions installed	·        Use limited to action types that can be described in a standardised way ·        Do not reflect the energy savings achieved for a given situation, but an average result for a population of actions ·        Can require significant preliminary efforts (if many different action types included in the scope of the incentive scheme) ·        Quality depending on the data available to define deemed savings ·        Possible gaps between deemed savings and actual savings (due to prebound and rebound effects, and to performance gaps) ·        Additional method needed to evaluate ex-post the additionality of the savings
Engineering calculations (see Specific Guidance 15)	·        Can be used for almost all action types (including deep renovations) ·        Can enable to automatize energy savings calculations (through standardised formula for simple cases) ·        Can reflect the energy savings achieved for a given situation (specific calculations) ·        Limited delay in getting the results (calculations can be done before the actions are installed)	·        Require to collect data for each case (so can be costly if data collected only for this purpose and for large numbers of actions / projects) ·        Possible gaps between engineering estimates and measured savings (see references below) ·        Additional method needed to evaluate ex-post the additionality of the savings
Billing analysis (see Specific Guidance 8)	·        Provide data about actual energy consumption / energy savings (capturing prebound and rebound effects, as well as performance gaps) ·        Can be used to evaluate ex-post net savings (depending on the type of comparison group chosen, or whether adjustment factors can be assessed with an additional method)	·        Can only be used for ex-post evaluation ·        Frequent difficulties to collect billing data (unless data collection carefully planned and prepared in advance, e.g. collecting participants’ consent when actions are installed) ·        Difficulties to get representative samples (cf. sampling bias + data losses along the evaluation process) ·        Delays in getting the result (at least one year to get the consumption after installing actions + time to process and analyse data) ·        Difficulties to find relevant control or comparison groups (when assessing net or additional savings)
Direct measurements	·        Provide data about actual energy consumption (for the baseline and/or for the situation with energy saving actions) or about actual values for key parameters (e.g. indoor temperature) ·        Can be used to assess performance gaps	·        Can be costly if measurements only done for this purpose and for large numbers of actions ·        If sampling is used, attention should be paid to avoid sampling bias (if data are to be extrapolated) ·        Additional method needed to evaluate ex-post the additionality of the savings ·        Delay in installing the actions (if used to verify the baseline, then time needed to make the measurements, unless data are already available) ·        Delay in getting the results (if used to verify the situation with energy saving actions, then time needed to make measurements after the actions are installed + time to analyse the data)

7.1 Calculating avoided CO2 emissions

Depending on the priority objectives of the incentive scheme, the evaluation objectives can include the assessment of the results in terms of CO2 savings (i.e. avoided CO2 emissions).

In practice, deemed savings are first calculated in terms of energy savings. Then avoided CO2 emissions can be evaluated from the energy savings by applying emission factors. Four key aspects are to be taken into account when choosing the emission factor(s):

1. Emission factors vary according to the energy type, so the deemed savings need to be defined per energy type.
2. Emission factors for a given type of energy can vary over time (especially for electricity).
3. Emission factors can take into account:
   1. Direct emission factors: that take into account the emissions generated when producing the energy used;
   2. Lifecycle emission factors: that take into account all the emissions generated from the extraction of the energy resources up to the dismantling of the energy plant.

Due to the differences that the choice of emission factor(s) can induce, it is important to document what emission factor(s) has(have) been used.

Emission factors used for the incentive scheme can for example be based on official national emission factors used for the national inventory of emissions of greenhouse gases.

The conversion of electricity savings into CO2 savings is however a special case, depending on the national mix for electricity production. Several choices are indeed possible, for example:

• Average emission factor, calculated from the total annual emissions from electricity production (possibly taking into account national imports and exports) divided by the annual amount of electricity consumed: this is a simple approach, but that might not reflect the fact that end-uses can have different times of use and thus correspond to different load profiles (while the emission factor for electricity can vary significantly between base load and peak load).
• Emission factors per type of end-use: this requires more sophisticated calculations (e.g. by decomposing the national load curves per type of end-use) that will be meant to use emission factors reflecting the differences in time of use (e.g. daily, seasonally).

The choice between the two options above will depend on the national electricity mix (cf. emission factor varying significantly with time of production or not) and the type of end-uses covered by the incentive scheme (and especially if savings related to electric heating can represent a significant share of the savings).

If the deemed savings cumulate savings over time, it can also be needed to define a scenario about the evolution of the national electricity mix over the period of calculation (e.g. taking into account the objectives of shares of electricity produced from renewable energy sources).

The avoided emission of other greenhouse gasses due to energy savings are not taken into account here, as these emissions (and more specifically their reductions) are generally negligible compared to CO2 for actions in the residential sector.

When needed, IPCC (Intergovernmental Panel on Climate Change) provides a detailed database of peer-reviewed emission factors.

7.2 Calculating cost-effectiveness

Cost-effectiveness is the ratio between costs to achieve energy savings and the amount of savings and possibly other benefits.

A distinction can be made according to the point of view adopted to assess cost-effectiveness:

• Cost-effectiveness for the end-user or participant (e.g. payback time)
• Cost-effectiveness from the point of view of the public authority funding the incentive (e.g. comparing different types of policy measures according to the public cost of energy savings)
• Cost-effectiveness for society at large (e.g. social net present value)

For more details about the different perspectives, see for example (Breitschopf et al., 2018).

The table below summarizes the main costs and benefits taken into account according to the perspective adopted.

Point of view	Costs	Benefits
Participants	·         Part of the investments paid by the participants	·         Gross energy savings (note: the financial aids received from the incentive scheme is taken into account in the fact that only the part of the investments paid by the participants is included in the calculation)
Public authorities	·         Financial incentives paid to the participants ·         Administration costs of the scheme ·         Losses in tax revenues (due to additional energy savings)	·         Increases in tax revenues (due to additional investments made in energy saving actions vs. baseline scenario) ·         Net energy savings (e.g. if the indicator calculated is about the public cost of energy savings)
Society	·         Part of the investments paid by the participants (for additional actions only) ·         Financial incentives paid to the participants ·         Administration costs of the scheme	·         Net energy savings

NOTE: the table above does not deal with non-energy impacts. Depending on the context and objectives of the incentive scheme, non-energy benefits can be larger than the benefits from energy savings. When assessing the cost-effectiveness of an incentive scheme from the society’s point of view, it is therefore recommended to consider if it is relevant to include non-energy impacts in the scope of analysis (see Calculating other co-benefits below for more details).

Cost-effectiveness indicators are usually calculated taken into account final energy savings. Likewise, these indicators are most often calculated over the lifetime of the energy saving actions (to take into account the full benefits from the energy saving actions), or over a given horizon of time (when considering the time perspective of investors, possibly including the State as a general public investor). The relevant metric of energy savings is therefore cumulative or lifetime-cumulated energy savings. This means that these calculations require assumptions about the lifetime of the actions and the evolution of energy savings over time.

Depending on the indicator(s) used to assess cost-effectiveness, it can be needed to use a discount factor (e.g. when the indicator is Net Present Values). The discount factor should be applied consistently to all terms of the calculations (i.e. all costs and benefits, including the energy savings). Different discount factors can be used according to the perspective adopted (e.g. households, State). It is important to document the use of discount factors, and if possible to make a sensitivity analysis (testing several values or ranges of discount factors). As this can affect significantly the results. For more discussions about discount factors, see for example the Horizon 2020 projects Briskee and Cheetah.

Likewise, the calculations of cost-effectiveness indicators will usually require to consider scenarios of energy prices over given periods. The assumptions about trends in energy prices should be documented. Whenever possible, it is recommended to make a sensitivity analysis (testing several scenarios of energy prices). See for example the EPATEE case study about the Better Energy Homes scheme in Ireland (Broc, 2017).

For more details about assessing cost-effectiveness or cost-benefit analysis, see for example (Clinch and Healy 2001; Collins and Curtis, 2017). And more generally about assessing the effectiveness of incentive schemes, see for example (Stern et al ., 1986; Studer and Rieder, 2019).

7.3 Calculating other Co-benefits

Possible co-benefits from saving energy can concern:

• Reduction of energy poverty
• Better indoor climate
• Extra employment
• Impact on economic activity
• Reduced dependency on (insecure) energy import

It should be noted that the impacts from incentive schemes on each of these aspects are usually positive, but can also be negative (e.g. on State budget or distributional effects). Therefore, it is in general more appropriate to speak about non-energy impacts.

For a general background about non-energy impacts, see here

Analysis about how non-energy impacts can have an influence on the political support or rationale for an incentive scheme can be found for example in (Kerr et al., 2017; Rosenow, 2013).

EPATEE case studies:

• Broc, J.S. (2017). Better Energy Homes scheme (Ireland). Case study prepared by IEECP for the EPATEE project, funded by the European Union’s Horizon 2020 programme.
• Meulen, J. & Menkveld, M. (2018). Subsidy scheme for energy savings in social housing of Amsterdam. Case study prepared by ECN-TNO for the EPATEE project, funded by the European Union’s Horizon 2020 programme.
• Niessink, R. (2018). The Case-Study Special Subsidy Scheme for Energy Labelling Improves Social Housing Sector Amsterdam Seen in the Perspective of the European H2020 Project EPATEE. Proceedings of IEPPEC 2018.