10 – Evaluating Grants & subsidies measures on energy efficiency in the Residential buildings using Deemed savings

The primary audience for this guide concerns energy efficiency programme designers, implementers or supervisors, and evaluators looking for guidance on the evaluation process of energy savings in the scope of this guide.

Although the application of the guide will generally concern the (sub)national level, account will be taken of issues at EU level when relevant (e.g. the specific format of saving figures for the EED).

This guide is not about the preceding step in the evaluation process, the choice of the method. About this previous step in the evaluation process, see the guidance provided here. However, after presenting the capabilities and limitations of the guide at hand, the user will be offered alternatives for the method within this guide (see section 6).

The objective of this guide is to provide:

• Information on the scope of the guide that enables the user to decide whether this guide is suited to his/her needs, and whether complementary or additional method(s) could be needed or useful (section 2);
• Guidance about specifying the evaluation objectives and requirements (section 3);
• Guidance about key methodological choices to calculate energy savings (section 4);
• Guidance about the inputs (data requirements) and outputs of the method (energy savings metrics) (section 5);
• Possible alternative methods (with pros and cons) (section 6)
• Background about evaluation results other than energy savings (section 7);
• Relevant examples, case studies and/or good practices (section 8);
• Relevant references for further reading (section 9).

The guide is intended for assessing realised (ex-post) energy savings. However, account is taken of earlier (ex-ante) evaluations of expected savings, if available (see section 4).

The focus of the guide is on impact evaluation, i.e. determining the energy savings, but not on how this has been reached through a step by step process with intermediate results (process evaluation).

Readers looking for the basic and general principles of energy efficiency evaluation may find the following link useful.

Furthermore, the following guidance presents a brief analysis of the results of a scenario modelling procedure to calculate the theoretical CO2 savings from four of the most important measures offered under UK Warm Front programme, launched in 2000.

2.1 About grants & subsidies

More information and examples on the different subtypes residing under the main type Grants & Subsidies (GS) can be found can be can be found here and here and also in this link and this link.

The focus of this guide can be extended to tax-rebates (part of the main type Fiscal / Tariffs) that stimulate energy savings in the same way as subsidies. Grants and subsidies are sometimes combined with tax-rebates to reduce energy consumption.

Subsidy schemes on residential buildings generally concern grants covering part of the investments needed for various saving actions, such as glass/wall/roof/floor insulation and more efficient boilers and/or improved heating control. The grant could be a fixed amount (e.g. for a new boiler) or an amount per m2 (insulation). The grant can also be dependent on the quality of the saving actions, e.g. higher for very efficient boilers than for just more efficient boilers.

The subsidy scheme is often managed by an agency that reports to government about provided subsides, the number of executed saving actions and the savings realized. Often an evaluation of the effectiveness of the scheme is performed as part of the general control on government spending.

More detailed information on the evaluation of grants can be can be found here.

2.2 Evaluation for a combination of policy measure types

Grants are often combined with information (part of main policy type Information & Education), in order to inform dwelling owners (or renters) on the possibilities for saving energy. They may also be combined with the policy type Voluntary Agreements between a group of end-users and government/agency, or they may be combined with obligation scheme. For such combinations it is assumed that the overall savings are mainly due to the grants, although the evaluation concerns the combined savings effect of both policy measures.

The guide is not capable of attributing part of the (overall) calculated savings to each of the policy measures (see also Double counting in the section on Calculating Gross and Net savings).

2.3 Evaluation when combined with energy taxes

Energy taxes may also be present, which stimulate to invest into all kind of saving actions. The calculated savings effect for grants will overlap with that of the energy tax. The guide is not capable of attributing part of the (overall) calculated savings to either the policy measures at hand or the energy tax. For dealing with this overlap see section on Gross to Net savings.

2.4 About residential buildings

Information on (sub)sectors defined in the Toolbox can be found here, chapter 2, p.17

Other, equivalent descriptions of the sector are “dwellings”, “houses” and “apartments”.

The residential sector consists mainly of single-family homes (row, two under one roof or detached) and multi-family apartment complexes. Residents may own or rent the unit they reside in.

Total energy use within a home may not necessarily be the same as the billed energy consumption, on-site energy production and micro-generation technologies.

Energy savings for new dwellings are generally realised with minimum efficiency standards. Important considerations when addressing the residential sector include rebound, spill-over and free-rider effects.

2.5 Evaluation for a combination of policy measure types

Information about the various evaluation methods can be found here, table 1 and 2. This source also covers the combination of the method at hand with other methods, which will be dealt with below.

2.6 Evaluation for cross-sector saving actions

No cross-sector saving actions are applicable as the actions concern only residential dwellings.

2.7 About stock modelling

Information about the various evaluation methods, including stock modelling, can be found here, table 1 and 2. This source also covers the combination of the method at hand with other methods, which will be dealt with below.

Building stock models for energy consumption represent a key guide to assist with the efficient and rational implementation of policy. Stock modelling concerns changes, due to energy policy, in the stock of energy using systems, such as dwellings, appliances or cars. Every year, part of the old systems is replaced by a more efficient new one and these saving actions lead to increasing overall savings over time. The (yearly) number of saving actions is covered in stock modelling but the unitary savings to some extent as well. In most cases (see Knowledge Base) deemed savings are used to calculate the unitary savings, but for dwellings engineering estimates can be used as well.

Among others, stock models can be used to estimate the ‘baseline’ energy consumption of the residential housing sector, disaggregated by building or social categories and energy end-uses; explore the technical and economic effects of different CO2 emission reduction strategies over time, including the impact of new technologies, such as renewables and smart metering; as well as identify the effect of emission reduction strategies on indoor environmental quality.

2.8 Complementary methods to determine total savings

Complementary methods are methods that are required, in addition to the primary selected method, to calculate energy savings.

The method at hand concerns the determination of the number of actions, i.e. the number of participants that replace e.g. old inefficiency boilers with new high-efficiency boiler and/or old insulation with new more energy efficient insulation. In order to provide the total savings, the unitary savings per action or participant should also be calculated. To calculate the total gross savings one will need the unitary savings per action and the change in stock. For more information about methods to calculate unitary savings, see the link here, table 2.

2.9 Additional methods to increase reliability of the results

An additional method can be applied on top of stock modeling to improve the reliability of the evaluation results and/or the cost-effectiveness of the evaluation approach. Stock modeling can be combined with deemed savings method and/or engineering estimates method.

For possible combinations with an additional method see chapter 6 here.

3.1 Meeting evaluation goals and ambition

The table shows whether this guide can be used to report on general evaluation goals or criteria. See also this document.

General types of evaluation goals or criteria	Level of ambition	Remarks
Calculation of realized energy savings from saving actions	Fair/Good	Depends on the complementary method to calculate unitary saving
Calculation of energy savings attributed to the policy measure(s)	Good	Number of actions to be corrected for free riders
Cost-effectiveness of saving action (for end-users)	Good	Focus on specific uniform saving actions with savings and cost data
Cost-effectiveness of policy (government spending)	Fair	Effectiveness of grant scheme influenced by free riders and rebound/prebound effects
CO2-emission reduction from saving actions	Good	Reduction per saving action clear
CO2-emission reduction attributed to the policy measure(s)	Fair	Effectiveness of grant scheme influenced by rebound/prebound effects

For more information on verification of actual energy savings and attribution/baseline/corrections, see section 4, and for cost-effectiveness and emission reduction see section 7.

3.2 Reporting expectations

• Gross and Net savings
• Yearly or cumulative savings
• Free rider, rebound and prebound effect.

3.3 Time frame for evaluation

The length of the period under evaluation is dependent on

• the active period of the policy measure;
• the need to monitor developments before the implementation of savings actions (in case of methods based on before/after saving actions); and
• the time needed to present (reliable enough) results or impacts that fit into the decision making process. In some cases, the periodicity of evaluation can be set by law.

The planning of evaluation activities concerns regular monitoring of energy consumption and factors that define consumption, intermediate check of (ex-ante) estimated (unitary) savings through measuring or surveys, intermediate evaluations to improve the policy implementation and the final evaluation and reporting. See also here.

The unitary savings are calculated with a complementary method. If deemed savings is used the evaluation period might have to start before the grant scheme in order to have reliable deemed values. For billing analysis as complementary method the same period as for the diffusion method can be chosen.

The planning of evaluation activities concerns regular monitoring of energy consumption and factors that define consumption, intermediate check of (ex-ante) estimated (unitary) savings through measuring or surveys, intermediate evaluations to improve the policy implementation and the final evaluation and reporting. See also planning of evaluation in the link here.

3.4 Expertise needed for chosen method

To be able to evaluate the effects of grants on energy efficiency improvements in dwellings, one needs to have good insight in dwelling stock, the buying behavior of different income groups and all other external factors that define the choice to renovate dwellings. With this knowledge, a reliable reference situation can be constructed.

3.5 Boundaries for the evaluation

In this guide on dwellings, the method is normally applied to uniform saving actions, such as various insulation measures (such as roof, ground floor, windows, walls, etc. and replacement of the boiler. The guide covers only part of overall energy consumption of households that is billed. Moreover, the interaction with other (already present) saving actions is not covered in this guide.

The geographical boundary of dwellings when applying stock modelling method can vary. However, since grant schemes are normally set at the national level, the geographical area for evaluating grant and subsidy schemes for energy efficiency actions in dwellings is limited to individual countries.

This section deals with key methodological choices to be considered when calculating energy savings: consistency between ex-ante and ex-post evaluation, baseline, normalization and adjustment factors. These choices are important to document when reporting energy savings, to ensure the transparency of the results.

General principles of calculating realized savings using different methods can be found here and here.

The stock modelling method uses knowledge about the composition of the dwelling stock to construct a reference situation. Renovation behavior is related to the family composition, the residential environment, household income, comfort and the trust of consumers in the economic situation.

4.1 Matching method with earlier ex-ante evaluation

If an ex-ante evaluation has been performed before the start of the grant scheme the choice of the ex-post evaluation method can be matched with that of the ex-ante evaluation. This will depend on the evaluation objectives, timeframe and data available. For stock modelling, it is possible to apply the method also for ex-ante evaluation provided that scenario data are available on all factors that define the purchase decisions that change the numbers of e.g. more efficient boilers.

From the viewpoint of methodological consistency and data availability, using the same method in the ex-ante evaluation and in this guide on ex-post evaluation might be an obvious choice. However, for ex-ante evaluation only a few methods are usually considered, namely deemed savings and engineering estimate (see section on complementary methods).

A different method than the one(s) used for the ex-ante evaluation can be applied for the ex-post evaluation, depending on the evaluation objectives, timeframe and data available for the situation after implementing the actions.

For instance, the ex-ante calculation of unitary savings with the deemed savings method can be combined with the ex-post calculation of unitary savings using measurement (see section on additional methods).

For possible combinations of methods applied ex-ante and ex-post, see chapter 7, calculation approaches in this document.

If the stock modelling method does not provide an acceptable combination with the earlier applied ex-ante method it might be useful to select another method (see examples of alternatives in section 6).

In practice, ex-ante and ex-post evaluations are applied consecutively. The ex-post evaluation builds on an ex-post evaluation that makes use of data coming from previous ex-post evaluation or studies (e.g. about previous periods of the same policy measure, or about the same types of energy saving actions as the ones promoted by the new policy measure). These previous ex-post studies could have used another type of method as well.

4.2 Calculation baselines

Energy savings are defined in general as the difference between the actual situation and a reference situation without the saving actions (and without the policy measures that influence these saving actions). In case of saving actions the reference situation can be defined using various calculation baselines, such as : Before/after, With/without, Trend, Target/control group and Minimum efficiency standards; see further here.

The applicable baseline in the case of stock modelling is ‘with/without’ (the purchase tax). Another possible baseline options is a minimum efficiency standard, but it is assumed that the purchase tax influences the above-standard fuel use. A brief description of both With/Without and Minimum efficiency standards:

• With/without: analysis of changes in energy consumption with/without the saving action, where the “without” situation acts as the baseline for the “with” situation. This option is for example suited for new efficient systems where no “before” situation is available. Results can be corrected for adjustment factors, providing Net savings. The With/without baseline can also be applied more generally for evaluation of policy measures, especially to evaluate energy savings additional to a predefined scenario (e.g., business-as-usual scenario or a scenario including other policy measures). In this case, this predefined scenario will be used to specify the “without” situation.
• Minimum efficiency standards: if a standard is in place, e.g. for appliances or cars, this baseline can be chosen so that only the above-standard savings count. The standard acts as a baseline for attributing savings due to other policy measures. Therefore, this baseline can be combined with the other baselines. The “minimum efficiency standards” option is a sub-category of the “with/without” option, where the “without” situation is defined as energy performance level required by the standards.

See also here.

4.3 Normalization factors

The calculation with the above-mentioned baseline calculation options provides a change in energy consumption that should be corrected for influences on energy consumption other than the saving actions. These so-called normalization factors can be weather (with effect on consumption), the rebound effect and changes in energy using activities, such as production (industry), occupation rate (buildings) or car usage (transport).

For the guide at hand, in which stock modelling is used to determine the number of saving actions in combination with deemed savings to determine the unitary savings per action, the normalization factors can concern the (direct) rebound effect (see table 1 in this link).

Energy consumption can be corrected for the rebound effect, e.g. the extra energy consumption of higher thermostat setting after saving actions on dwellings that lower the energy bill, or longer distances travelled after buying a more efficient vehicle. If required by evaluation goals and ambitions (see section 3) also the prebound effect can be taken into account (see normalization factors in this document.

Energy consumption can be corrected for occupation rate of buildings (with effect on heating/lighting), when this rate is different for the baseline situation and the situation after the saving actions.

4.4 Adjustment factors

Adjustment factors define which part of the calculated energy savings can be attributed to a policy measure or meets the definition of savings specified in the evaluation objectives or reporting requirements (see next section on “Calculating Gross and net savings).

In case of bottom-up methods, see here, table 1. For this particular specific guidance, adjustment factors to consider in priority are free rider and spill-over effect and non-compliance.

In case of another policy focusing on the same saving actions as evaluated with the specific guidance at hand the adjustment factor Double counting might be relevant. If the other policy is not covered in the guidance at hand however, double counting can only be accounted for at a higher level than individual guides; see here and here.

Most adjustment factors influence the unitary savings of a saving action (see Gross and net savings), but the free rider effect influences the number of actions. To determine the size of the free rider effect, a distinction must be made between saving actions due to the policy measure and actions, which would have been taken anyway. The stock modelling method does not provide this information directly, so other ways must be found, such as a survey among participants of the policy measure about their motivation, or application of Randomized Controlled Trial (RCT) or Quasi-experimental design (see further in this topical case study).

The calculation of the spill-over effect is often difficult due to lack of data to prove a causal relation with the policy measure/sector at hand. Non-compliance, an improper implementation of e.g. insulation measures asks for a sample-wise check of the quality of saving actions.

For checking on non-compliance (sample) data should be available on the real savings when replacing e.g. a specific old boiler for a new one from a certain efficiency class.

For more information on adjustment factors see the above links.

4.5 Calculating Gross and Net energy savings

savings concern the calculated savings from saving actions using a chosen baseline and normalization factors. Net savings concern the savings attributed to policy measures or eligible for a stakeholder (e.g. an energy company with an obligation to realise savings at their customers).

The gross savings can be calculated using the appropriate baseline for number of saving actions and for unitary savings (see section on baselines), and correct the unitary savings for normalization factors (see section on normalization).
Gross unitary savings can include a correction for the rebound effect in case of using the complementary methods deemed savings or engineering estimate (see section on baselines). In case of billing analysis the calculated change in energy consumption already incorporates the rebound effect, but gross savings should be corrected for weather (see section on normalization). No correction for changes in activity level is needed.

Net savings follow from gross savings by applying adjustment factors. Net unitary savings can be corrected for additionality and non-compliance. The net number of saving actions can be corrected for free riders and the spill-over effect (see section on adjustment factors).

The savings should be corrected for the Double counting effect, i.e. the overlap between the savings due to Grants and savings due to other policy measures. The overlap in the calculated savings of both policy measures cannot be processed at the level of a guide but must be corrected at the level of savings due to overall policy portfolios. For addressing double counting see here and here.

See section 8 on concrete examples.

5.1 Main data requirements, sources and collection techniques

Data requirements specified in the table below correspond to the calculation of energy savings, when using the baseline option With/Without. Requirements are dependent on the chosen complementary method for calculating the unitary savings (here data for deemed savings).

Calculation subject	Data requirements	Possible data sources and collection techniques
Energy consumption	Gas, electricity and heat consumption, number of inhabitants	Energy bills, occupancy rate, inhabitant behavior from interviews / surveys
Normalization factors affecting energy consumption	Rebound effect, prebound effect	Interviews/surveys
Adjustment factors	Non-compliance	Checking for correct energy standard implementation
Primary energy factors applied (for conversion from final to primary savings)	Conversion losses in transporting energy carriers, e.g. electricity, gas, etc.	Energy statistics
Number of actions	Number of dwellings with the energy standard studied	Building permits
Data about participants to ensure representativeness or good fit	Large enough sample size of participants interviewed	Interviews / surveys

Data issues when evaluating net energy savings

The main good practice to ensure the feasibility and reliability of the evaluation of net energy savings is to think about the method to be used when designing (or revising) the grant/subsidy scheme. Experience indeed shows that unless the data collection has been planned in advance, it will be very costly, time-consuming or even impossible to collect the data required to apply most of the methods that can be used to evaluate net energy savings.

The main challenges when using surveys on rebound and free rider are:

• to achieve a high answer rate, in order to limit sampling bias;
• to use question phrasing that can limit the risk of bias in the answers.

For more details about the evaluation of net energy savings, see the topical case study

To determine the gross savings, first the number of actions is needed. For this, data home upgrades (e.g. insulation, installation of high-efficiency boilers and/or improved heating control) and knowledge on the energy savings of these actions needs to be known. This can be corrected for the rebound effect. For the energy saved per action, a deemed energy savings number can be used. To determine net savings, the free rider and non-compliance effects need to be applied.

Data issues with the additional method

For possible other methods with different data demands see the section on alternatives for the chosen method.

5.2 Energy savings in final terms or in primary terms

Energy savings can be expressed in final terms or in primary terms. See definitions about primary and final energy here.

Stock modelling combined with a method to determine energy saved per car can calculate savings in final terms. It can also calculate savings in primary terms provided that savings at end-users are calculated for each energy carrier separately, and primary factors are available to convert the savings in final terms to savings in primary terms.

5.3 Energy savings over time

Implemented saving actions in a year lead to savings over a number of consecutive years (e.g. a more efficient boiler can save gas over its lifetime of about 15 years, insulation over up to 30 years and more efficient computers up to 5 years). Energy savings can be calculated in different metrics in terms of time reference, for example: year-to-year, annual, cumulated annual, cumulative. See the definitions here.

The calculated yearly savings concern the savings of all new saving actions in that year. In this approach only data for the savings in the chosen year are needed. Adding up the yearly savings over a period, provided that earlier saving actions are still delivering savings, leads to cumulative savings. For the cumulative savings data are needed for the whole period.

Another cumulative approach, to be applied for the EED, is to multiply the (new) savings in a year with the number of years up to a target year and sum this result with that for all other years up to the target year. This cumulative approach stimulates early saving actions, as these count more times to the target than later actions.
Stock modelling can provide yearly savings of new saving actions in that year. It can also provide cumulative savings watermark if data are available over a period.

Cumulative savings according to the Energy Efficiency Directive can be provided when the lifetimes of savings actions are known. Finally, the guide can provide discounted cumulative savings when discount factors have been defined for yearly savings over time. A relevant discount factor for efficient cars is that the energy efficiency of cars is improving anyway, so the baseline will improve over time.

Finally, savings from a saving action can be discounted and summed up over the lifetime of the action See link here.

6.1 Alternatives for the chosen method

Often other savings calculation methods can be applied as well, although they will all have pros and cons regarding various aspects dealt with in preceding sections.

The table below presents the pros and cons of the stock modelling method (covered in this guide) and for commonly used alternative methods for the same combination of policy measure and sector.

Type of method	Pros	Cons
Method covered in this guidance (stock modelling)	Clear and precise	Sizable data collection effort needed. Deemed savings to be used for free-riders
Alternative method: Diffusion indicator (1)	Calculation with diffusion indicator and deemed savings relatively simple	No check with actual energy consumption changes
Alternative method: Diffusion indicator (2)	Diffusion data can be retrieved from grant decisions.	Deemed savings need to be checked

7.1 Calculating avoided CO2 emissions

Avoided CO2 emissions can be evaluated from the energy savings by applying emission factors. Four key aspects are to be considered when choosing the emission factor(s):

1. Emission factors vary according to the energy type, so the data about energy savings need to be available per energy type.
2. Emission factors for a given type of energy can vary over time (especially for electricity).
3. Emission factors can take into account:
   1. Direct emission factors: that take into account the emissions generated when producing the energy used;
   2. Lifecycle emission factors: that take into account all the emissions generated from the extraction of the energy resources up to the dismantling of the energy plant.

Due to the differences that the choice of emission factor(s) can induce, it is important to document what emission factor(s) has(have) been used.

When the savings concern one fuel only, e.g. gas boilers, the reduction in CO2-emissions can be calculated from the savings with an emission factor for the fuel at stake.

The reduction in CO2-emissions can only be calculated when savings are calculated per relevant energy carrier and a specific emission factor is available for each energy carrier

The avoided emission of other greenhouse gasses due to energy savings are not taken into account here, as these emissions (and more specifically their reductions) are generally negligible compared to CO2 (apart from policy measures targeting the agriculture sector).

IPCC (Intergovernmental Panel on Climate Change) provides a detailed database of peer-reviewed emission factors.

7.2 Calculating cost-effectiveness

Cost-effectiveness is the ratio between costs to achieve energy savings and the amount of savings and possibly other benefits.

A distinction can be made according to the point of view adopted to assess cost-effectiveness:

• Cost-effectiveness for the end-user or participant
• Cost-effectiveness for society at large
• Cost-effectiveness for the party that takes responsibility for saving targets (government or actor with an Energy Efficiency Obligation)

See further here, report on Evaluation into Practice: lessons learnt from 23 evaluations of energy efficiency policies, section 13.1.
The calculation of cost-effectiveness for end-users demands, next to the savings, data on investments made, subsidies on investments, interest rates, lifetimes of the saving actions, energy prices (including taxes) per type of end-user and discount factors per type of end-user.

For cost-effectiveness from a societal viewpoint, no account is taken of subsidies and taxes, energy prices concern world market price, and a lower value of the discount factors is valid.

For cost-effectiveness from a government perspective the amount of subsidies is related to the (extra) savings of the saving actions due to the grant scheme.

7.3 Calculating other co-benefits

Possible co-benefits from saving energy concern:

• Extra employment
• Reduction of energy poverty
• Other emission reductions (NOx, SO2, fine particles, etc.)
• Better indoor climate
• Reduced dependency on (insecure) energy import

7.4 Other aspects of importance

None identified.

The Better Energy Homes (BEH) scheme in Ireland, which aimed at improving the energy efficiency of dwellings, and also reducing heating bills, CO2 and air pollutant emissions. Administered by the Sustainable Energy Authority of Ireland (SEAI), it provided direct Government fix grants (representing about 30% of the total investment costs) to homeowners (including landlords renting dwellings) to upgrade their dwellings with energy efficiency actions. Grants are available for the following energy-saving and renewable solutions: Attic insulation; Wall insulation – including cavity wall, internal dry lining and external insulation; Heating controls upgrade; Solar thermal solutions; and Heat pump systems (from 16 April 2018)For case study on the BEH scheme, please see here. This case study does not refer specifically to the stock modeling method, but includes other interesting issues of relevance, such as methodological choices in evaluating energy savings from the grant scheme.

Another interesting case study is the Lithuanian Renovation Programme, which was executed with EU-funding. The programme was launched in 2005, with the aim to renovate the 4,000 most heat consuming multi-apartment blocks (equivalent to about 120.000 dwelling units), and to implement 10,000 energy saving measures during the 2005-2020 period. Preferential loans were given to implement energy saving actions, see here.