11 – Evaluating Grants on energy efficiency in Dwellings using Diffusion indicators
This specific guide can be applied to evaluate the savings due to subsidies for saving actions in the sector Buildings/residential using the method Diffusion indicators. It includes guidance and explanations specific to this combination of types of policy measure, sector and method. As well as links to general guidance and explanations that can also apply to this combination.
The primary audience for this guide is energy efficiency programme designers, implementers or supervisors, and evaluators looking for guidance on the evaluation process of energy savings in the scope of this guide.
Although the application of the specific guide will generally concern the (sub)national level, account will be taken of issues at EU level when relevant (e.g. the specific format of saving figures for the EED).
This guide is not about the preceding step in the evaluation process, the choice of the method. About this previous step in the evaluation process, see the guidance provided here. This guide is not about the preceding step in the evaluation process, the choice of the method. About this previous step in the evaluation process, see the guidance provided here. However, after presenting the capabilities and limitations of the guide at hand, the user will be offered alternatives for the method within this guide (see section 6).
The objective of this guide is to provide:
- Information on the scope of the guide that enables the user to decide whether this guide is suited to his/her needs, and whether complementary or additional method(s) could be needed or useful (section 2);
- Guidance about specifying the evaluation objectives and requirements (section 3);
- Guidance about key methodological choices to calculate energy savings (section 4);
- Guidance about the inputs (data requirements) and outputs of the method (energy savings metrics) (section 5);
- Possible alternative methods (with pros and cons) (section 6)
- Background about evaluation results other than energy savings (section 7);
- Relevant examples, case studies and/or good practices (section 8);
- Relevant references for further reading (section 9).
The guide is intended for assessing realised (ex-post) energy savings. However, account is taken of earlier (ex-ante) evaluations of expected savings, if available (see section 4).
The focus of the guide is on impact evaluation, i.e. determining the energy savings, but not on how this has been reached through a step by step process with intermediate results (process evaluation).
Readers looking for the basic and general principles of energy efficiency evaluation may find the following link useful.
2.1 About Grants
More information and examples on the different subtypes residing under the main type Grants & Subsidies (GS) can be found here and here. The focus of this specific guide can be extended to tax-rebates (part of the main type Fiscal / Tariffs) that stimulate energy savings in the same way as subsidies.
Subsidy schemes on residential buildings generally concern grants covering part of the investments needed for various saving actions, such as glass/wall/roof/floor insulation and more efficient heating systems. The grant could be a fixed amount (e.g. for a new boiler) or an amount per m2 (insulation). The grant can also be dependent on the quality of the saving actions, e.g. higher for very efficient heating systems than for just more efficient ones.
The subsidy scheme is often managed by an agency that reports to government about provided subsides, the number of executed saving actions and the savings realized. Often an evaluation of the effectiveness of the scheme is performed as part of the general control on government spending.
More detailed information on the evaluation of Grants can be found here.
Grants are often combined with information (part of main policy type Information & Education), in order to inform dwelling owners on the possibilities for saving energy. They are also combined with the policy type Voluntary Agreements between a group of end-users and government/agency. For such combination it is assumed that the overall savings are mainly due to the grants, although the evaluation concerns the combined savings effect of both policy measures.
The guide is not capable of attributing part of the (overall) calculated savings to each of the policy measures (see also Double counting in the section on Calculating Gross and Net savings).
2.2 Evaluation when combined with energy taxes
Energy taxes may also be present, which stimulate to invest into all kind of saving actions. The calculated savings effect for grants will overlap with that of the energy tax.
This guide is not capable of attributing part of the (overall) calculated savings to either the policy measures at hand or the energy tax. For dealing with this overlap, see section on Gross to Net savings.
2.3 About Buildings/residential
The focus of the subsidy scheme is existing dwellings, as savings for new dwellings are generally realized with the deployment of minimum efficiency standards. The scheme can focus on owners of dwellings and/or on social housing corporations with rented dwellings. The scheme can focus on individual dwellings (detached, row) or apartment buildings with common facilities.
2.4 About Diffusion indicators
Diffusion indicators show the penetration of specific saving actions, in this case e.g. the fraction of High Efficiency boilers from all boilers, or the fraction of dwellings with roof insulation. It also can show the share of renovated buildings to certain energy efficiency class in total building stock.
The indicator generally covers the national stock of dwellings, but separate indicators may be available per type of dwelling.
The indicator is calculated yearly on the basis of statistical or other sources (see section on main data requirements).
The indicator concerns the number of saving actions (or fraction of dwellings) but does not provide information on the unitary savings per saving action (see next section on complementary methods).
Information about the various evaluation methods can be found here, table 1 and 2. This source also covers the combination of the method at hand with other methods, which will be dealt with below.
2.5 Complementary methods to determine total savings
Complementary methods are methods that are required, in addition to the primary selected method, to calculate total energy savings.
The method at hand concerns the determination of the number of actions. In order to provide the total savings the unitary savings per action should also be calculated. Therefore, the guide should also contain one of the methods to calculate unitary savings. Because diffusion indicators often concern uniform saving actions at sector level or beyond, the deemed savings method is often applied to calculate unitary savings. However, the methods engineering estimate or billing analysis can also be applied.
For more information about methods to calculate unitary savings, see table 2 in this link.
2.6 Additional methods to increase reliability of the results
An additional method can be applied on top of the chosen method to improve the reliability of the evaluation results and/or the cost-effectiveness of the evaluation approach.
Additional methods concern the calculation of unitary savings (see chapter 6 in this link). Because the chosen method Diffusion indicator calculates the number of actions, there will be no additional method available.
3.1 Meeting evaluation goals and ambition
The table shows whether this guide can be used to report on general evaluation goals or criteria. See also this document.
|General types of evaluation goals or criteria||Level of
|Calculation of realized energy savings from saving actions||Fair||Depends on the complementary method to calculate unitary saving|
|Calculation of energy savings attributed to the policy measure(s)||Limited||Number of actions to be corrected for free riders|
|Cost-effectiveness of saving action (for end-users)||Good||Focus on specific uniform saving actions with savings and cost data|
|Cost-effectiveness of policy (government spending)||Fair||Effectiveness of grant scheme influenced by free riders|
|CO2-emission reduction from saving actions||Good||Reduction per saving action clear|
|CO2-emission reduction attributed to the policy measure(s)||Fair||Effectiveness of grant scheme influenced by free riders|
For more information on verification of actual energy savings and attribution/baseline/corrections, see section 4, and for cost-effectiveness and emission reduction see section 7.
3.2 Reporting expectations
Possible reporting options:
- Gross and Net savings
- Yearly savings and cumulative of discounted savings
- Rebound and free rider effects.
3.3 Time frame for evaluation
The length of the evaluation period is dependent on:
- the active period of the policy measure (without major changes),
- the need to monitor developments before the implementation of savings actions
- the time needed to present (reliable) results or impacts that fit into the decision making process. In some cases, the periodicity of evaluation can be set by law.
For the guide at hand the calculation of the number of actions with the Diffusion indicator method asks for a period equal to the active period of the policy measure including previous year because the number of saving actions is calculated as the difference with the year before.
The unitary savings are calculated with a complementary method. If deemed savings is used the evaluation period might have to start before the grant scheme in order to have reliable deemed values. For billing analysis as complementary method, the same period as for the diffusion method can be chosen.
The planning of evaluation activities concerns regular monitoring of energy consumption and factors that define consumption, intermediate check of (ex-ante) estimated (unitary) savings through measuring or surveys, intermediate evaluations to improve the policy implementation and the final evaluation and reporting. See also planning of evaluation in the link here.
3.4 Expertise needed for chosen method
If corrections for normalization and adjustment factors are required when calculating the unitary savings expertise is needed on how these factors influence the savings and how to gather the data needed to assess the effects on gross and net savings.
3.5 Boundaries for the evaluation
In this guide on dwellings the diffusion indicator method can only be applied to uniform saving actions, such as various insulation measures and replacement of the boiler. The guide covers only part of overall energy consumption of households that is billed. Moreover, the interaction with other (already present) saving actions is not covered in the guide.
This section deals with key methodological choices to be considered when calculating energy savings: consistency between ex-ante and ex-post evaluation, baseline, normalization and adjustment factors. These choices are important to document when reporting energy savings, to ensure the transparency of the results. General principles of calculating realized savings using different methods can be found here and here.
Calculating energy savings with the method Diffusion indicators only works for uniform saving actions, which can be counted because each saving action has (more or less) the same unitary savings. Therefore, the saving actions in dwellings are split as to type (e.g. various insulation actions) and, if relevant, the kind of dwelling (to take into account the amount of insulated surface).
On basis of yearly monitoring of the number of (various) saving actions present, the extra saving actions per year are calculated per type. These numbers are multiplied by the unitary savings provided by the complementary method (see chapter 2) to have the total savings.
The increase in the number of saving actions concerns both the increase due to policy (in this case Grants) and the autonomous increase (e.g. replacing old boilers by new boilers on the market that are by definition more efficient). The increase connected to the subsidy scheme should be corrected for free riders (see section on Gross-to-Net savings).
4.1 Matching method with earlier ex-ante evaluation
If an ex-ante evaluation has been performed before the start of the grant scheme the choice of the ex-post evaluation method can be matched with that of the ex-ante evaluation. This will depend on the evaluation objectives, timeframe and data available.
From the viewpoint of methodological consistency and data availability using the same method in ex-ante evaluation and in this guide on ex-post evaluation might be an obvious choice. However, for an ex-ante evaluation usually no Diffusion indicator method is applied because this method is based on observed data (see table 4 in this link).
The Diffusion indicator method can only be applied ex-ante if a detailed scenario study supplies the same data as used in the ex-post evaluation. Using the same method enables to compare the expected values with the observed values, which can lead to better ex-ante evaluations.
The (ex-post) diffusion indicator method could be matched with a different method in the ex-ante evaluation. But the generally available ex-ante methods deemed savings or engineering estimate concern the calculation of unitary savings and cannot be matched with the diffusion indicator method on number of saving actions (see “Different methods ex-ante and ex-post in chapter 7 here. However, the ex-post diffusion method can be matched with the ex-ante method Stock modeling.
To calculate total savings the diffusion indicator method has to be combined with a complementary method on calculating unitary savings. This method on unitary savings (ex-post) can be matched as well with the same method or another method in the ex-ante evaluation. Given the generally available ex-ante methods the match is confined to the deemed savings or the engineering estimate method.
If the Diffusion indicator method does not provide an acceptable (but requested) match with the earlier applied ex-ante method it might be useful to select another specific guide (see examples of alternatives in section 6).
4.2 Calculation baselines
Energy savings are defined in general as the difference between the actual situation and a reference situation without the saving actions (and without the policy measures that influence these saving actions). In case of saving actions the reference situation can be defined using various calculation baselines: Before/after, With/without, Trend, Target/control group and Minimum efficiency standards; see further here.
Total savings are found by multiplying the unitary savings per action by the number of actions. The chosen baseline can define the total savings through the unitary savings, but also through the number of actions.
The focus of the method Diffusion indicator is on calculating the number of saving actions. The baseline With/without or Trend can be used to calculate the number of actions due to the grants.
The unitary savings are calculated with the complementary method deemed savings (or engineering estimate or billing analysis, see section 2). Here the baseline option With/without can be chosen. In case of above-standard-savings the baseline Minimum efficiency standard can be chosen ; see table 1 in this link.
Depending on the requirements in section 3, preference should be given to one or the other baseline.
See further information on baselines in this document.
4.3 Normalization factors
The calculation, using the baseline(s) considered in the previous section, provides a change in energy consumption that should be corrected for influences on energy consumption other than saving actions. These so-called normalization factors can be weather (with effect on consumption), the rebound effect and changes in energy using activities, such as production (industry), occupation rate (buildings) or car usage (transport).
Total savings result from unitary savings per action times the number of actions. For the calculation of number of actions with the Diffusion indicator method no normalization factors are valid. The calculation of the unitary savings can involve normalization for the rebound effect (the extra energy consumption due to a higher thermostat setting after saving actions that lower the energy bill). For dwellings, no normalization of changes in energy using activities is needed (see table 1 in this link).
If the complementary method deemed savings is applied for calculating unitary savings no normalization for weather is needed. But for the complementary method billing analysis a correction for weather is needed. Normalization factor Weather concerns a correction of energy consumption data for differences in outdoor temperature during the heating season or cooling season. Actual and baseline energy consumption data are corrected for yearly deviations from long term mean values, expressed in heating degree days for heating or cooling degree days for cooling.
A correction for the Rebound is not automatically done, as it asks for additional data to be gathered (see section 5). The same is true for a correction for the Pre-bound effect and/or Performance gap.
4.4 Adjustment factors
Adjustment factors define which part of the calculated energy savings can be attributed to a policy measure or meets the definition of savings specified in the evaluation objectives or reporting requirements (see next section on “Calculating Gross and net savings).
Adjustment factors can concern the Free rider effect, the Spill-over/multiplier effect, Additionality, Non-compliance and Double counting (see here).
With this specific guide, total savings are found by multiplying the unitary savings by the number of actions. The focus of the method Diffusion indicator is on calculating the number of actions due to grants, where the free rider and spill-over effect can be taken into account. A complementary method is used to calculate unitary savings, where the adjustment factors additionality and non-compliance can influence the unitary savings (see table 2 in this link).
If the deemed savings method is used to calculate unitary savings the adjustment factor non-compliance might be relevant, but additionality may not be needed.
Free riders concern a distinction between saving actions due to the policy measure and actions, which would have been taken anyway. The method Diffusion indicators does not provide directly this information, thus other ways must be found, such as a survey among participants to the policy measure about their motivation, or application of Randomized Controlled Trial (RCT) or Quasi-experimental design (see further in this topical case study).
The calculation of the spill-over effect is often difficult due to lack of data to prove a causal relation with the policy measure/sector at hand. Non-compliance, an improper implementation of e.g. insulation measures asks for a sample-wise check of the quality of saving actions.
Finally an adjustment may be needed for double counting, in case of another policy focusing on the same saving actions as evaluated here. But double counting can only be accounted for at a higher level than individual specific guides (see section on Gross and Net savings).
In case of top-down methods adjustment factors can concern autonomous savings (or technological progress) and price-induced energy efficiency progress.
4.5 Calculating Gross and Net energy savings
Gross savings concern the calculated savings from saving actions using a chosen baseline and normalization factors. Net savings concern the savings attributed to policy measures or eligible for a stakeholder (e.g. an energy company with an obligation to realise savings at their customers).
The gross savings can be calculated using the appropriate baseline for number of saving actions and for unitary savings (see section on baselines), and correct the unitary savings for normalization factors (see section on normalization).
Gross unitary savings can include a correction for the rebound effect in case of using the complementary methods deemed savings or engineering estimate (see section on baselines). In case of billing analysis the calculated change in energy consumption already incorporates the rebound effect, but gross savings should be corrected for weather (see section on normalization). No correction for changes in activity level is needed.
Net savings follow from gross savings by applying adjustment factors. Net unitary savings can be corrected for additionality and non-compliance. The net number of saving actions can be corrected for free riders and the spill-over effect (see section on adjustment factors).
The savings should be corrected for the Double counting effect, i.e. the overlap between the savings due to Grants and savings due to other policy measures. The overlap in the calculated savings of both policy measures cannot be processed at the level of an advanced guide, but must be corrected at the level of savings due to overall policy portfolios. For addressing double counting see this link or here.
See also section 8 on concrete examples.
5.1 Main data requirements, data sources and collection technics
Data requirements specified in the table below correspond to the calculation of energy savings, when using the baseline option With/without. Requirements are dependent on the chosen complementary method for calculating the unitary savings (here data for deemed savings).
|Calculation subject||Data requirements||Possible data sources and collection technics|
|Diffusion indicators per saving action type||Yearly number of implemented saving actions per type (or yearly fraction of dwellings with each saving action)||Grant decisions on each saving action type (or penetration rates per saving action from surveys)|
|Deemed savings (if chosen as complementary method)||Estimated savings per type of saving action||Literature (based on earlier measurement, etc.)|
|Normalization factor Rebound effect||Pre-saving and after-saving thermostat setting (in case of no thermostat present no change in temperature setting assumed)||Survey on thermostat setting before and after the saving action(s)|
|Adjustment factor Free riders||Number of saving actions implemented without the grant||Survey on reason to implement the saving action(s)|
|Adjustment factor Non-compliance||Fraction of proper implemented subsidized saving actions||Sample-wise check on implemented saving actions|
Data issues with alternative method
If billing analysis is applied as complementary method to calculate unitary savings, energy consumption data should be made available with/after the saving action and without/before the saving action. These consumption data have to be corrected for weather, but a correction for the rebound is not needed as this is already covered in the with/after consumption.
For more details about the evaluation of net energy savings, see this topical case study.
For possible other methods with different data demands see the section on alternatives for the chosen method.
5.2 Energy savings in final terms or in primary terms
Energy savings can be expressed in final terms or in primary terms; see definitions about primary and final energy here.
This guide can calculate savings in final terms. It can also calculate savings in primary terms provided that savings at end-users are calculated for each energy carrier apart, and primary factors are available to transform the savings in final terms to savings in primary terms.
5.3 Energy savings over time
Implemented saving actions in a year lead to savings over a number of consecutive years. E.g. a more efficient boiler can save gas over its lifetime of about 15 years, insulation over up to 60 years and more efficient computers up to 5 years. For savings from behavioral changes due to a media campaign the life time might be not much longer than that of the campaign. Energy savings can be calculated in different metrics in terms of time reference, for example: year-to-year, annual, cumulated annual, cumulative. See the definitions here.
The calculated yearly savings concern the savings of all new saving actions in that year. In this approach only data for the savings in the chosen year are needed.
Adding up the yearly savings over a period, provided that earlier saving actions are still delivering savings, leads to cumulative savings. For the cumulative savings data are needed for the whole period.
Another cumulative approach, to be applied for the EE directive, is to multiply the (new) savings in a year with the number of years up to a target year and sum this result with that for all other years up to the target year. This cumulative approach stimulates early saving actions, as these count more times to the target than later actions.
Finally, savings from a saving action can be discounted and summed up over the lifetime of the action See link here.
This guide can provide yearly savings of new saving actions in that year. It can also provide cumulative savings provided that data are available over a period.
Cumulative savings according to the Energy Efficiency Directive can be provided when the lifetimes of savings actions are known.
6.1 Alternatives for the chosen method
Often other savings calculation methods can be applied as well, although they will all have pros and cons regarding various aspects dealt with in preceding sections.
The table below presents the pros and cons of the method at hand and for commonly used alternative methods for the same combination of policy measure and sector.
The method at hand uses a combination of diffusion indicators to calculate number of actions and a complementary method (deemed savings) to calculate unitary savings. The alternative method applies billing analysis and is already part of the 30 specific guides.
|Type of method||Pros||Cons|
|Diffusion indicators||Calculation with diffusion indicator and deemed savings relatively simple
Diffusion data from grant decisions
|No check with actual energy consumption changes
Deemed savings to be checked
|Billing analysis (alternative method)||Check with observed changes in energy consumption
Focus method on unitary savings and number of action from grant decisions
|Unclear results with more than one saving action at the same time
No savings per specific savings action > no cost-effectiveness
7.1 Calculating avoided CO2 emissions
Avoided CO2 emissions can be evaluated from the energy savings by applying emission factors. Four key aspects are to be taken into account when choosing the emission factor(s):
- Emission factors vary according to the energy type, so the data about energy savings need to be available per energy type.
- Emission factors for a given type of energy can vary over time (especially for electricity).
- Emission factors can take into account:
- Direct emission factors: the emissions generated when producing the energy used;
- Lifecycle emission factors: all emissions generated from the extraction of the energy resources up to the dismantling of the energy plant.
Due to the differences that the choice of emission factor(s) can induce, it is important to document what emission factor(s) has(have) been used.
For this guide the savings of the different saving actions concern generally only one fuel (mainly natural gas). Therefore, the direct reduction of CO2-emissions can be calculated easily from the savings with an emission factor for the fuel at stake.
The avoided emission of other greenhouse gasses due to energy savings are not taken into account here, as these emissions are generally negligible compared to CO2.
IPCC (Intergovernmental Panel on Climate Change) provides a detailed database of peer-reviewed emission factors.
7.2 Calculating cost-effectiveness
Cost-effectiveness is the ratio between costs to achieve energy savings and the amount of savings and possibly other benefits.
A distinction can be made according to the point of view adopted to assess cost-effectiveness:
- Cost-effectiveness for the end-user or participant
- Cost-effectiveness for society at large
- Cost-effectiveness for the party that takes responsibility for saving targets (government or actor with an Energy Efficiency Obligation)
The calculation of cost-effectiveness for end-users demands, next to the savings, data on investments made, subsidies on investments, interest rates, lifetimes of the saving actions, energy prices (including taxes) per type of end-user and discount factors per type of end-user.
For cost-effectiveness from a societal viewpoint no account is taken of subsidies and taxes, energy prices concern world market price, and a lower value of the discount factors is valid.
For cost-effectiveness from a government perspective the amount of subsidies is related to the (extra) savings of the saving actions due to the grant scheme.
7.3 Calculating other Co-benefits
Possible co-benefits from saving energy concern:
- Extra employment
- Reduction of energy poverty
- Other emission reductions (NOx, SO2, fine particles, etc.)
- Better indoor climate
- Reduced dependency on (insecure) energy import
Due to the detailed calculation approach per saving action type it is possible to determine other co-benefits, provided that specific data become available per co-benefit, e.g. hours spent on insulation measures (employment) or imports per type of energy carrier (import dependency).
7.4 Other aspects of importance
Energy efficiency and savings calculation for countries, regions and cities, ISO 17742, chapter 4 on Indicator based savings calculations, 2015
About Diffusion indicators
- Top-down evaluation methods of energy savings – Summary report, B. Lapillonne et al, EIE 06 128 EMEEES, WI, March 2009
- Annex to the summary report on top-down evaluation methods: ODYSSEE and ODEX indicators that can be used in top-down evaluation of energy savings, Report EIE_06_128 EMEEES, Bruno Lapillonne, ENERDATA, March 2009.
- Definition of ODEX indicators in ODYSSEE database-Ademe, 2014
Relevant case studies
- No case studies available