15 – Evaluating financial incentives for energy efficiency in residential buildings using engineering estimates
This guide can be applied to evaluate the savings due to financial incentives for existing residential buildings using the method engineering estimates. It includes guidance and explanations specific to this combination of types of policy measure, sector and method. As well as links to general guidance and explanations that can also apply to this combination.
1. USE OF THE GUIDE–AUDIENCE, OBJECTIVES AND FOCUS
The primary audience for this guide is energy efficiency programme designers, implementers or supervisors, and evaluators looking for guidance on the evaluation process of energy savings in the scope of this guide.
Although the application of the guide will generally concern the (sub)national level, account will be taken of issues at EU level when relevant (e.g. the specific format of saving figures for the EED, and more particularly its article 7).
This guide is not about the preceding step in the evaluation process, the choice of the method. About this previous step in the evaluation process, see the guidance provided here. However, after presenting the capabilities and limitations of using engineering estimate, alternative methods are presented in section 6.
The objective of this guide is to provide:
- Information on the scope of the guide that enables the user to decide whether this guidel is suited to his/her needs, and whether complementary or additional method(s) could be needed or useful (section 2);
- Guidance about specifying the evaluation objectives and requirements (section 3);
- Guidance about key methodological choices to calculate energy savings (section 4);
- Guidance about the inputs (data requirements) and outputs of the method (energy savings metrics) (section 5);
- Possible alternative methods (with pros and cons) (section 6)
- Background about evaluation results other than energy savings (section 7);
- Relevant examples, case studies and/or good practices (section 8);
- Relevant references for further reading (section 9).
The guide is intended for assessing realised (ex-post) energy savings. However, account is taken of earlier (ex-ante) evaluations of expected savings, if available (see section 4).
The focus of the guide is on impact evaluation, i.e. determining the energy savings, but not on how this has been reached through a step by step process with intermediate results (process evaluation).
Readers looking for the basic and general principles of energy efficiency evaluation may find the following link useful.
2. SCOPE OF THE GUIDE – POLICY, SECTOR and METHOD
2.1 About financial incentives
Financial incentives can take various forms: direct non-refundable aids (grants, subsidies, tax credits), loans with low interest rates, financial guarantees on loans, etc.
More information and examples on the different subtypes residing under the main type (financial incentives) can be found in the MURE database and in the EPATEE Knowledge Base.
Analysis of financial incentives for energy efficiency in buildings can for example be found in (Federici et al., 2018; Maki et al., 2016; Olubunmi et al., 2016; Studer and Rieder, 2019).
The focus of this guidance is restricted to the most common types of financial incentives offered to households or other owners of residential buildings: direct non-refundable aids (grants and subsidies) and loans with low interest rates. However, most of this guidance can also be applicable to the evaluation of schemes using other types of financial incentives (e.g. financial guarantees, third-party financing).
The incentives can be defined as a fixed amount (lump sum) or as a percentage of the investment costs. The incentive amount or rate can depend on the type of action or on the level of energy performance achieved (e.g. the higher the performance, the higher the incentive). The base to calculate the financial incentives may make a distinction between labour costs and costs for materials or products. The scope of costs taken into account is particularly important when assessing the cost-effectiveness of the incentive scheme (see section 7).
Limiting the risks of inflation effect is often critical in the design of the incentives (which can therefore be a subject of evaluation, but is out of the scope of this guidance).
The incentives can also depend on the income level or other criteria related to the dwelling owner or occupant (e.g. eligibility to social benefits). This is especially the case when the objectives of the incentive scheme include alleviating energy poverty. When the design of the scheme considers income levels or social criteria, the evaluation objectives often include the analysis or assessment of distributional effects or effects on energy poverty.
Schemes offering financial incentives often represent significant budget commitments for the state, regional authorities or other public bodies that decide about funding the schemes. These schemes are often implemented by a mandated body (also named “entrusted party” or “implementing public authority” in the EED). This managing body usually monitors the scheme (e.g. number of participants, amounts of funding provided, number and types of actions funded) and reports to the government or other public institutions (e.g. Parliament, Court of Auditors) about the results of the scheme. Cost-effectiveness indicators can thus be given a high importance in the reporting needs.
More detailed information on the evaluation of financial incentives can be found here and in (Wade and Eyre 2015).
2.2 Evaluation for a combination of policy measure types
When financial incentives are combined with other types of policy measures (e.g. tailored energy advice, training programmes for building professionals), it is assumed here that the overall savings due to energy efficiency improvements in buildings are mainly resulting from the financial incentives. However, in practice, the evaluation often encompasses the combined savings effect of all policy measures promoting the implementation of the types of actions in the scope of the evaluation.
This guidance is not capable of attributing part of the (overall) calculated savings to each of the policy measures (see also Double counting in the section on Gross to Net savings).
For discussions about policy mix or comparisons of different types of policy measures, see for example (Boza-Kiss et al., 2013; Murphy, 2014).
2.3 Evaluation when combined with energy taxes
The calculated savings effect for financial incentives will overlap with that of the energy tax. The energy tax indeed increases the cost-effectiveness of the energy saving actions from the point of view of the investors (most often the building owners). Therefore, both policy measures, the financial incentives and the energy tax improve the economic conditions for the investments in energy saving actions. The financial incentives will most often be equivalent to reducing the investment cost. While the energy tax will increase the economic benefit from the actions (by increasing the economic value of the energy savings), thereby reducing the payback time.
This guidance is not capable of attributing part of the (overall) calculated savings to either the incentive scheme or the energy tax.
Distinguishing the effects of financial incentives and energy taxes can be investigated with modelling of investment behaviours, by simulating scenarios with different policy packages. Calibrating this type of model usually requires data about the number of actions implemented in the country (or other area considered for the evaluation).
This type of analysis can be complemented with surveys or modelling about the willingness to pay for energy saving actions (see for example, Collins and Curtis, 2018). Such studies can be helpful for the design of the financial incentives (e.g. identifying the level of incentive that can trigger actions) and to provide a basis to evaluate the additionality of the financial incentives (or the free-rider effects) (see also section 4).
2.4 About existing residential buildings
This guidance is focused on energy saving actions for improving the energy efficiency of existing dwellings. However, most of it is also applicable to the case of new buildings. The main difference lies in the types of baseline that can be used. When using deemed savings for new buildings, the choice of baseline is usually equivalent to the option “minimum energy performance standards” (see Calculation baselines in section 4).
This guidance covers both, individual houses and multifamily buildings. In practice, the incentive schemes can be focused on specific segments of the dwellings stock. For example, to tailor the incentive scheme in order to tackle specific barriers, such as decision processes in co-owned buildings (or condominiums) or split incentives in the case of the rental sector. This guidance does not consider possible differences in incentives according to the types of buildings. This can however affect the way to analyse additionality or free-rider effects, when evaluating additional or net energy savings (see also section 4).
In terms of end-uses, this guidance deals with the thermal end-uses, i.e. HVAC (heating, ventilation and air conditioning), and domestic hot water. Energy saving actions in the scope of this guidance thus includes actions improving the energy performance of the building envelope (e.g., wall insulation, roof insulation, windows replacement), heating and cooling systems (e.g., boiler replacement, hydraulic balancing of the heat distribution system), ventilation systems (e.g. heat recovery ventilation) or the system for domestic hot water.
The guidance does not deal with actions on electrical end-uses, such as appliances and lighting. About savings from financial incentives for appliances, see Specific Guidance 14 .
Financial incentives for the energy renovation of residential buildings can be to encourage single actions (e.g. wall/roof/floor insulation, boiler replacement) or deep renovations (i.e. aiming at improving the global energy performance of the buildings). In most cases, the incentives are conditional upon minimum energy performance requirements. The incentive scheme may also require other conditions, such as quality label or installation by a qualified building professional.
In practice, the use of engineering estimates is preferred when evaluating policies or programmes promoting deep renovations. The engineering calculations can then take into account the interactions between the different energy saving actions implemented (e.g. insulation and replacement of the heating system). Engineering estimates can also be used for single actions, especially when the scheme includes a systematic data collection about technical parameters of the actions. See section 6 for a comparison with alternative methods.
Information on (sub)sectors defined in the Toolbox can be found here, chapter 2, p.17 .
2.5 Evaluation for cross-sector saving actions
Most of this guidance can also apply in principle to schemes promoting the renovation of tertiary buildings (service sector). One major difference is that tertiary buildings can be much more diverse than residential buildings, as it encompasses many sub-sectors (e.g., offices, hospitals, stores, schools).
The engineering calculations might then need to be based on more detailed modelling.
Moreover, it is more common for tertiary buildings than for residential buildings to have their HVAC systems operated with BAS (Building Automation System) that can provide detailed data of energy consumption together with data for other relevant variables such as temperature setpoint or patterns of use of the buildings (e.g. working days vs. week-ends). For more guidance about cases where data from BAS can be used, see for example (Romberger, J. 2017). And more generally for more guidance about cases when data can be collected through an energy management system, see for example (Stewart, 2017).
2.6 About engineering estimates
The Energy Efficiency Directive (2018(2002), Annex V(1.c)) relates engineering estimates to “scaled savings”. They are the results of calculation formulas or models based on physics’ principles (e.g. about heat transfers). These calculations can go from very simplified formulas up to very detailed and sophisticated building energy modelling, depending on the data available and the evaluation objectives (especially in terms of accuracy and factors to take into account).
The main difference with deemed savings is that engineering estimates are using data about the buildings and the energy saving actions that are specific to the cases under evaluation. While deemed savings are mostly based on reference values, and define average energy savings ratios (see Specific Guidance 9 for more details about deemed savings).
In practice, engineering estimates often correspond to the results from Energy Performance Certificates (EPC) or similar simplified modelling. Some incentive schemes can indeed require the participants to provide an EPC after the renovation works (thereby providing the data needed for the engineering estimates). Similarly, some schemes require an energy audit before the application for the incentive can be approved.
Whenever possible, it is recommended to calibrate the formula or model used, based on metered energy consumption on samples of buildings. Similarly, it is recommended to perform sensitivity analysis to test the main assumptions used in the calculations. For more details about the use of engineering methods, see for example the section on engineering analysis in the Chapter 6.4 of (TecMarket Works et al., 2004).
About issues raised on the use of engineering estimates (especially about sources of uncertainties), see for example (Sipma et al. 2019).
Information about the various evaluation methods can be found in this EMEEES report, or here, table 1 and 2. This source also covers the combination of the method at hand with other methods, which will be dealt with below.
2.7 Complementary methods to determine total savings
Complementary methods are methods that are required, in addition to the primary selected method (here engineering estimates), to calculate total energy savings.
Engineering estimates are meant to calculate unitary savings. In case of actions in the residential sector, the unit can be a participant (household), an equipment (e.g. boiler), a dwelling, a m² of area insulated, etc. For example, when dealing with deep renovations, the unit will most often be the dwelling or building.
These unitary savings then need to be multiplied by the number of actions or participants in order to have the calculated total savings. The number of actions or participants can be obtained in various ways. See this link, table 2 and 3.
In the case of an incentive scheme, the number of actions is usually directly available from the monitoring system, as the participants need to submit an application file, or at least to provide a minimum set of information, in order to receive the financial incentive.
This guidance does not deal with monitoring systems. For more details about the links between monitoring and evaluation, see the topical EPATEE case study about the links between monitoring and evaluation (Maric et al., 2018).
It should be noted that the data collected by the monitoring system might be sufficient to get the number of participants or actions, but that it might not collect systematically the data needed to calculate the engineering estimates.
When it is not possible to collect the data along the implementation of the scheme (e.g. if requiring data from participants could decrease participation rate or significantly increase administration costs), then an alternative is to collect the data by surveying samples of participants. Particular care should then be taken in the sampling method, so that the results can be extrapolated from the samples to the whole participants. For more details about surveys, see for example (Baumgartner, 2017). And about sampling, see for example (Khawaja et al., 2017).
2.8 Additional methods to increase reliability of the results
An additional method can be applied on top of using engineering estimates to improve the reliability of the evaluation results and/or the cost-effectiveness of the evaluation approach.
Engineering estimates are the result of calculations or modelling that, by essence, provide a simplified representation of the reality. It is therefore possible that they do not reflect all the specificities of the cases under evaluation, and especially the specific behaviours of the occupants. Indeed, engineering methods commonly use standard assumptions about occupants’ behaviours (e.g. in terms of indoor temperature, number or rooms heated, night intermittency, duration of the heating season).
The energy savings calculated with engineering estimates thus reflect above the technical improvement of the energy performance of the building, which might be different from the actual changes in energy consumption (see also section 4 about normalization factors).
Depending on the evaluation objectives, other methods can be used to investigate the reliability of the energy savings (e.g. verifying the relevance of the data or assumptions used) or assess actual energy savings, as for example illustrated in the table below.
|Type of method||Short description||Objective|
|Survey of participants||Surveys can be used to collect data complementary to the ones collected by the monitoring system (e.g. about occupants’ behaviours)
For more details, see for example (Baumgartner, 2017)
|To verify / update the assumptions used in the engineering estimates|
|Direct measurement||Direct measurement can be done on a specific parameter (e.g. sensors to monitor duration of use), or on the energy consumed by a given equipment or process (e.g. sub-metering).
For more details, see e.g. (Mort, 2017)
|To verify / update the assumptions used in the engineering estimates|
|Billing analysis||Calculation of energy savings from metered data of energy bills.
For more details, see Specific Guidance 8
|To assess actual energy savings (as can be experienced by the end-users), and to update assumptions used in the engineering estimates|
As the number of actions or participants monitored for an incentive scheme is usually very large, this could be costly to apply these additional methods to the whole actions or participants. Therefore, a common practice is to apply these methods on samples of actions or participants. As sampling can create bias and uncertainties in the results, a particular attention should be paid to limit sampling bias and obtain samples as representative as possible. For more details about sampling, see for example (Khawaja et al., 2017).
Combining different methods can be a cost-effective approach to increase the reliability of the savings figures and to get a better understanding of the impacts of the energy saving actions and the incentive scheme.
For possible combinations with an additional method see also chapter 6 here.
3. EVALUATION OBJECTIVES and REQUIREMENTS
3.1 Meeting evaluation goals and ambition
Typical objectives of using engineering estimates can be:
- Getting savings estimates along the implementation of the actions (limited time-lag);
- Taking into account the diversity of actions and context of implementation;
- Getting savings estimates that reflect the specificities of each action and technical energy efficiency improvements;
- Providing visibility to building owners or occupants about the energy savings they can expect from different packages of actions (for example, when the incentive scheme requires an energy audit to be performed before deciding of the renovation works).
At the opposite, engineering estimates are not the most appropriate method in case the primary evaluation objective is to assess actual energy savings (as experienced by the end-users) or to take into account possible changes in users’ behaviours.
The table shows whether this guide can be used to report on general evaluation goals or criteria. See also this document.
|General types of evaluation goals or criteria||Level of ambition||Remarks|
|Calculation of realized energy savings from saving actions||Medium||Depending on the quality of the engineering estimates, and especially the factors taken into account in the calculations.
In any case, complementing engineering estimates with a method to verify actual energy savings is recommended for this objective (see above in section 2)
|Calculation of energy savings attributed to the EEO||Low to medium||Additionality criteria (e.g. about defining the baseline) or default adjustment factors can be included in the calculations to make that engineering estimates correspond to additional savings.
It is recommended for this objective to complement engineering estimates with further ex-post analysis (see below in section 4)
|Cost-effectiveness of saving action (for end-users)||Medium to high||Engineering estimates can reflect the characteristics of the actions and conditions of implementation specific to each case.
When the calculations can be calibrated or benchmarked, they can offer a reliable estimate for an investment plan.
As the calculations have to be done for each case, cost data and assumptions about savings lifetime are usually available to or made by the company or consultancy making the calculations.
|Cost-effectiveness of the EEO (from a society’s perspective)||Medium||Depending on the quality of the engineering estimates (see previous line).
Further assumptions about savings lifetime might be needed.
(see below in section 7)
Whenever possible, it is also recommended to complement engineering estimates with ex-post verifications of savings (see above in section 2)
|CO2-emission reduction from saving actions||Medium||The basis will be the estimated energy savings.
As engineering estimates are calculated specifically for each action or project, this will generally make it easy to apply standard emission factors to calculate CO2 savings from energy savings.
(see below in section 7)
|CO2-emission reduction attributed to the policy measure(s)||Low to medium||The basis will be the estimated energy savings (see previous line).
Additionality criteria (e.g. about defining the baseline) or default adjustment factors can be included in the calculations to make that engineering estimates correspond to additional savings.
(see below in section 4 and in section 7)
3.2 Reporting expectations
As engineering estimates are calculated for each specific case, they can be used at the level of a renovation project (e.g. to inform the building owners or occupants, to provide a basis for an energy performance contracting), or at the policy level. The calculations at project level are usually done for M&V purposes. For more details and guidance about the project level, see for example the option C of the IPMVP (EVO, 2016), or (Jacobson, 2017).
This guidance deals with the policy level, i.e. the evaluation of energy savings from an incentive scheme. In this case, engineering estimates are used to provide results for a group of participants, and whenever possible for the whole participants.
Engineering estimates can be reported in a variety of energy savings (or even CO2 savings) metrics. The choice of the metric(s) will mostly depend on the evaluation objectives and the data available for the evaluation.
The table below discusses possible options of metrics.
|Nature of the objective||Energy savings
|Choice mostly depending on the primary objective of the incentive scheme. The results can be reported in several metrics when needed.
(for more details about CO2 savings see section 7)
|Duration for which the results are counted||Annual (or first-year)
Cumulative over the obligation period
|Choice depending on the objectives of the incentive scheme, and on the diversity of actions possible (cf. heterogeneity or not in action lifetimes). A “lifetime cumulated” unit can be chosen to value long-lifetime actions that are often the priority when dealing with energy efficiency in buildings. However, this requires to estimate energy savings over long durations, thereby increasing uncertainties (for more details, see Energy savings over time in section 5).|
|Energy basis (if nature = energy savings)||Primary energy savings
Final energy savings
|Choice depending on the objectives and scope of the incentive scheme. For example, primary energy savings can be chosen if the scope includes actions dealing with energy switching (change in the energy source), or if the priority is on electricity savings. Final energy savings can be chosen for example if the focus is on reducing end-users’ energy bills (e.g. to alleviate energy poverty).
About specific data needs related to assessing final or primary energy savings, see section 5.
|Energy unit (if nature = energy savings)||PJ / ktoe / TWh / …||Choice usually depending on the energy unit commonly used in the country (e.g. for the national energy balance), or on the objectives or scope of the incentive scheme. For example, ktoe can be used if the priority is to save fossil fuels. TWh can be used if the priority is on electricity savings.|
|Evaluation perspective||Gross / Additional / Net||For more explanations, see Calculating Gross and Net energy savings in section 4.|
Engineering estimates can be first calculated in the metric chosen according to the main policy objective, and then converted in other metrics for other reporting purposes (e.g. reporting to European Commission for the article 7 of the EED), or for communicating to different audiences (e.g. bill savings might be more explicit when communicating to the general public or to buildings’ occupants).
Whatever the energy savings metrics used, it is important to keep the documentation of the engineering estimates. The details of the calculations will indeed be needed to convert the results from one metric to the other. Guidance to document engineering estimates can be found in the standard ISO 50046.
3.3 Time frame for evaluation
The period under evaluation depends on the evaluation objectives and on the current context of the incentive scheme. If the financial incentives have been changed (e.g. changes in incentive rates, in the list of eligible actions) at different points in time, then it can be interesting to evaluate separately each period between changes. The period under evaluation can also be defined according to the budget years of the scheme, for example when results need to be reported to the Parliament or to the Court of Auditors.
One of the main advantages of engineering estimates is that it enables to assess energy savings with a limited time lag. There is no need to wait for energy bills after the renovation works. The evaluation can start as soon as data about the renovation projects completed are available from the monitoring system. This is one of the reasons why engineering estimates can be chosen when the objective is to evaluate results on a regular basis (e.g. each year) or to get a quick feedback loop to detect if adaptations are needed.
It should be noted however that the time needed to obtain the engineering estimates strongly depends on the monitoring system. If the monitoring system allows a systematic data collection from the participants, and that this includes the data needed to calculate engineering estimates, then the energy savings can be evaluated quickly. This is for example the case for the Italian tax credit scheme that uses an online platform to collect technical details from the participants on an on-going basis (and the participants need to submit these data for their application to be processed).
In other cases, the monitoring system does not provide directly the data needed to perform the calculations. Then further data collection is needed, for example through a survey of a representative participants. This is for example the case for the programme of the KfW Bank for the renovation of dwellings in Germany. The evaluators have to do a new survey of participants every year to collect the data needed to run their models. This makes that the results are usually available with a time lag of one year (results for year n available in year n+1).
The use of engineering estimates can be complemented by other methods to verify their reliability, and possibly update them (see Additional methods to increase reliability of the results in section 2. ). These additional methods have different timeframes. It is thus recommended to consider how the combination of evaluation activities should be planned. Especially to ensure the feasibility of the corresponding data collection and to optimize the use of resources (time and budget).
About evaluation planning, see also general guidance here.
3.4 Expertise needed for chosen method
The use of engineering estimates requires expertise about the following:
- Technical expertise about the energy efficiency actions eligible to the incentive scheme (e.g. about building components (building envelope) and HVAC (Heating, Ventilation and Air Conditioning) systems): this is needed to select the essential parameters to be included in the calculation formula or model.
- Expertise with building energy modelling, and especially with calibrating models.
- Expertise about data or trends in energy consumption in dwellings and related energy efficiency markets: this might be needed to define the baseline.
When the monitoring system cannot provide all the data needed for the evaluation, then expertise about survey and sampling methods can be required to prepare the complementary data collection.
4. KEY METHODOLOGICAL CHOICES FOR CALCULATION OF ENERGY SAVINGS
General principles of calculating realized savings using different methods can be found for example in the EMEEES bottom-up methodology and review of evaluation methods.
This section deals with key methodological choices to be considered when calculating energy savings: consistency between ex-ante and ex-post evaluation, baseline, normalization and adjustment factors. These choices are important to document when reporting energy savings, to ensure the transparency of the results.
4.1 Matching method with earlier ex-ante evaluation
From the viewpoint of methodological consistency and data availability using the same method in the ex-ante evaluation and in the ex-post evaluation is recommended.
In the case of an incentive scheme, the ex-ante evaluation can take the form of an impact assessment of the scheme, including assumptions about participation rates. An impact assessment can indeed be required by law for any scheme committing public budget (possibly depending on a threshold of budget per year). More generally, an impact assessment is commonly used to size the scheme, and particularly to define quantitative objectives and estimate the budget needed. Whenever possible, the impact assessment can include several scenarios to compare impacts according to distinct alternatives (e.g. in terms of incentive rates, list of eligible actions, eligibility criteria for participants, etc.).
The ex-post evaluation will then aim at assessing if the objectives have been met, if differences occurred in the mix of action types installed by the participants (compared to the expectations), etc.
Engineering estimates can be used for both, the ex-ante and the ex-post evaluation. Respectively to set the objectives and to verify their achievement. When it is not the case, it is important to use the same principles when defining the baseline(s). When engineering estimates are used for ex-ante evaluations, it is usually combined with building stock modelling to simulate different policy scenarios.
If engineering estimates do not provide an acceptable combination with the method applied earlier for the ex-ante evaluation, it might be useful to select another method (see examples of alternative methods in section 6).
In practice, ex-ante and ex-post evaluations are applied consecutively. The ex-post evaluation builds on an ex-ante evaluation that made use of data coming from previous ex-post evaluation or studies (e.g. about previous periods of the same incentive scheme, or about the same types of energy saving actions as the ones promoted by a new policy measure). These previous ex-post studies could have used another type of method as well.
As discussed in section 2 (see Additional methods to increase reliability of the results), additional method(s) can be used to verify ex-post the energy savings initially estimated with engineering estimates. In that case, the results from these ex-post verifications can be used to improve the engineering estimates for future ex-ante or ex-post evaluations. When doing so, the representativeness of the results from ex-post verifications should be analysed to see if these results can be used for updating the assumptions used in the engineering calculations (see for example normalization factors in section 4). It is important here to distinguish ex-post verifications done as part of controls (e.g. to mitigate risks of frauds), and ex-post verifications done as part of an evaluation.
Controls are commonly decided on a risk-based approach, in order to focus the efforts of control where the risks are higher or more critical. Therefore, results from controls are not meant to be representative. At the opposite, evaluation methodologies are usually designed to look at the whole scheme, taking into account representativeness, risks of sample bias, etc. Depending on the evaluation objectives, evaluation results can thus be expected to be representative. This is why data from verifications done as part of a M&V system might not always be usable for an ex-post evaluation (or under certain limitations).
For a general discussion about possible combinations of methods applied ex-ante and ex-post, see chapter 7, in this document.
4.2 Calculation baselines
Gross energy savings are defined in general as the difference between the situation including the implementation of energy saving actions and a reference situation without the saving actions. This reference situation can be defined using various calculation baselines. For further background see further here.
When evaluating the energy savings from an incentive scheme, the baseline can also be defined to represent what the situation would have been in the absence of the incentive scheme. This is in particular linked to the concept of additionality, as defined in the amended Energy Efficiency Directive (EU2018(2002)):
“To determine the savings that can be claimed as additional, Member States shall have regard to how energy use and demand would evolve in the absence of the policy measure in question by taking into account at least the following factors: energy consumption trends, changes in consumer behaviour, technological progress and changes caused by other measures implemented at Union and national level” (Energy Efficiency Directive 2018(2002), Annex V(2) point (a)).
Depending on the baseline option chosen, the energy savings can therefore mean to be gross or additional savings. Then gross energy savings estimated with engineering estimates can also serve as basis to evaluate net energy savings, when combined with further analysis (see below Calculating Gross and Net energy savings).
The table below specifies what type of energy savings can be obtained according to the baseline option.
|Baseline option||Explanation||Type of calculated energy savings|
|“before” situation (energy consumption before the renovation works)||This baseline corresponds to a before/after comparison.
|Gross energy savings
Additional or net energy savings can then be calculated by applying adjustment factors (see below).
In special cases, this option can also be used to calculate additional energy savings (see discussion below).
|Market average||This baseline corresponds to a with/without comparison, where it is assumed that the market average represents the counterfactual (what the participants would have done in the absence of the incentive scheme).
Usually the market average first provides the baseline level of energy performance for the type of product or equipment installed by the participants. This parameter is then combined with other data (e.g. reference values for dwelling characteristics) to calculate the baseline energy consumption.
|Additional energy savings
(i.e. savings additional compared to a business-as-usual scenario represented by the market average)
|Trend scenario||This baseline corresponds to a with/without comparison, where the counterfactual is estimated based on extrapolations of previous trends in the energy saving actions promoted by the incentive scheme trends (e.g. number of renovation works and related performance levels), or in energy consumption of the stock of dwellings targeted by the scheme.
The trend scenario then provides either baseline levels of energy performance per action type, or baseline energy consumption per dwelling type.
|Additional energy savings
(i.e. savings additional compared to a business-as-usual scenario based on extrapolation of previous trends)
|Minimum efficiency standards||This baseline corresponds to a particular type of with/without comparison, commonly used to assess savings additional compared to current regulations.
The minimum efficiency standards usually provide the baseline level of energy performance either for the type of product or equipment installed by the participants, or for the overall energy performance of the dwelling (e.g. in kWh/m².year). This parameter is then combined with other data (e.g. reference values for dwelling characteristics) to calculate the baseline energy consumption.
|Additional energy savings
(i.e. savings additional compared to current regulations)
The option “market average” or “minimum efficiency standards” are more commonly used when the scheme promotes single actions (or predefined action packages). It cannot be used for schemes promoting deep renovations. In this case, the option “trend scenario” can be an alternative, especially when labels about energy efficiency renovations are in place and can provide a proxy to estimate the trend in deep renovations.
When the implementation rate of a type of action or renovation project is low, a simplifying assumption can be to neglect the few actions that would have been done anyway and therefore use the “before” situation as a baseline to calculate additional energy savings (i.e. gross and additional savings are the same in this case). This is for example a case considered in the in the amended Energy Efficiency Directive (EU2018(2002)) for the renovation of existing buildings (see paragraph b in Annex V(2)).
Another particular case can be when a pilot phase has been used to specify the incentive scheme, and that it was possible to make an ex-post evaluation of the energy savings during this pilot phase. As recommended by the E2e initiative (Evidence for Action on Energy Efficiency) in the US, it can then be possible to use a RCT (Randomised Controlled Trials) method to evaluate net energy savings. The RCT method uses a comparison between a control group (not subject to the pilot scheme) and a participants’ group (subject to the pilot scheme), where individuals (e.g. building owners or occupants) are taken randomly in the population and randomly assigned to one of the two groups. The population means here all the persons who will be eligible to the scheme once it will be implemented in full scale.
The RCT method usually calculates net energy savings, as the difference observed between the two groups in their changes in energy consumption (method also named Difference-in-Differences). The baseline is then a combination of before situation and control group.
For more details about the RCT method, see Specific Guidance 8 .
If this approach has been used to define the engineering estimates (e.g. by including a special additionality factor), then the ex-post evaluation (done after the scheme is implemented on full scale) can be used to investigate the possible differences between the pilot phase and the full-scale phase of the scheme, to analyse if the assumptions about additionality as defined from the pilot phase are still relevant to evaluate savings from the full-scale phase.
For a general discussion about calculation baselines, see this document.
4.3 Normalization factors
The calculation of engineering estimates is usually made based on normalized (or conventional) conditions of use. The types of energy saving actions considered in this guidance are related to space heating or cooling. This means that the calculation is made taking into account normalized weather conditions (e.g. reference Heating Degree Days, and possibly Cooling Degree Days, as defined in building regulations).
The normalized weather conditions are thus an input data of the calculation (even most commonly directly integrated in the calculation model). When using this approach, another input of the calculation will be normalized assumptions about heating behaviours (e.g. indoor temperature per type of rooms, night and absence mode). This means that the engineering estimates will be based on conventional energy consumption.
In that case, it is recommended to consider if it would be relevant to apply factors to take into account prebound and rebound effects:
- prebound effect: lower baseline energy consumption compared to the estimated conventional energy consumption. This can occur for example in very inefficient dwellings, for which a normalized use of space heating would not be affordable for the occupants. This can also occur when the statistics about the building stock do not take into account the fact that a share of the dwellings can have been renovated since their construction.
- rebound effect: higher energy consumption than the estimated conventional energy consumption for the situation after implementing the energy saving action(s). This can occur for example when the energy efficiency improvement makes affordable for the occupants to increase the indoor temperature, heat more rooms, or heat for longer periods.
Prebound effect can for example be derived from comparisons between conventional energy consumption from Energy Performance Certificates and metered energy consumption. Prebound effect can also be considered in the calibration of the calculation model. Experience shows that the prebound effect is influenced by the level of energy performance of the buildings before renovation works. It can therefore be relevant to consider different values for a “prebound factor” according to ranges of energy performance (e.g. based on energy classes used for Energy Performance Certificates).
Rebound effect can for example be derived from previous surveys, measurement campaigns or the literature. Similarly to the prebound effect, experience shows that the extent of the rebound effect might vary according to the level of energy performance achieved after renovation works.
For more details about prebound and rebound effects, see for example:
- the topical case study about the comparison between estimated and measured energy savings (Sipma et al., 2019);
- the experience sharing webinar about “How and what can we learn from verifying energy savings first estimated with engineering calculations?”;
- examples of studies who investigated these effects (Hong et al., 2009; Milne and Boardman, 2000 ; Raynaud et al., 2014).
It should be noted that prebound and rebound effects can be analysed with different perspectives according to the main policy objectives. For example, if the main objective is to reduce the absolute energy consumption in dwellings, then rebound effect will be perceived as a negative effect. At the opposite, if the main objective is to alleviate energy poverty, then part or all of the rebound effect can be perceived positively, as an improvement of the thermal comfort for the occupants. Which can then be connected to co-benefits such as improved indoor air quality and sanitary conditions, and thereby reductions in health expenses.
About the energy consumption for the situation after installing the energy saving action(s), another factor might need to be taken into account: possible performance gaps.
Performance gaps correspond to cases where the observed energy performance of the energy saving action(s) installed is lower than the expected energy performance. For example about insulation actions, because the insulation materials have not been installed properly (e.g. leaving thermal bridges) or the insulation materials have lower performance than claimed. About condensing boilers, an example can be when settings and conditions of use (e.g. types of radiators and related temperature for hot water) do not enable the boiler to condense and recover the heat from steam water in flue gas.
Enforcing quality requirements can help to minimize the risks of performance gaps.
Performance gaps can for example be derived from previous studies on samples of actions (e.g. through on-site inspections).
An example of use of correction factors to take into account rebound effect and performance gaps can be found in the case study about the UK Supplier Obligation. See in this file the explanations about the “in-use factor”.
4.4 Adjustment factors
Adjustment factors define which part of the calculated energy savings can be attributed to a policy measure or meets the definition of savings specified in the evaluation objectives or reporting requirements (see next section on “Calculating Gross and net savings).
For a general introduction about adjustment factors, see table 1 here.
When using engineering estimates, adjustment factors can concern the Free rider effect and the Spill-over/multiplier effect. For related definitions, see this document. Free-rider and spill-over effects can also be encompassed in a single additionality factor.
These adjustment factors are used when the evaluation objective is to assess net energy savings. In that case, they mostly apply when the chosen baseline option is the “before” situation used for a before/after comparison.
They can also be applied when another baseline option has been chosen. But then, they should be defined to avoid overlaps with the effects already taken into account in the definition of the baseline. For example, if the baseline option is “minimum efficiency standards”, then the scope of free-rider effect taken into account in the adjustment factor will be focused on the number of actions, and will not deal with the performance level (as this aspect is already assumed to be taken into account in the baseline).
These effects can be taken into account in engineering estimates by including factors defined from previous studies or surveys. For an example, see the EPATEE case study about Danish EEO scheme.
It should be noted that in most of the available experience (especially in Europe), adjustment factors have been defined to take into account free-rider effect only. Spill-over effects have indeed proven to be more difficult to assess quantitatively.
Adjustment factors can also be applied afterwards, when the ex-post evaluation includes further analysis to assess them specifically for the period under evaluation (e.g. through dedicated surveys of participants and non-participants). Then other methods can be used. For more details, see the EPATEE topical case study about assessing net energy savings (Voswinkel et al., 2018).
In case another policy can overlap with the incentive scheme (promoting the same types of energy saving actions), then it is needed to consider the risks of double counting. Methods to tackle double counting or interaction between policy measures go beyond the scope of this guidance.
4.5 Calculating Gross and Net energy savings
Gross energy savings are energy savings calculated from the point of view of the final consumers, i.e. independently of whether the participants to the incentive scheme would have acted the same or differently in the absence of the scheme. The baseline energy consumption used to calculate gross energy savings is the energy consumption before the energy saving actions were installed or implemented.
The calculation of gross energy savings can include the use of normalization factors to ensure that the baseline energy consumption and the energy consumption with the energy saving action are comparable. For the residential sector, this mainly concerns weather conditions (see Normalization factors above). When relevant and feasible, the occupancy rate can also be useful to consider.
Depending on the data and approach used to calculate gross savings, it can be relevant to use also correction factors for prebound and rebound effects, as well as performance gaps (see Normalization factors above). These effects indeed affect energy savings from the point of view of the final consumers.
Net energy savings are calculated from the point of view of the public authority or other stakeholder that provides funding to the incentive scheme. Therefore, this calculation takes into account effects related to the causality or attribution of the actions or energy savings to the incentive scheme.
In the context of the EU Energy Efficiency Directive, it is also common to speak about additional savings instead of net savings, due to the terminology of the EED article 7. At national level, the concept of additional savings can also be used when the incentive scheme is designed to have effects additional to current regulations, or more generally to trigger savings that are additional compared to a baseline scenario defined to identify the needs in energy efficiency policies.
Two main approaches are possible when using engineering estimates:
- Take into account additionality in the definition of the baseline (option presented above in Calculation baselines).
- Apply adjustment factors to gross energy savings (option presented above in Adjustment factors).
Additional methods can then be used to assess net or additional savings ex-post (e.g. when evaluating the latest period of the incentive scheme). Explanations and guidance about methods for this type of ex-post evaluation can be found in the EPATEE topical case study about assessing net energy savings (Voswinkel et al., 2018).
Additional or net savings should also be corrected for double counting, in case of possible overlap between the incentive scheme and other policy measures. The overlap in the calculated savings should be analysed at the level of the overall policy portfolio or sector. For addressing double counting see the IEA evaluation guidebook or the EMEEES bottom-up methodology.
5. INPUT AND OUTPUT
5.1 Main data requirements and data sources and collection technics
Data requirements specified in the two tables below correspond to the calculation of gross energy savings, when using the baseline option [before/after], i.e. a baseline representing the situation before energy saving actions are implemented. The case about using baseline options to calculate additional energy savings is discussed later on (see Data issues when evaluating net energy savings).
The table describes the input data commonly needed for the engineering calculations. The level of details can vary according to the type of calculation formula or model used. It is therefore important to clarify what formula or model will be used when planning the evaluation, and whenever possible from the outset of the incentive scheme, simultaneously to the design of the monitoring system.
|Calculation subject||Data requirements||Possible data sources and collection technics|
|Size and shape of the building||Heated area (or when not available useful floor area), shape coefficient (area of the building envelope against the volume of the building), ratio of glass surface for the external walls, type of roofs (e.g. flat, sloping), etc.
(the data needed depend on the formula / model used)
|Data reported by the participants or their contractors (e.g. from Energy Performance Certificates, energy audits or alike) in their application file
Survey of participants
Databases of standard building design according to construction period and building type
|Energy performance of the building envelope (heat transfer coefficient: U-values in W/m².K; or thermal resistance: R-values in m².K/W)||Values before and after the renovation works for the building components taken into account in the calculations||Data reported by the participants or their contractors (e.g. from Energy Performance Certificates, energy audits or alike) in their application file
Survey of participants
Databases of standard values per construction period and type of buildings (for “before”) and per type of actions, from manufacturers’ data (for “after”)
|Efficiency of the heating system||Data reported by the participants in their application file
Survey of participants
|Data reported by the participants or their contractors in their application file
Survey of participants
Databases of efficiency per type of heating system (or from manufacturers’ data)
In some cases, the detailed data will have be used by an energy auditor or advisor to prepare the Energy Performance Certificates (EPC) or energy audits. In that case, the evaluation of the incentive scheme can calculate directly the energy savings as the difference of energy consumption before and after renovation works, as provided in the EPCs or energy audit reports. Such approach has been subject to criticisms, as the reliability of the energy savings then depend strongly on the quality of the EPCs or energy audits. For practical examples where these criticisms are discussed, see for example (Sipma et al., 2019) or the EPATEE case study about the WAP (Weatherization Assistance Program) in the US.
It is therefore recommended to keep the detailed data in the documentation about the renovation projects and the respective calculations. So that these calculations can be verified, and corrected/improved when needed.
The table below describes complementary data needed to take into account normalization and correction factors.
|Calculation subject||Data requirements||Possible data sources and collection technics|
|Normalization factor for weather condition||Normalized Heating Degree Days (per climate zone or location)||Heating Degree Days specified in building regulations, according to climate zones.
Data from the national meteorological office or weather service, according to the ZIP code of the building.
|Correction factors (e.g. to take into account prebound and rebound effects, performance gaps)||To be considered if the calculation model could not be calibrated based on statistics of metered energy consumption.||Available studies on these effects.
Data from the literature.
Default values based on conservative assumptions.
Engineering estimates are defined for a unit of action (see About deemed savings in section 2). Therefore, a complementary source of data is needed to obtain the number of actions (or other units of actions): see Complementary methods to determine total savings in section 2.
Data issues when evaluating net energy savings
In the context of the article 7 of the EU Energy Efficiency Directive, it is more common to speak of additional energy savings than net energy savings. When using engineering estimates, additionality has to be taken into account in the calculation assumptions (see Calculating Gross and Net energy savings in section 4).
- Take into account additionality criteria in the definition of the baseline (option presented above in Calculation baselines).
- Apply adjustment factors to gross energy savings (option also presented in Adjustment factors).
When using the approach of taking into account additionality criteria in the definition of the baseline, the following data may be needed:
- for the baseline option “market average”: data about trends or recent market shares per energy class (or similar categories reflecting energy performance levels) to calculate the market average for energy consumption or energy performance characteristics (in case energy consumption is calculated with intermediate parameters).
- for the baseline option “minimum efficiency standards”: data about the current minimum energy performance requirements set in national or European regulations, per action type (and information about the scheduled updates of these regulations).
The corresponding values might need to be updated on a regular basis or case-by-case (e.g. to take into account the most recent years to define the market average, or changes in the current regulations about minimum efficiency standards).
When using the approach to apply adjustment factors to the gross savings, the data needed to define these factors can be obtained from previous studies (e.g. previous surveys or market analysis) or the literature, or through complementary ex-post studies.
When using data from previous studies, it is recommended to use as far as possible data from studies on the same policy measure. Literature indeed shows that values for adjustment factors can vary significantly from one policy measure to another, one country from another, etc.
For more details about the evaluation of net energy savings through ex-post studies, see the corresponding EPATEE topical case study (Voswinkel et al., 2018).
5.2 Energy savings in final terms or in primary terms
Energy savings can be expressed in final terms or in primary terms. See definitions about primary and final energy here.
Engineering estimates can equally be calculated in final or primary terms. One advantage of choosing to calculate in final energy is that it enables to compare engineering estimates with energy savings obtained from metered energy consumption (e.g. energy bills).
Calculations in primary energy savings are often used when actions can include energy switching or on-site cogeneration (CHP). Primary energy savings can also be used to reflect the energy savings taking into account the whole supply chain (and particularly the losses along the electricity supply).
Engineering estimates can also be converted in monetary savings, i.e. savings on energy bills (see Reporting expectations in section 3). This requires to define average energy prices per energy carrier, as well as a scenario of energy prices if the savings are calculated in cumulative terms, e.g. over the action lifetime. In this case, a discount rate can also be applied to the calculation (see below Energy savings over time).
For consistency, the metrics should be the same for setting the objectives of the incentive scheme and then evaluating the achievements. The energy savings results can be reported in different metrics according to the reporting purposes (e.g. reporting in the context of the EED article 7), provided that the data needed to convert from one metric to the other is available. It is thus important to identify from the outset the needs to report the results in different metrics, so that data used in the energy savings calculations are documented enough to enable future conversions.
In any case, the metric used to calculate the energy savings should be made explicit when presenting the results.
5.3 Energy savings over time
Implemented saving actions in a year lead to savings over a number of consecutive years. E.g. a more efficient boiler can save gas over its lifetime of about 15 years, and insulation over up to 60 years. For savings from behavioural changes (e.g. if the incentive scheme promotes energy display devices with tailored energy advice) might be not much longer than the period of the behavioural intervention (e.g. period over which tailored advice or feedback is provided to the occupants).
Energy savings can be calculated in different metrics in terms of time reference, for example: year-to-year, annual, cumulated annual, cumulative. See the definitions here.
The incentive scheme can count first-year savings only, or up to lifetime-cumulated savings (i.e. savings over the lifetime of the energy saving action).
If only first-year savings are counted and needed for reporting, then no further data is needed, apart from monitoring when the actions are installed (data usually collected as part of the process to pay the incentive).
In other cases, the data about when the actions are installed will need to be complemented with data about the estimated lifetime of the savings.
Examples of lifetime values can be found in the following sources:
- CWA 15693:2007. Saving lifetimes of Energy Efficiency Improvement Measures in bottom-up calculations. CEN Workshop Agreement, April 2007.
- EN 15459:2017. Energy performance of buildings — Economic evaluation procedure for energy systems in buildings. CEN standard, June 2017. (see annex D).
- Ecodesign Impact Accounting – Status Report 2017. Prepared by VHK for the European Commission December 2017. (see annex A, pp.73-76).
On top of cumulating savings over savings lifetime or crediting durations, the savings calculations can also include the application of a discount factor. This discount factor can be used:
- For economic reasons: for example when assessing cost-effectiveness (see also section 7. ), so that costs and benefits occurring over time can be taken into account according to a given investor’s perspective (i.e. a given preference given to present).
- For technical reasons: for example, to take into account that energy savings can decrease over time (e.g. for behavioural actions).
When using an economic discount factor, it is defined according to the investor’s perspective chosen (and not according to the action type). Discount factors are usually taken from the literature or from reference values (e.g. when considering the public authorities’ perspective). For more discussions about discount factors, see for example the Horizon 2020 projects Briskee and Cheetah.
When using technical discount factors, they might need to be differentiated per action type, as the changes in energy savings over time can depend on the action type. However, there is limited evidence about decrease (or increase) of energy savings over time, especially when dealing with renovations of buildings (for more details about this issue, see for example Hoffman et al., 2015 or McCoy and Kotsch, 2018).
A general default discount factor (i.e. uniform for all action types), encompassing both economic and technical aspects, can also be decided to reflect the risks related to the investment in the energy saving action, as perceived by the final customers or investors.
The choice of the metric depends on the policy or evaluation objectives and reporting needs. For example, when the main objective is to keep the evaluation simple, the choice can be yearly savings, as no further data or assumption is then needed. However, this will favour short-term actions against longer term action. Incentive schemes for energy efficiency in buildings usually aim at promoting actions with long lifetimes, such as wall/roof/floor insulation, which can therefore support the choice of lifetime-cumulated savings. Lifetime-cumulated savings are also needed when assessing cost-effectiveness (see section 7).
The choice of the metric can also depend on reporting needs, for example in the context of EED article 7. In this framework, a particular type of cumulative savings is needed, counting the savings achieved within the obligation period (currently 2014-2020, and then 2021-2030).
As mentioned above about energy savings in final or primary energy terms, it is important to identify from the outset the needs to express the results in different metrics, so that data used in the energy savings calculations are documented enough to enable future conversions. When dealing with energy savings over time, this mostly means to keep records of the energy savings per main type of action, as lifetimes can be different from one type to the other.
6. ALTERNATIVE FOR CHOSEN METHOD
6.1 Alternatives for the chosen method
Engineering estimates are often chosen in case of large or specific energy saving projects, and can also provide results with limited time lag. Engineering calculations are for example commonly chosen to evaluate the savings from the incentive scheme when they are used anyway as part of the process of the application for the incentive (e.g. when an Energy Performance Certificate is required as part of the application file). In this case, there is no extra cost of using these data for the evaluation of the incentive scheme.
Deemed savings can be appropriate to evaluate savings from action types that can be described in a standardised way. They are then a cost-effective way to assess savings from large numbers of similar actions (e.g. case of an incentive scheme promoting single actions), without delay. They are less relevant for energy saving projects that would be very specific, thereby requiring case-by-case calculations. This is for example why engineering estimates will usually be preferred when assessing energy savings from a scheme promoting deep renovations. For more details about engineering calculations, see the Specific Guidance 9 .
Billing analysis is then commonly chosen when the evaluation objective is more specifically to assess or verify the energy savings actually achieved or to assess the cost-effectiveness or efficiency of the scheme. Billing analysis are particularly used in case of scheme promoting deep renovations. In such cases, the types of actions implemented are not necessarily monitored. This makes that methods based on what types of actions have been implemented (e.g. deemed savings or engineering methods) would be more difficult to use. Also, deep renovations imply that it can be important to take into account the interactions between the different actions (e.g. insulation of building envelope and replacement of heating system). Which can be taken into account in detailed engineering estimates, but more difficultly in deemed savings. The main drawback of billing analysis is the time lag between the installations of the actions and when the evaluation results can be available (usually from 18 to 30 months). For more details about billing analysis, see the Specific Guidance 8 .
Direct measurements are more rarely used for the evaluation of incentive scheme as considered in this guidance. They can for example be used as part of specific verifications or on-site inspections to complement another method.
The table below presents the pros and cons of these methods commonly used for evaluating energy savings from EEO scheme (see also Additional methods to increase reliability of the results in section 2).
|Type of method||Pros||Cons|
|Engineering calculations||· Can be used for almost all action types (including deep renovations)
· Can enable to automatize energy savings calculations (through standardised formula for simple cases)
· Can reflect the energy savings achieved for a given situation (specific calculations)
· Limited delay in getting the results (calculations can be done before the actions are installed)
|· Require to collect data for each case (so can be costly if data collected only for this purpose and for large numbers of actions / projects)
· Possible gaps between engineering estimates and measured savings (see references below)
· Additional method needed to evaluate ex-post the additionality of the savings
(see Specific Guidance 9)
|· Provide visibility to stakeholders
· No delay in getting results from the monitoring system
· Low running cost (once the set of deemed savings has been defined)
· Calculations directly related to the energy saving actions installed
|· Use limited to action types that can be described in a standardised way
· Do not reflect the energy savings achieved for a given situation, but an average result for a population of actions
· Can require significant preliminary efforts (if many different action types included in the scope of the incentive scheme)
· Quality depending on the data available to define deemed savings
· Possible gaps between deemed savings and actual savings (due to prebound and rebound effects, and to performance gaps)
· Additional method needed to evaluate ex-post the additionality of the savings
(see Specific Guidance 8)
|· Provide data about actual energy consumption / energy savings (capturing prebound and rebound effects, as well as performance gaps)
· Can be used to evaluate ex-post net savings (depending on the type of comparison group chosen, or whether adjustment factors can be assessed with an additional method)
|· Can only be used for ex-post evaluation
· Frequent difficulties to collect billing data (unless data collection carefully planned and prepared in advance, e.g. collecting participants’ consent when actions are installed)
· Difficulties to get representative samples (cf. sampling bias + data losses along the evaluation process)
· Delays in getting the result (at least one year to get the consumption after installing actions + time to process and analyse data)
· Difficulties to find relevant control or comparison groups (when assessing net or additional savings)
|Direct measurements||· Provide data about actual energy consumption (for the baseline and/or for the situation with energy saving actions) or about actual values for key parameters (e.g. indoor temperature)
· Can be used to assess performance gaps
|· Can be costly if measurements only done for this purpose and for large numbers of actions
· If sampling is used, attention should be paid to avoid sampling bias (if data are to be extrapolated)
· Additional method needed to evaluate ex-post the additionality of the savings
· Delay in installing the actions (if used to verify the baseline, then time needed to make the measurements, unless data are already available)
· Delay in getting the results (if used to verify the situation with energy saving actions, then time needed to make measurements after the actions are installed + time to analyse the data)
7. ADDITIONAL EVALUATION RESULTS
7.1 Calculating avoided CO2 emissions
Depending on the priority objectives of the incentive scheme, the evaluation objectives can include the assessment of the results in terms of CO2 savings (i.e. avoided CO2 emissions). In practice, engineering estimates are first calculated in terms of energy savings. Then avoided CO2 emissions can be evaluated from the energy savings by applying emission factors. Four key aspects are to be taken into account when choosing the emission factor(s):
- Emission factors vary according to the energy type, so the engineering estimates need to be defined per energy type.
- Emission factors for a given type of energy can vary over time (especially for electricity).
- Emission factors can take into account:
- Direct emission factors: that take into account the emissions generated when producing the energy used;
- Lifecycle emission factors: that take into account all the emissions generated from the extraction of the energy resources up to the dismantling of the energy plant.
Due to the differences that the choice of emission factor(s) can induce, it is important to document what emission factor(s) has(have) been used. Emission factors used for the incentive scheme can for example be based on official national emission factors used for the national inventory of emissions of greenhouse gases.
The conversion of electricity savings into CO2 savings is however a special case, depending on the national mix for electricity production. Several choices are indeed possible, for example:
- Average emission factor, calculated from the total annual emissions from electricity production (possibly taking into account national imports and exports) divided by the annual amount of electricity consumed: this is a simple approach, but that might not reflect the fact that end-uses can have different times of use and thus correspond to different load profiles (while the emission factor for electricity can vary significantly between base load and peak load).
- Emission factors per type of end-use: this requires more sophisticated calculations (e.g. by decomposing the national load curves per type of end-use) that will be meant to use emission factors reflecting the differences in time of use (e.g. daily, seasonally).
The choice between the two options above will depend on the national electricity mix (cf. emission factor varying significantly with time of production or not) and the type of end-uses covered by the incentive scheme (and especially if savings related to electric heating can represent a significant share of the savings).
If the engineering estimates cumulate savings over time, it can also be needed to define a scenario about the evolution of the national electricity mix over the period of calculation (e.g. taking into account the objectives of shares of electricity produced from renewable energy sources).
The avoided emission of other greenhouse gasses due to energy savings are not taken into account here, as these emissions (and more specifically their reductions) are generally negligible compared to CO2 for actions in the residential sector (except in the case of schemes dealing with certain AC systems).
When needed, IPCC (Intergovernmental Panel on Climate Change) provides a detailed database of peer-reviewed emission factors.
7.2 Calculating cost-effectiveness
Cost-effectiveness is the ratio between costs to achieve energy savings and the amount of savings and possibly other benefits.
A distinction can be made according to the point of view adopted to assess cost-effectiveness:
- Cost-effectiveness for the end-user or participant (e.g. payback time)
- Cost-effectiveness from the point of view of the public authority funding the incentive (e.g. comparing different types of policy measures according to the public cost of energy savings)
- Cost-effectiveness for society at large (e.g. social net present value)
For more details about the different perspectives, see for example (Breitschopf et al., 2018).
The table below summarizes the main costs and benefits taken into account according to the perspective adopted.
|Point of view||Costs||Benefits|
|Participants||· Part of the investments paid by the participants||· Gross energy savings
(note: the financial aids received from the incentive scheme is taken into account in the fact that only the part of the investments paid by the participants is included in the calculation)
|Public authorities||· Financial incentives paid to the participants
· Administration costs of the scheme
· Losses in tax revenues (due to additional energy savings)
|· Increases in tax revenues (due to additional investments made in energy saving actions vs. baseline scenario)
· Net energy savings (e.g. if the indicator calculated is about the public cost of energy savings)
|Society||· Part of the investments paid by the participants (for additional actions only)
· Financial incentives paid to the participants
· Administration costs of the scheme
|· Net energy savings|
NOTE: the table above does not deal with non-energy impacts. Depending on the context and objectives of the incentive scheme, non-energy benefits can be larger than the benefits from energy savings. When assessing the cost-effectiveness of an incentive scheme from the society’s point of view, it is therefore recommended to consider if it is relevant to include non-energy impacts in the scope of analysis (see Calculating other co-benefits below for more details).
Cost-effectiveness indicators are usually calculated taken into account final energy savings. Likewise, these indicators are most often calculated over the lifetime of the energy saving actions (to take into account the full benefits from the energy saving actions), or over a given horizon of time (when considering the time perspective of investors, possibly including the State as a general public investor). The relevant metric of energy savings is therefore cumulative or lifetime-cumulated energy savings. This means that these calculations require assumptions about the lifetime of the actions and the evolution of energy savings over time.
Depending on the indicator(s) used to assess cost-effectiveness, it can be needed to use a discount factor (e.g. when the indicator is Net Present Values). The discount factor should be applied consistently to all terms of the calculations (i.e. all costs and benefits, including the energy savings). Different discount factors can be used according to the perspective adopted (e.g. households, State). It is important to document the use of discount factors, and if possible to make a sensitivity analysis (testing several values or ranges of discount factors). As this can affect significantly the results. For more discussions about discount factors, see for example the Horizon 2020 projects Briskee and Cheetah.
Likewise, the calculations of cost-effectiveness indicators will usually require to consider scenarios of energy prices over given periods. The assumptions about trends in energy prices should be documented. Whenever possible, it is recommended to make a sensitivity analysis (testing several scenarios of energy prices). See for example the EPATEE case study about the Better Energy Homes scheme in Ireland (Broc, 2017).
For more details about assessing cost-effectiveness or cost-benefit analysis, see for example (Clinch and Healy 2001; Collins and Curtis, 2017). And more generally about assessing the effectiveness of incentive schemes, see for example (Stern et al ., 1986; Studer and Rieder, 2019).
7.3 Calculating other Co-benefits
Possible co-benefits from saving energy in buildings can concern for example:
- Reduction of energy poverty
- Better indoor climate
- Extra employment in the building industry
- Impact on economic activity
- Reduced dependency on (insecure) energy import
It should be noted that the impacts from incentive schemes on each of these aspects are usually positive, but can also be negative (e.g. on State budget or distributional effects). Therefore, it is in general more appropriate to speak about non-energy impacts.
For a general background about non-energy impacts, see here.
Analysis about how non-energy impacts can have an influence on the political support or rationale for an incentive scheme can be found for example in (Kerr et al., 2017; Rosenow, 2013). An example of study looking at multiple impacts of renovations can be found in (Coyne et al., 2018). An example of study about distributional effects can be found in (Daussin-Benichou and Mauroux, 2014).
8. CONCRETE EXAMPLES
EPATEE case studies:
- Broc, J.S. (2017). Better Energy Homes scheme (Ireland). Case study prepared by IEECP for the EPATEE project, funded by the European Union’s Horizon 2020 programme.
- Broc, J.S. (2018a). Energy premium for refurbishment in Belgium (Wallonia). Case study prepared by IEECP for the EPATEE project, funded by the European Union’s Horizon 2020 programme.
- Broc, J.S. (2018b). Warm Front scheme (England). Case study prepared by IEECP for the EPATEE project, funded by the European Union’s Horizon 2020 programme.
- Maric, L., Matosovic, M., & Kojakovic, A. (2017). Energy Renovation Programme for Public Sector Buildings (2014 – 2015). Case study prepared by EIHP for the EPATEE project, funded by the European Union’s Horizon 2020 programme.
- Meulen, J. & Menkveld, M. (2018). Subsidy scheme for energy savings in social housing of Amsterdam. Case study prepared by ECN-TNO for the EPATEE project, funded by the European Union’s Horizon 2020 programme.
- Niessink, R. (2018). The Case-Study Special Subsidy Scheme for Energy Labelling Improves Social Housing Sector Amsterdam Seen in the Perspective of the European H2020 Project EPATEE. Proceedings of IEPPEC 2018.
- Skema, R., & Dzenajaviciene, F. (2017). Lithuanian Renovation Programme with EU funding. Case study prepared by LEI for the EPATEE project, funded by the European Union’s Horizon 2020 programme.