01 – Evaluating Normative legislative policies on energy efficiency in Residential buildings using Measurement

The primary audience for this guide concerns energy efficiency programme designers, implementers or supervisors and evaluators looking for guidance on the evaluation process of energy savings within the scope of this tool.

Although the application of the tool will generally concern the (sub)national level, account will be taken of issues at the EU level whenever relevant (e.g. the specific format of saving figures for the EED).

This guide is not about the preceding step in the evaluation process, the choice of the method. About this previous step in the evaluation process, see the guidance provided here. However, after presenting the capabilities and limitations of the tool at hand, the user will be offered alternatives for the method within this tool (see section 6).

The objective of this guide is to provide:

• Information on the scope of the tool that enables the user to decide whether this tool is suitable for his/her needs, and whether complementary or additional method(s) could be needed or useful (section 2);
• Guidance about specifying the evaluation objectives and requirements (section 3);
• Guidance about key methodological choices to calculate energy savings (section 4);
• Guidance about the inputs (data requirements) and outputs of the method (energy savings metrics) (section 5);
• Possible alternative methods (with pros and cons) (section 6)
• Background about evaluation results other than energy savings (section 7);
• Relevant examples, case studies and/or good practices (section 8);
• Relevant references for further reading (section 9).

The tool is intended for assessing realised (ex-post) energy savings. However, account is taken of earlier (ex-ante) evaluations of expected savings, if available (see section 4).

The focus of the tool is on impact evaluation, i.e. determining the energy savings, but not on how this has been reached through a step by step process with intermediate results (process evaluation).

Readers looking for the basic and general principles of energy efficiency evaluation may find the following link useful.

2.1 About normative legislative policy measures (building codes)

Normative legislative measures in residential buildings in practice come down to efficiency standards or building codes. For standards (sometimes indicated as Minimum Energy Performance Standards), a distinction can be made between specific standards on for example insulation (e.g. the heat resistance of double glazing) or boilers (minimum conversion efficiency standards) and system-wide standards (e.g. maximum gas consumption per m² for new dwellings). This distinction is for example made in the European Directive on the energy performance of buildings (EPBD, Directive 2010/31/EU). The policy measure type described in this guide is restricted to system-wide standards.

The EPBD requires all Member States to enforce minimum energy performance requirements for both new and existing buildings, and to review them at least every five years. The directive does not define levels or values for these requirements, but it specifies that they shall be set “with a view to achieving cost-optimal levels” (EPBD article 4). More specifically for new buildings, the EPBD (article 9) stipulates that Member States shall adopt a definition for “nearly Zero Energy Buildings” (nZEB) and that the minimum requirements shall be updated to ensure that every new buildings are nZEB from 31 December 2020 on. The provisions of the EPBD set the background common to all EU Member States. Then each country can adapt them to its national specificities, and can choose to adopt more stringent requirements.

These standards are sometimes associated with energy performance certificates (EPC, see EPDB article 11) and coefficients. Allocating an energy performance certificate or coefficient is no guarantee that the actual energy consumption will be in line with the energy consumption as estimated in the EPC. The same issue arises more generally when the expected energy consumption of the new buildings (or existing buildings after refurbishment) is estimated from building energy modelling.

To get to know the actual energy consumption, an evaluation based on measurements or billing analysis is necessary. One of the aspects that have a large influence on energy consumption in this case is the behaviour of the occupants and the occupancy rate of dwellings (see section 4 for more details).

More information and examples on the different subtypes residing under the main type (legislative / normative measures) can be found here and here.

More detailed information on the evaluation of legislative / normative measures can be found here.

2.2 Evaluation for a combination of policy measure types

When a normative legislative measure is combined with other policy measures types, it is assumed that the overall savings are mainly resulting from the normative policy measure at hand. However, the evaluation concerns the combined savings effect of both policy measures.

The tool is not capable of attributing part of the (overall) calculated savings to different policy measures (see also Double counting in the section on Gross to Net savings).

In general, it can be useful to distinguish two categories of complementary policy measures:

1. Policy measures aiming at ensuring the effectiveness of the minimum energy performance requirements (e.g. training programmes, compliance schemes).
2. Policy measures aiming at promoting energy performance beyond the minimum requirements (e.g. voluntary labels, grant schemes).

For the first category of policy measures, one possible way to separate the effects can be to make comparisons between periods or regions (without vs. with the complementary measures). See for example (Rasin et al. 2013; Storm 2015).

For the second category of policy measures, one possible way to separate the effects and avoid double counting is to set the minimum energy performance requirements as the baseline for calculating energy savings resulting from the other policy measures assessed.

2.3 About (new) residential buildings

Information on (sub)sectors defined in the Toolbox can be found here, chapter 2, p.17
Other, equivalent descriptions of the sector are “dwellings” and “households”.

2.4 About the measurement method

Information about the various evaluation methods can be found here, table 1 and 2. This source also covers the combination of the method at hand with other methods, which will be dealt with below.

This source also covers the combination of the evaluation method at hand with other methods, which will be dealt with below. Also see the IPMVP (International Performance Measurement and Verification Protocol) volume dedicated to the case of new construction in the further reading section (EVO 2006).

Measurement in case of energy performance standards for dwellings comes down to measuring the energy consumption of comparable dwellings as for size, but with different performance standards, while occupancy and behaviour are taken into account as well to get a reliable estimate of the effect of the performance standard. Interviews are needed to get data on occupancy rates and behaviour, which have large influence on energy consumption.

2.5 Complementary methods to determine total savings

Complementary methods are methods that are required, in addition to the primary selected method, to calculate energy savings.

The method at hand is meant to calculate unitary savings for one dwelling, possibly distinguishing unitary savings according to a typology taking into account building types, regions or other relevant criteria. These unitary savings (often expressed in kWh/m²/year) should be multiplied by the number of saving actions or participants in order to get the calculated total savings. In the case of energy performance standards for new dwellings, the unit of action is often the area of dwellings built (possibly distinguished by type of building, region, etc.).

The number of actions or participants can be obtained in various ways.

See the link here, table 2 and 3 for more information.

The number of actions is sometimes directly available from the monitoring of policy measures, such as in case of a subsidy scheme. In case of energy performance standards for dwellings, the number of actions (or more specifically the total area of dwellings built) can be acquired from the registered number of licenses for new dwellings or related statistics at the national office in charge of housing statistics.

2.6 Additional methods to increase reliability of the results

An additional method can be applied on top of measurement to improve the reliability of the evaluation results and/or the cost-effectiveness of the evaluation approach.

Billing analysis concerns overall energy consumption of a dwelling, which can be influenced by many other factors than the policy measure/saving actions at hand (e.g. weather conditions, occupancy, user behaviour). It is not always possible to monitor data to take into account changes in these other factors. To regularly check the results, measurement as additional method can be applied to focus on energy consumption of electrical or heating equipment that is directly targeted by the saving action to make sure the division into different energy application types (space heating, hot water, cooking) has been done correctly. The combination can increase the reliability of the savings figures in a cost-effective way if a limited number of measurements are used to calibrate the results from the complete collection of buildings under investigation.

For possible combinations with an additional method see chapter 6 here.

3.1 Meeting evaluation goals and ambition

The table shows whether this tool can be used to report on general evaluation goals or criteria. See also this document.

General types of evaluation goals or criteria	Level of ambition	Remarks
Calculation of realized energy savings from saving actions	Medium to High	Reliability will depend, among other issues, on how normalization factors (particularly weather conditions and occupancy) can be taken into account in the billing analysis, and on how results are extrapolated to the whole new dwellings counted for the policy (if billing analysis is performed on a sample).
Calculation of energy savings attributed to the policy measure(s)	Medium to High	The choice of the baseline is important. Reliability will depend, among other issues; on how factors such as compliance, performance gap and rebound effect can be taken into account in the billing analysis.
Cost-effectiveness of saving action (for end-users)	Medium to High	Feasibility of assessing cost-effectiveness will mostly depend on availability of data about construction costs for a baseline scenario and the actual situation.
Cost-effectiveness of policy (government spending)	Medium to High	There is no government spending at the saving action level for legislative policies, but the government spending on the policy administration can be compared to the combined savings result Feasibility of assessing cost-effectiveness will mostly depend on availability of data about administration costs.
CO2-emission reduction from saving actions	Medium to High	See comments above about Calculation of realized energy savings from saving actions
CO2-emission reduction attributed to the policy measure(s)	Medium to High	See comments above about Calculation of energy savings attributed to the policy measure(s)

For more information on verification of actual energy savings and attribution/baseline/corrections, see section 4, and for cost-effectiveness and emission reduction, see section 7.

When evaluating a building code or similar energy efficiency standards, the evaluation objectives can include investigating if the corresponding requirements are implemented in practice (cf. compliance issue), or if the requirements are defined in a relevant way and how they could be improved.

These issues are essential for the (cost-) effectiveness of the building code or similar standards. They require complementary analysis, either by including more variables in the billing analysis or by performing specific surveys. More information can be found in the part 4 section about normalization factors and adjustment factors.

3.2 Reporting expectations

The measurement method will make it possible to report about the savings effects of applying efficiency standards in relation to the number of inhabitants and the occupancy rate.

The unit used to express the savings will usually depend on the unit used to set the requirements in the building code or similar standards. Common types of units are:

• Primary energy savings: often used when the requirements also deal with a minimum share of energy supply from renewable energy sources or more generally with on-site energy generation, or when electric heating represents a significant share of the space heating market.
• Final energy savings: often used when the focus of the requirements is on the energy performance of the building envelope, or when electric heating represents a small share of the space heating market.
• Lifecycle energy savings: emerging unit used to take into account that when the level of energy performance of a building is high, then the energy embodied in the construction of the dwelling (e.g. due to the production of the construction materials) can be as important (or even more important) as the heating consumption over the course of the building’s lifetime.
• CO2 savings: often used when the regulation includes a focus on climate change impacts.
• Lifecycle CO2 savings: same approach as for lifecycle energy savings.

Billing analysis will first provide results in terms of final energy savings, as energy bills meter the final energy consumption. Complementary data (e.g. conversion factors, data about materials) will be needed for calculating other units.

3.3 Time frame for evaluation

The length of the period under evaluation is dependent on the active period of the policy measure and the time needed to present (reliable enough) results or impacts that fit into the decision making process. In some cases, the periodicity of evaluation can be set by law. More specifically to building codes or similar standards, the evaluation will usually be planned so that results can be taken into account when preparing the revision or update of the building code. For example, to investigate if the way the requirements are specified should be improved or if additional provisions would be needed to ensure a better compliance.

The planning of evaluation activities concerns regular monitoring of energy consumption and factors that define consumption, intermediate check of (ex-ante) estimated (unitary) savings through measuring or surveys, intermediate evaluations to improve the policy implementation and the final evaluation and reporting.

To ensure the feasibility and optimise the costs of data collection for billing analysis, it is strongly recommended to plan the evaluation in advance. The early phase of the evaluation process should in particular be used to define what data is needed and how it will be collected. Depending on the country, collecting billing and other personal data can raise legal questions that need to be anticipated.

Billing analysis can only be done at least after one complete year, as all seasons have to be included in the evaluation time range of energy consumption. As interviews are necessary to include information about occupancy rates and behaviour of the inhabitants, enough time for this has to be taken into account. In practice, the time lag when using billing analysis can be 18 months or more, between the time energy consumption are metered and the time the results of the billing analysis is available (see also: chapter “Evaluation planning and preparation” in the link here. )

3.4 Expertise needed for chosen method

Expertise in billing analysis (econometric method) and in methods for the correction and normalisation of weather conditions.

Expertise in sampling methods. As in most cases, billing analysis will be done on a sample of new dwellings and then extrapolated to the whole new dwellings counted for the period under evaluation.

Knowledge about behaviour of inhabitants, designing interviews and interviewing skills are needed for a proper evaluation.

Knowledge about how the requirements of the building code or similar standards have been specified and are enforced. This is important for analysing the effectiveness of the building code, and to analyse issues related to compliance, performance gap and rebound effect.

General principles of calculating realized savings using different methods can be found here and here

This section deals with key methodological choices to be considered when calculating energy savings: consistency between ex-ante and ex-post evaluation, baseline, normalization and adjustment factors. These choices are important to document when reporting energy savings, to ensure the transparency of the results.

The measurement method for calculating the energy savings in dwellings in conformity with energy norms can be complemented with interviews, but also with additional measurements involving e.g. monitoring of the indoor temperature or with the difference-in-differences method. Two sufficiently large groups are compared, each with their own energy efficiency standard. Billing analysis can also be used to compare with engineering estimates (for the same group of dwellings). When combined with occupancy rates and inhabitants behaviour, the effect of the improved energy efficiency standard can be determined.

4.1 Calculation baselines

Energy savings are defined in general as the difference between the actual situation and a reference situation without the saving actions (and without the policy measures that influence these saving actions). In case of saving actions the reference situation can be defined using various calculation baselines: Before/after, With/without, Trend, Target/control group and Minimum efficiency standards; see further here.

Energy savings in this case are calculated by applying the with/without method: analysis of changes in energy consumption with/without the saving action, where the “without” situation acts as the baseline for the “with” situation. This option is for example suitable for new efficient systems where no “before” situation is available, as is the case for new dwellings. Results can be corrected for adjustment factors, providing net savings. The with/without baseline can also be applied more generally for evaluation of policy measures, especially to evaluate energy savings additional to a predefined scenario (e.g., business-as-usual scenario or a scenario including other policy measures).

In this case, this predefined scenario will be used to specify the “without” situation. The applicable baseline is the energy consumption of dwellings with a less strict energy standard. The baseline can for example be the requirements set for the period previous to the enforcement of the building code or standards under evaluation, or the requirements in force at a given date.

It should be noted that dwellings from previous construction period can have been subject to renovation, which can change the baseline energy consumption compared to the energy requirements in force at the time the dwellings were built. As it is often difficult to correct for the influence of renovation works (unless this was already analysed for other purposes), an alternative is to remove dwellings that have been subject of renovation works from the baseline group.

Another alternative is to use a group of dwellings built within a few months or years before the building code or standards under evaluation came into force as baseline group. In that case, the probability for these dwellings to have been subject of renovation is low.

See also here.

When dealing with EED article 7, it should be noted that the amendment to the EED (EU Directive 2018/2002) brought the following clarifications (for point (b) of Annex V(2)):

“Savings resulting from the implementation of national minimum requirements established for new buildings prior to the transposition of Directive 2010/31/EU can be claimed as energy savings for the purpose of point (a) of Article 7(1), provided that the materiality criterion referred to in point 3(h) of this Annex is ensured and those savings have been notified by Member States in their National Energy Efficiency Action Plans in accordance with Article 24(2).”

In other words, energy savings from building codes or similar standards for new dwellings in place before the transposition of the EPBD can be accounted for the obligation period 2014-2020 of the EED article 7. Then these savings are no longer eligible for the obligation period 2021-2030.

4.2 Normalization factors

The calculation based on comparing a group of dwellings with the energy standard to be evaluated to a group of dwellings with a lower energy standard provides a change in energy consumption that should be corrected for influences on energy consumption other than the saving actions. When dealing with buildings, these so-called normalization factors can be weather (with effect on consumption), the rebound effect, a performance gap and changes in occupation rate or patterns. All normalization factors affect total savings through the unitary savings.

For the specific guidance at hand the normalization factors can concern weather, rebound effect and occupation rate. See table 1 in this link.
Normalization is normally not needed as the factors will be the same for the target group and the comparison group. However, this should be checked and possibly the factors should be applied nevertheless. For example if the time series available for both groups are limited (which could thus mean that the energy savings would be calculated for a particular period, not necessarily representative of typical weather conditions).

Energy consumption can be corrected for the occupation rate of buildings (with effect on heating/lighting), when this rate is different for the baseline situation and the situation after the saving actions.

The analysis of the different factors that can influence energy consumption (in addition to the energy efficiency improvements) usually requires to collect additional data, for example from surveys, measurements or monitoring system (e.g. about indoor temperature). This type of analysis helps to identify the possible side-effects of building codes or standards, as well as aspects of the requirements that could be improved.

4.3 Adjustment factors

Adjustment factors define which part of the calculated energy savings can be attributed to a policy measure or meets the definition of savings specified in the evaluation objectives or reporting requirements (see next section on “Calculating gross and net savings).

In general, adjustment factors can concern the Free rider effect, the spill-over/multiplier effect, additionality and Non-compliance; see table 1 in link here. Additionality and non-compliance are connected to unitary savings, while free riders and spill-over work through the number of actions. In case of another policy focusing on the same saving actions as evaluated with the Specific Guidance at hand, the adjustment factor double counting might be relevant. If the other policy is not covered in the Specific Guidance at hand, double counting can only be accounted for at a higher level than individual Specific Guidances (see Distinction of energy efficiency improvement measures by type of appropriate evaluation method, see here and here.

For the evaluation method at hand, the adjustment factors free rider effect and additionality are not relevant.

Further information on calculation methods can be found in chapter 8 of this document.

For practical reasons, the primary focus in this guide is on the adjustment factor Non-compliance. However, for specific evaluation methods, other factors can be taken into account. In order to correct for non-compliance due to inappropriate implementation of saving actions, data should be available of sample-wise checks on the implementation.

4.4 Calculating Gross and Net energy savings

Gross savings concern the calculated savings from saving actions using a chosen baseline and normalization factors. Net savings concern the savings attributed to policy measures or to a stakeholder (e.g. an energy company with an obligation to realise savings at their customers).

When calculating the savings a distinction must be made for most specific guides to the unitary savings and number of actions; see here.

The gross unitary savings can be calculated using the baseline without the savings action (using the with/without methodology) and correcting for relevant normalization factors: weather conditions, performance gap, non-compliance, rebound effects. The correction for non-compliance concerns inappropriate implementation of saving actions. To be able to apply this correction, data should be available of sample-wise checks on the implementation. Gross savings can be corrected for the rebound effect. However, sometimes the calculated change in energy consumption already incorporates the rebound effect. For billing analysis for example, the “after” energy consumption includes the rebound effect, and thus gross savings should not be corrected for the rebound effect in this case, as it is already included in the result from the billing analysis.

The gross number of actions is determined by applying the complementary method of using the registered number of licenses for new dwellings (see chapter 2). Total gross savings are equal to the gross unitary savings multiplied by the gross number of actions, possibly taking into account a typology of dwellings as unitary savings can change according to dwelling type, region (e.g. due to differences in climate conditions), etc. In most cases when using billing analysis, the unitary savings are evaluated on a sample of dwellings. Therefore, the results obtained for the sample need to be extrapolated to the whole group of new dwellings for the policy measure. For more details about sampling methods and related issues, see for example (Baumgartner 2017).

Net unitary savings can be determined from gross unitary savings applying the relevant adjustment, in this case non-compliance. The net savings are determined from the gross savings by applying relevant adjustment factors. Also see here and here.

The savings should be corrected for the Double counting effect, i.e. the overlap between the savings due to the efficiency norm and savings due to other policy measures. The overlap in the calculated savings of both policy measures cannot be processed at the level of a Specific Guidance but must be corrected at the level of savings due to overall policy portfolios. For addressing double counting see the earlier links regarding adjustment factors.

5.1 Main data requirements, data sources and collection techniques

Data requirements specified in the table below correspond to the calculation of energy savings when using the baseline option with/without or comparison group.

Calculation subject	Data requirements	Possible data sources and collection techniques
Energy consumption	Gas, electricity and heat consumption, number of inhabitants	Energy bills
Normalization factors affecting energy consumption	performance gap, rebound effect, occupancy rate	Performance gap: checking technical implementation, rebound effect: interview questions about changed behaviour, occupancy rate: interview
Adjustment factors	Non-compliance	Checking for correct energy standard implementation
Number of actions	Number of dwellings with the energy standard studied	Building permits, national housing statistics
Data about participants to ensure representativeness or good fit	Large enough sample size of participants interviewed	Interviews / surveys, national statistics (e.g. national housing survey)

Data issues when evaluating net energy savings

The main good practice to ensure the feasibility and reliability of the evaluation of net energy savings is to think about the evaluation method to be used when designing (or revising) the energy standard.

Experience indeed shows that unless the data collection has been planned in advance, it will be very costly, time-consuming or even impossible to collect the data required to apply most of the methods that can be used to evaluate net energy savings. This means that in practice, using surveys will remain the only option possible (or considered feasible).

The main challenges when using surveys are:

• to achieve a high answer rate, in order to limit sampling bias;
• to use question phrasing that can limit the risk of bias in the answers.

For more details about the evaluation of net energy savings, see the topical case study

For a proper way to determine savings, knowledge about energy consumption is not enough, but knowledge about occupancy rates is needed as well. This means that surveys have to be carried out for reliable results.

For possible other methods with different data requirements, see the section on alternatives for the chosen method.

5.2 Energy savings in final terms or in primary terms

Energy savings can be expressed in final terms or in primary terms See definitions about primary and final energy here.

The measurement or billing analysis method can calculate savings in final terms. It can also calculate savings in primary terms, provided that savings at end-users are calculated for each energy carrier separately, and that primary factors are available to convert the savings in final terms to savings in primary terms.

If the building code or similar standards are set in primary energy terms, conventional primary factors are usually defined together with the regulation.

5.3 Energy savings over time

Implemented saving actions in a certain year lead to savings over a number of consecutive years. A more efficient boiler for example can save gas over its lifetime of about 15 years, insulation over up to 30 years. Energy savings can be calculated in different metrics in terms of time reference, for example: year-to-year, annual, cumulated annual, cumulative over a series of years and especially over the assumed lifetime of the buildings. See the definitions here.

The calculated yearly savings concern the savings of all new saving actions in that year. Only data for the savings in the chosen year are needed in this approach: data for the new dwellings built in a given year.

Adding up the yearly savings over a period, provided that earlier saving actions are still delivering savings, leads to cumulative savings. For cumulative savings, data is needed for the whole period: data for the new dwellings built over the period taken into account.

Another cumulative approach, to be applied for the energy efficiency directive, is to multiply the (new) savings in a year with the number of years up to a target year (2020 for the first obligation period 2014-2020; 2030 for the second obligation period 2021-2030), and add this result to that for all other years up to the target year. This cumulative approach stimulates early saving actions, as these add more times to the savings than later actions.

As buildings are often assumed to have long lifetime (e.g. over 40 years) and correspond to important investments, energy savings can also be calculated over the lifetime of the building. This is for example the case when assessing the cost-optimal level of energy performance as required by the EPBD. The value(s) used for the lifetime is usually an assumption defined in the regulation. This assumption is often based on the observed renewal rate of the building stock or similar sources.

Finally, savings from a saving action can be discounted and summed up over the lifetime of the action See link here.

When a discounting approach is used, it is important to clarify if this discounting is related to technical or economic reasons (or both). When evaluating the energy savings from a building code or similar standards, a technical reason for discounting could be to consider for example ageing of insulation materials, improper maintenance of the buildings, etc.

Similarly, a standard discount factor can also be decided to reflect the risks related to the investment in the energy saving action (particularly due to the long lifetime vs. high upfront cost), as perceived by the final customers or investors.

The measurement and billing analysis methods can provide yearly savings of new saving actions in that year. It can also provide cumulative savings provided that data is available over a longer period, or assuming that the energy savings will not change significantly over time. An alternative assumption can be to apply a default discounting factor, as discussed above.

6.1 Alternatives for the chosen method

Often, other savings calculation methods can be applied as well, although they will all have pros and cons regarding various aspects dealt with in preceding sections.
The table below presents the pros and cons of the ‘measurement’ method for evaluating energy efficiency standards in residential buildings, and for commonly used alternative methods for the same combination of policy measure and sector.

Type of method	Pros	Cons
Method at hand (measurement or billing analysis in combination with surveys)	Precise, provided that the billing or other measured data is reliable and that the billing analysis is done properly. Reflects the actual energy savings, i.e. as observed by the occupants or owners of the dwellings.	Takes time (cf. time lag to get the billing data, at least one year after the dwellings are completed). Depending on the cases, data and analysis can be needed to separate the consumption due to different end-uses (e.g. space heating, domestic hot water, cooking). Complementary data needed to correct for factors other than the energy efficiency requirements (e.g. weather conditions, occupancy), which can be expensive depending on the sample size and the way the data is collected.
Alternative method (measurement or billing analysis in combination with estimated occupancy rates and inhabitant behaviour)	Easier to implement (compared to using surveys to collect the complementary data)	Takes time (cf. time lag to get the billing data, at least one year after the dwellings are completed). Depending on the cases, data and analysis can be needed to separate the consumption due to different end-uses (e.g. space heating, domestic hot water, cooking). Less precise (compared to using surveys to collect the complementary data, as the assumptions about occupancy and behaviours might not reflect the actual occupancy and behaviours).

7.1 Calculating avoided CO2 emissions

Avoided CO2 emissions can be evaluated from the energy savings by applying emission factors. Four key aspects are to be taken into account when choosing the emission factor(s):

1. Emission factors vary according to the energy type, so the data about energy savings needs to be available per energy type.
2. Emission factors for a given type of energy can vary over time (especially for electricity).
3. Emission factors can take into account:
   1. Direct emission factors: that take into account the emissions generated when producing the energy used;
   2. Lifecycle emission factors: that take into account all the emissions generated from the extraction of the energy resources up to the dismantling of the energy plant. This indicator can also be important for the emissions related to the construction of new dwellings, as for the new generation of buildings their embodied energy (e.g. due to the production of the construction materials) can be as important, or even larger than the consumption due to the use of the buildings.

Due to the differences that the choice of emission factor(s) can induce, it is important to document what emission factor(s) has (have) been used. When the building code or similar standards include a metric related to emissions of CO2 or greenhouse gases, the regulation will usually include conventional emission factors.

The reduction in CO2-emissions can only be calculated when savings are calculated per relevant energy carrier and a specific emission factor is available for each energy carrier.

The avoided emissions of other greenhouse gases due to energy savings are not taken into account here, as these emissions (and more specifically their reductions) are generally negligible compared to CO2.

IPCC (Intergovernmental Panel on Climate Change) provides a detailed database of peer-reviewed emission factors.

7.2 Calculating cost-effectiveness

Cost-effectiveness is the ratio between costs to achieve energy savings and the amount of savings and possibly other benefits.

A distinction can be made according to the point of view adopted to assess cost-effectiveness:

• Cost-effectiveness for the end-user (here the building owner or occupant)
• Cost-effectiveness for society at large
• Cost-effectiveness for the party that takes responsibility for saving targets (here the government or other public authority in charge of the corresponding regulation)

See here

The calculation of cost-effectiveness for end-users requires, apart from the savings, data on investments made, possibly interest rates, lifetimes of the saving actions, energy prices (including taxes) per type of end-user and discount factors per type of end-user. The calculation is based on the additional costs of a dwelling that conforms to the efficiency standard for the home owner (compared to the costs of the dwelling in the baseline scenario) and the related energy savings results. It can be challenging to define the costs of the dwelling in the baseline scenario. An alternative is to assume a share of additional cost to the actual total cost of the new dwelling. This share can be based on a market survey or expert judgement.

For cost-effectiveness from a societal viewpoint no account is taken of taxes, energy prices concern world market price, and a lower value of the discount factors is valid. The calculation of the societal cost-effectiveness is based on the costs of setting up the policy and the combined energy saved by all dwellings involved.

7.3 Calculating other co-benefits

Possible co-benefits from saving energy concern:

• Extra employment
• Reduction of energy poverty
• Other emission reductions (NOx, SO2, fine particles, etc.)
• Better indoor air quality
• Reduced dependency on (insecure) energy import

The following co-benefits are regarded as most relevant and/or feasible to calculate in conjunction to this tool (see table)

The calculation of the co-benefits is explained in the references.

Type of co-benefits	Why it can be relevant (and for whom)	References where more details can be found
Increased comfort	Can increase support for savings measures	Evaluation of the tightening of the energy performance coefficient of new dwellings https://publicaties.ecn.nl/ECN-E–10-043

7.4 Other aspects of importance

For the following aspect an evaluation can be executed (see table).

Type of aspects	Why it is evaluated	References where more details can be found
Indoor air quality	It can worsen if the  ventilation with heat recovery is not working well	Evaluation of the tightening of the energy performance coefficient of new dwellings https://publicaties.ecn.nl/ECN-E–10-043

A concrete example can be found in a report describing research on the difference between energy consumption in dwellings with two different values for an energy performance coefficient. The costs involved to achieve the higher efficiency standard are included as well. See the further reading section for a reference to the report (Menkveld, M. & Leidelmeijer, K. (2010) – “Evaluatie EPC-aanscherping woningen” (Evaluation of energy performance standards in dwellings; full text in Dutch only, English abstract)).

Examples of evaluations based on a comparison of dwellings built in different periods, according to different energy requirements:

Jacobsen, G. D., & Kotchen, M. J. (2013). Are building codes effective at saving energy? Evidence from residential billing data in Florida. Review of Economics and Statistics, 95(1), 34-49. https://www.nber.org/papers/w16194.pdf

Kotchen, M. J. (2017). Longer-run evidence on whether building energy codes reduce residential energy consumption. Journal of the Association of Environmental and Resource Economists, 4(1), 135-153. https://www.journals.uchicago.edu/doi/pdfplus/10.1086/689703

Lee, A. & Perussi, M. (2017). Do Energy Codes Really Save Energy? Proceedings of IEPEC 2017. https://www.iepec.org/?attachment_id=13771

Novan, K., Smith, A., & Zhou, T. (2017). Residential building codes do save energy: Evidence from hourly smart-meter data. E2e Working Paper 031. http://e2e.haas.berkeley.edu/pdf/workingpapers/WP031.pdf

Examples of evaluations based on comparison between states or regions with different energy requirements (studies about US states):

Aroonruengsawat, A., Auffhammer, M., & Sanstad, A. H. (2012). The impact of state level building codes on residential electricity consumption. Energy Journal-Cleveland, 33(1), 31. https://www.researchgate.net/profile/Maximilian_Auffhammer/publication/227363608_The_Impact_of_State_Level_Building_Codes_on_Residential_Electricity_Consumption/links/0c9605151e81a4513a000000/The-Impact-of-State-Level-Building-Codes-on-Residential-Electricity-Consumption.pdf

Deason, J. & Hobbs, A. (2012). Codes to Cleaner Buildings: Effectiveness of U.S. Building Energy Codes. Proceedings of IEPEC 2012. https://www.iepec.org/wp-content/uploads/2018/02/064-9.pdf

Example of an evaluation using an alternative method combining data from energy audits with an engineering model (and a survey with professionals of the construction sector):

Tiedemann, K.H. & Sulyma, I.M. (2016). Measuring the Impact of a Residential Energy Code. Proceedings of IEPPEC 2016. https://www.iepec.org/wp-content/uploads/2018/04/Paper-Tiedemann.pdf

Example of possible pitfalls when using econometric methods to evaluate building codes:

Nadel, S., 2015. California building codes: to analyze the forest you need to understand the trees. Blog post on ACEEE website, 25 February 2015. https://aceee.org/blog/2015/02/california-building-codes-analyze

This blog post discusses the corresponding econometric study:

Levinson, A. (2016). How much energy do building energy codes save? Evidence from California houses. American Economic Review, 106(10), 2867-94. http://faculty.georgetown.edu/aml6/pdfs&zips/BuildingCodes.pdf

The main criticisms made by Nadel to Levinson’s study is that he focused his analysis on electricity consumption, while most of the effects of the Californian building codes are to be found in gas consumption. This is an example of the difficulties that can be encountered to separate the energy consumption from various end-uses when using billing data.

For examples of evaluation of building codes or standards for buildings in the IEA guidebook of 2005, see:

http://www.ieadsm.org/wp/files/Exco%20File%20Library/Key%20Publications/Volume%202%20total.pdf

General guidance on evaluations

Baumgartner, R. (2017). Chapter 12: Survey Design and Implementation for Estimating Gross Savings Cross-Cutting Protocol. The Uniform Methods Project: Methods for Determining Energy-Efficiency Savings for Specific Measures. Prepared for NREL (National Renewable Energy Laboratory), September 2017. http://www.nrel.gov/docs/fy17osti/68568.pdf

EVO, 2006. International Performance Measurement & Verification Protocol: Concepts and Practices for Determining Energy Savings in New Construction – Volume III Part I. Efficiency Valuation Organization, January 2006. Available in English, Bulgarian, Croatian, Czech, Polish and Romanian. https://evo-world.org/en/ipmvp-current/archive/applications-volume-iii/volume-iii-part-i-2006

About efficiency standards for residential dwellings

Menkveld, M. & Leidelmeijer, K. (2010) – “Evaluatie EPC-aanscherping woningen” (Evaluation of energy performance standards in dwellings; full text in Dutch only, English abstract) https://publicaties.ecn.nl/ECN-E–10-043

Wade, Joanne, and Nick Eyre, 2015. Energy efficiency evaluation: The evidence for real energy savings from energy efficiency programmes in the household sector. London: UK Energy Research Centre. Available at: http://www.ukerc.ac.uk/programmes/technology-and-policy-assessment/energy-efficiency-evaluation.html (see more specifically pp.41-42 for a review of papers about evaluations of building codes)

About compliance and related issues:

Pan, W., & Garmston, H. (2012). Compliance with building energy regulations for new-build dwellings. Energy, 48(1), 11-22.

Rasin, J., Chappell, C. & Gouin, A. (2013). Evaluating Code Compliance Enhancement – A Love Story. Proceedings of IEPEC 2013. https://www.iepec.org/wp-content/uploads/2018/02/153.pdf

Rovito, M. & Pettit, T. (2015). Cracking the Code: Impact Evaluation Methods of ARRA-Funded Energy Code Initiatives. Proceedings of IEPEC 2015. https://www.iepec.org/wp-content/uploads/2018/02/2015paper_rovito_pettit-1.pdf

Storm, P. (2015). How Do We Know Commercial Codes Deliver Energy Savings? Designing and Testing a New Methodology for Assessing Commercial Code Compliance. Proceedings of IEPEC 2015. https://www.iepec.org/wp-content/uploads/2018/02/2015paper_storm_phoutrides-1.pdf

Vine, E. L. (1996). Residential building code compliance: Implications for evaluating the performance of utility residential new construction programs. Energy, 21(12), 1051-1058. https://www.osti.gov/servlets/purl/418459

Yu, S., Evans, M., & Delgado, A. (2014). Building energy efficiency in India: compliance evaluation of energy conservation building code (No. PNNL-23217). Pacific Northwest National Lab.(PNNL), Richland, WA (United States). https://www.pnnl.gov/main/publications/external/technical_reports/PNNL-23217.pdf

About rebound effect:

Aydin, E., Kok, N., & Brounen, D. (2017). Energy efficiency and household behavior: the rebound effect in the residential sector. The RAND Journal of Economics, 48(3), 749-782. https://sustainable-finance.nl/upload/researches/Aydin-et-al_Energy-Efficiency-and-Household-Behavior.pdf

About long-term effects (persistence of savings, innovation, etc.):

El-Shagi, M., Michelsen, C., & Rosenschon, S. (2017). Empirics on the Long-Run Effects of Building Energy Codes in the Housing Market. Land Economics, 93(4), 585-607.

About non-energy impacts:

Bruegge, C., Deryugina, T., & Myers, E. (2019). The Distributional Effects of Building Energy Codes. Journal of the Association of Environmental and Resource Economists, 6(S1), S95-S127. https://web.stanford.edu/~cbruegge/papers/2018-01-05-Draft.pdf

About lifecycle analysis and embodied energy:

Crawford, R. H., Bartak, E. L., Stephan, A., & Jensen, C. A. (2016). Evaluating the life cycle energy benefits of energy efficiency regulations for buildings. Renewable and sustainable energy reviews, 63, 435-451. https://minerva-access.unimelb.edu.au/bitstream/handle/11343/121775/Evaluating%20the%20life%20cycle_file%20for%20upload_RSER-S-15-02911.pdf?sequence=5