24 – Evaluating Information measures on energy efficiency in Services using Billing analysis
This specific guide can be applied to evaluate the savings due to Information on saving options in the sector Services (except buildings) using the method Billing analysis. It includes guidance and explanations specific to this combination of types of policy measure, sector and method. As well as links to general guidance and explanations that can also apply to this combination.
The primary audience for this guide concerns energy efficiency programme designers, implementers or supervisors, and evaluators looking for guidance on the evaluation process of energy savings in the scope of this specific guide.
Although the application of the specific guide will generally concern the (sub)national level, account will be taken of issues at EU level when relevant (e.g. the specific format of saving figures for the EED).
This specific guide is not about the preceding step in the evaluation process, the choice of the method. About this previous step in the evaluation process (the choice of the method), see the guidance provided here. However, after presenting the capabilities and limitations of the specific guide at hand, the user will be offered alternatives for the method within this specific guide (see section 6).
The objective of this guide is to provide:
- Information on the scope of the specific guide that enables the user to decide whether this specific guide is suited to his/her needs, and whether complementary or additional method(s) could be needed or useful (section 2);
- Guidance about specifying the evaluation objectives and requirements (section 3);
- Guidance about key methodological choices to calculate energy savings (section 4);
- Guidance about the inputs (data requirements) and outputs of the method (energy savings metrics) (section 5);
- Possible alternative methods (with pros and cons) (section 6)
- Background about evaluation results other than energy savings (section 7);
- Relevant examples, case studies and/or good practices (section 8);
- Relevant references for further reading (section 9).
The specific guide is intended for assessing realised (ex-post) energy savings. However, account is taken of earlier (ex-ante) evaluations of expected savings, if available (see section 4).
The focus of the specific guide is on impact evaluation, i.e. determining the energy savings, but not on how this has been reached through a step by step process with intermediate results (process evaluation). Readers looking for the basic and general principles of energy efficiency evaluation may find the following link useful.
2.1 About Information on savings options
The focus of this specific guide is restricted to dedicated information on possible saving actions in the sector Services excluding buildings. It is assumed that an energy efficiency advisor suggests possible saving action to an interested end-user, either through an information center or as part of a (subsidized) audit. It does not concern information campaigns through media like television or internet.
This so-called facilitating policy measure shows the end-user which saving actions could be implemented. But information but will not always lead to savings. The incentive to actually implement actions can be provided by other policy measures, such as energy taxes or voluntary agreements between a group of end-users and government.
The information activity can start from government sponsored agencies but also from branch organizations or energy companies involved in energy savings activities. Actual dissemination of information can be done by (certified) advisors on energy efficiency.
More detailed information on the evaluation of the policy measure at hand can be found here.
2.2 Evaluation for a combination of policy measure types
When dedicated information is combined with financial support (grants or tax rebates), it is normally assumed that the overall savings are mainly the result of the financial support. In this case a specific guide on financial incentives (and on Services or on billing analysis) might be preferred, also because these specific guides generally provide better data on the number of saving actions based on decisions about subsidies for saving actions.
When dedicated information is combined with other policy measures, such as energy taxes or voluntary agreements, the specific guide at hand can be applied to calculate the combined savings effect of both policy measures. However, the specific guide is not capable of attributing part of the (overall) calculated savings to each of the policy measures (see also Double counting in section 4 on Calculating Gross and Net savings).
2.3 Evaluation when combined with energy taxes
The calculated savings effect for subsidies will overlap with that of the energy tax. The energy tax indeed increases the cost-effectiveness of the energy saving actions from the point of view of the investors (most often the building owners). Therefore, both policy measures, the subsidies and the energy tax, improve the economic conditions for the investments in energy saving actions. The subsidies will be reducing the investment cost. While the energy tax will increase the economic benefit from the actions (by increasing the economic value of the energy savings), thereby reducing the payback time.
The guide is not capable of attributing part of the (overall) calculated savings to either the policy measures at hand or the energy tax. It would indeed be very difficult to find a comparable group of buildings and occupants who would not subject to a different energy tax, as energy taxes are usually applied nation-wide.
Distinguishing the effects of subsidies and energy taxes can be investigated with modelling of investment behaviours, by simulating scenarios with different policy packages. Calibrating this type of model usually requires data about the number of actions implemented in the country (or other area considered for the evaluation). See (Boonekamp, 2004).
This type of analysis can be complemented with surveys or modelling about the willingness to pay for energy saving actions. Such studies can be helpful for the design of the financial incentives (e.g. identifying the level of incentive that can trigger actions) and to provide a basis to evaluate the additionality of the financial incentives (or the free-rider effects) (see also section 4).
2.4 About Services excluding buildings
Information on (sub) sectors defined in the Toolbox can be found in this link, chapter 2, p.17
Services excluding buildings concerns energy consumption for a large array of activities in the buildings, e.g. ICT equipment in offices, food cooling in supermarkets or restaurants, compressed air systems in garages, hot water preparation in laundries, restaurants and hotels or ovens in bakeries.
All systems (and saving actions) connected to space condition in buildings are not covered.
2.5 Evaluation for cross-sector saving actions
This specific guide is not applicable to evaluate cross-sector savings, such as more efficient electric motors, because the method (see next section) is not fit to show the (minor) effect of these saving actions on overall energy consumption.
2.6 About billing analysis
Billing analysis concerns total energy consumption of individual end-users. It can show a change in consumption after implementing one or more saving actions, which represents the unitary savings of the saving action(s). Combined with the number of actions (see next section on complementary method) this provides total savings.
However, for Services a large part of total energy consumption is generally connected to the conditioning of the building. Therefore, the observed change may not be due to evaluated saving actions outside the realm of the building. In order to have reliable saving figures total energy consumption of end-users should be corrected for energy used for the building functions.
Often the energy consumption for non-building purposes concerns mainly electricity, while electricity use for the building is limited. If that is the case, the evaluation can be executed based on billing analysis for electricity consumption only.
General information about the various evaluation methods can be found be found here, table 1 and 2. This source also covers the combination of the method at hand with other methods, which will be dealt with below.
2.7 Complementary methods to determine total savings
Complementary methods are methods that are required, in addition to the primary selected method, to calculate total energy savings.
The method at hand is meant to calculate unitary savings. These unitary savings should be multiplied by the number of actions to have the calculated total savings. The number of actions or participants can be obtained in various ways. See this link, table 2 and 3.
If there is a subsidy scheme (see section on combination of policy measures) the number of actions is often directly available from the monitoring of subsidized saving actions.
For more information about methods to calculate unitary savings, see this link, table 2.
2.8 Additional methods to increase reliability of the results
An additional method can be applied on top of Information on saving options to improve the reliability of the evaluation results and/or the cost-effectiveness of the evaluation approach.
Billed energy consumption can be influenced by many other factors than the policy measure/saving actions at hand. Part of these can be corrected for (see section 4, chapter on normalization). However, a regular check of the results, using an additional method, is needed to assure reliable results.
The additional method Measurement can be applied for a selection of end-users, where the measurement concerns (as much as possible) the energy consumption that is relevant for the saving actions at stake.
Because measurement is only done for a limited number of end-users and not for every year this can increase the reliability of the savings figures in a cost-effective way.
For possible combinations with an additional method see chapter 6 in this link.
3.1 Meeting evaluation goals and ambition
The following table shows whether this tool can be used to report on general evaluation goals or criteria. See also this document.
|General types of evaluation goals or criteria||Level of ambition||Remarks|
|Calculation of realized energy savings from saving actions||Limited||Due to uncertain follow up suggested saving actions|
|Calculation of energy savings attributed to the policy measure(s)||Low||See above and possible contribution of other policy|
|Cost-effectiveness of saving actions (for end-users)||Limited||Cost effectiveness per saving actions can be calculated but not number of actions (see above)|
|Cost-effectiveness of policy (government spending)||Low||Cost of providing information known but not the results (see above)|
|CO2-emission reduction from saving actions||Limited||See above|
|CO2-emission reduction attributed to the policy measure(s)||Low||See above|
For more information on verification of actual energy savings and attribution/baseline/corrections, see section 4, and for cost-effectiveness and emission reduction see section 7.
3.2 Reporting expectations
Possible reporting options:
- Gross and Net savings
- Yearly savings (no cumulative savings because no specific saving actions, with life times, are regarded, see section 5)
- Follow-up rate of suggested saving options to end-users, per type of saving action and given other policy measures focusing on the same end-users (see section 2).
3.3 Time frame for evaluation
The length of the period under evaluation is dependent on the active period of the policy measure, the need to monitor developments before the implementation of savings actions (in case of methods based on before/after saving actions), and the time needed to present (reliable enough) results or impacts that fit into the decision making process. In some cases, the periodicity of evaluation can be set by law.
The application of the Before/after baseline (see chapter 4) asks for billing data before the saving actions due to the policy measure are implemented. Therefore, the time frame is larger than the active period of the policy.
Before the introduction of the policy measure at hand the system of monitoring the suggested saving actions and the check on follow-up of the suggested saving actions should be set up; see also the report on planning of an evaluation.
3.4 Expertise needed for chosen method
The monitored savings concern a large array of energy uses in many different subsectors. Expertise is needed on the various saving options and the way information can be disseminated. For the check on the follow-up of advised saving actions expertise is needed on the set up of sample surveys.
3.5 Boundaries for the evaluation
The field of application is restricted to the non-building energy consumption, i.e. without energy consumption for heating, cooling or lighting of the building.
In part of this sector, e.g. shops or restaurants, the business space can be combined with the private space for living. In that case the energy consumption should only concern that for business. If no separate metering is applied for business and private uses, the bill must be corrected for private energy consumption.
This section deals with key methodological choices to be considered when calculating energy savings: consistency between ex-ante and ex-post evaluation, baseline, normalization and adjustment factors. These choices are important to document when reporting energy savings, to ensure the transparency of the results.
The method calculates the savings due to dedicated information on saving options to end-users. The method should not only take into account the actual implementation of saving actions but also the path from information to implementation.
It is assumed that an energy efficiency advisor suggests possible saving action to an interested end-user, either through an information center or as part of a (subsidized) audit. The advices are registered and the follow-up, the actual implementation of specific saving actions, is checked sample wise.
The total savings are calculated from the number of advised savings actions, the follow-up rates and the unitary savings from billing analysis.
4.1 Matching method with earlier ex-ante evaluation
From the viewpoint of methodological consistency and data availability using the same method in the ex-ante evaluation and in this specific guide on ex-post evaluation might be an obvious choice. However, for ex-ante evaluation billing is not applicable due to lack of observed data.
The billing analysis method for the ex-post evaluation can be combined with other methods for the ex-ante evaluation such as deemed savings, engineering estimate and stock modeling, depending on the evaluation objectives, timeframe and data available for the situation after implementing the actions. For possible combinations of methods applied ex-ante and ex-post, see chapter 7, calculation approaches here.
Ex-post billing analysis can be matched with the ex-ante method deemed savings, where deemed savings are defined for each saving action. The method engineering estimate is often used for more complex energy using systems like buildings, which are not regarded here. Ex-ante stock modeling is not obvious given the large variety in saving actions for Services excluding buildings (see section 2).
In practice, ex-ante and ex-post evaluations are applied consecutively. The ex-ante deemed savings evaluation can be improved by checking the deemed savings through the ex-post billing analysis.
4.2 Calculation baselines
Energy savings are defined in general as the difference between the actual situation and a reference situation without the saving actions (and without the policy measures that influence these saving actions). In case of saving actions the reference situation can be defined using various calculation baselines: Before/after, With/without, Trend, Target/control group and Minimum efficiency standards.
For further background, see here.
Total savings are found by multiplying the unitary savings per action by the number of actions. The chosen baseline can define the total savings through the unitary savings, but also through the number of actions.
For billing analysis the applicable baseline option can be “Before/after”, where “before” represents the situation before the implemented savings actions and information dissemination.
An alternative is the baseline option “Target/control group” that enables to correct for (almost) all other influences than information, including saving actions in the absence of dedicated information. But this baseline asks for a (sample wise) monitoring of a group of non-participating end-users. The choice between baseline options will be dependent on the requirements formulated in chapter 3.
See also this link.
4.3 Normalization factors
With the chosen baseline a change in energy consumption can be calculated, but this change should be corrected for influences other than the saving actions. These so-called normalization factors can be weather (with effect on consumption), the rebound effect and changes in energy using activities per end-user, such as production (industry), occupation rate (buildings) or car usage (transport). All normalization factors affect total savings through the unitary savings.
The rebound effect can be ignored as it generally concerns space heating in dwellings or fuel use in cars; moreover, this effect is already incorporated in the “after” billing data.
Using the baseline Before/after a correction must be applied on billing data for weather. Also a correction can be applied for changes in activity level (dependent on subsector and saving action).
For the baseline “Target/Control group” all other normalization factors can be left aside as they will be the same for Target and Control group, given a sound sample set-up.
4.4 Adjustment factors
Adjustment factors define which part of the calculated energy savings can be attributed to a policy measure or meets the definition of savings specified in the evaluation objectives or reporting requirements (see next section on “Calculating Gross and net savings).
Adjustment factors can concern the Free rider effect, the Spill-over/multiplier effect, Additionality, Non-compliance and Double counting (For a general discussion about adjustment factors, see here.
Additionality and non-compliance are connected to unitary savings, while free riders and spill-over work through the number of actions.
Double counting might be relevant in case of another policy measure focusing on the same saving actions, but can only be accounted for at a higher level than individual specific guides (see section on Calculating Gross and Net savings and in this link and also here and here.
Here the adjustment factor in priority is only free riders. The adjustment factor free rider effect is defined as participants making use of the advice facilities without actually implement them. The free rider effect is set equal to the follow-up fraction, the savings weighted fraction of suggested saving actions not implemented (after some years).
Spill-over (implementation by non-participants) is possible but not such obvious to be taken into account. Additionality concerns specific saving action and is left out because dedicated information concerns various saving actions which have different follow-up rates. The adjustment factor Non-compliance can be ignored as it generally is connected to improper acting as to regulation.
In case of the Target/control baseline most adjustment factors can be ignored as the comparison controls for the effects.
4.5 Calculating Gross and Net energy savings
Gross savings concern the calculated savings from saving actions using a chosen baseline and normalization factors. Net savings concern the savings attributed to policy measures or to a stakeholder (e.g. an energy company with an obligation to realise savings at their customers).
When calculating savings a distinction must be made to the unitary savings and number of actions (see for example this document).
The gross unitary savings can be calculated using the Before/after baseline. Per participant the unitary savings follow from the change in billed energy consumption before and after the implementation of saving actions. The number of actions is the number of advised saving actions times the follow-up rate.
Net unitary savings are equal to gross unitary savings because adjustment factors additionality and non-compliance are not to be applied (see section on adjustment factors). The net number of actions follows from the gross number, corrected for the adjustment factor free riders (see section on adjustment factors).
In case of applying the Target/control group baseline there is no need for correcting the (unitary) savings for normalization factors and for adjustment factors the same holds as for Before/after.
Finally total savings can be corrected for the Double counting effect (see section on adjustment factors), to be done at the level of the overall policy portfolios.
See also section 9 on References.
5.1 Main data requirements, sources and collection techniques
Data requirements specified in the table below correspond to the calculation of energy savings, when using the baseline option Before/after.
|Calculation subject||Data requirements||Possible data sources and collection techniques|
|Energy consumption (excluding building related)||Billed energy consumption of participants and fraction for building||Bills from energy companies and (average) fraction for buildings per subsector from literature)|
|Advised saving actions||Number of contacted end-users (per subsector) and advised actions (per type)||Monitoring of advice activity of information centers or the content of executed audits|
|Fraction of suggested saving actions implemented (follow-up rate)||Implemented fraction of advised actions for sample of participants||Survey on sample of participants|
Data issues with the additional method
If the measurement method is applied the energy consumption data can cover only the part connected to the saving actions to be evaluated. However, measurement for each participant is too costly and will only be done for part of the participants. In this way total energy consumption from billing can be corrected for building-related energy use and possibly for other energy use not connected to the saving actions.
For more details about the evaluation of net energy savings, see this topical case study.
5.2 Energy savings in final terms or in primary terms
Energy savings can be expressed in final terms or in primary terms. See definitions about primary and final energy here.
With the specific guide at hand the savings in final terms can be calculated. It can also calculate savings in primary terms provided that billing analysis shows the savings per energy carrier and primary factors are available to convert the savings in final terms to savings in primary terms.
5.3 Energy savings over time
Implemented saving actions in a year lead to savings over a number of consecutive years. E.g. a more efficient boiler can save gas over its lifetime of about 15 years, insulation over up to 30 years and more efficient computers up to 5 years. There is also an effect of season and climate that may have to be taken into consideration.
Energy savings can be calculated in different metrics in terms of time reference, for example: year-to-year, annual, cumulated annual, cumulative. See the definitions here.
Billing analysis cannot show the yearly savings per type of implemented saving action. It cannot provide cumulative savings because of the different lifetimes per saving action. For the same reason it cannot provide cumulative savings according to the Energy Efficiency Directive nor discounted cumulative savings.
6.1 Alternatives for the chosen method
Often other savings calculation methods can be applied as well, although they will all have pros and cons regarding various aspects dealt with in preceding sections.
The table below presents the pros and cons of the deemed savings method compared to method applied for this specific guide.
|Type of method||Pros||Cons|
|Method at hand||Billing data easily available and showing actual developments||Total energy consumption to be corrected for building-related energy use|
|Deemed savings method||Savings are available for specific saving actions
Method can also to be applied for ex-ante evaluation
|No savings in other formats
Data needed over longer period
To be estimated for many saving actions in many different situations
7.1 Calculating avoided CO2 emissions
Avoided (equivalent) CO2 emissions can be evaluated from the energy savings by applying emission factors. Four key aspects are to be taken into account when choosing the emission factor(s):
- Emission factors vary according to the energy type, so the data about energy savings need to be available per energy type.
- Emission factors for a given type of energy can vary over time (especially for electricity).
- Emission factors can take into account:
- Direct emission factors: that take into account the emissions generated when producing the energy used;
- Lifecycle emission factors: which take into account all the emissions generated from the extraction of materials and resources up to the dismantling and recycling of the studies system.
Due to the differences that the choice of emission factor(s) can induce, it is important to document and justify what emission factor(s) has(have) been used and give details on their representativeness and uncertainty.
For the specific guide at hand the reduction in CO2-emissions can only be calculated when savings are calculated per relevant energy carrier and a specific emission factor is available for each energy carrier
The avoided emission of other greenhouse gases due to energy savings are not taken into account here, as these emissions (and more specifically their reductions) are generally negligible compared to CO2 (apart from policy measures targeting the agriculture sector).
IPCC (Intergovernmental Panel on Climate Change) provides a detailed database of peer-reviewed emission factors.
7.2 Calculating cost-effectiveness
Cost-effectiveness is the ratio between costs to achieve energy savings and the amount of savings and possibly other benefits.
A distinction can be made according to the point of view adopted to assess cost-effectiveness:
- Cost-effectiveness for the end-user or participant
- Cost-effectiveness for society at large
- Cost-effectiveness for the party that takes responsibility for saving targets (government or actor with an Energy Efficiency Obligation)
See further here, report on Evaluation into Practice: lessons learnt from 23 evaluations of energy efficiency policies, section 13.1.
For the evaluation method and sector in this specific guide no cost-effectiveness can be calculated because the savings are not known by specific saving actions, each with their investments, lifetime and yearly savings.
7.3 Calculating other co-benefits
Possible co-benefits from saving energy concern:
- Employment (maintenance or creation)
- Reduction of energy poverty
- Other emission reductions (NOx, SO2, fine particles, etc.)
- Better indoor comfort
- Reduced dependency on (insecure) energy import
For the same reason as cost-effectiveness no benefits can be calculated.
7.4 Other aspects of importance
See references in section 9.
About the policy measure Information/audits on saving options
- Elias Andersson, Oskar Arfwidsson, Victor Bergstrand, Patrik Thollander, Journal of Cleaner Production 142 (2017) 2133-2139: A study of the comparability of energy audits program evaluations, Andersson et al, Linkoping University, Journal of Cleaner Production, 2016,
- Comprehensive monitoring system – essential tool to show the results of the energy audit and voluntary agreement programmes, Ulla Suomi, Motiva Oy, ECEEE 2007 Summer Study,
- Evaluation of free-of-charge audits, Dyhr-Mikkelsen and Bach, ECEEE Summer Study 2005 Assessment of existing evaluation practices and experience, SEA, 2005),
- The Effect of Energy Audits in Danish Industry – Evaluation of a DSM Programme, Larsen et al, Energy Studies Review Vol. 14. No, 2. 2006 pp 30-41 (>20 MWh electricity),
- Goater, Aurélie (Alpheeis) for ADEME, Compteurs communicants gaz, pratiques des ménages et économies d’énergie (Gas smart meters, household practices and energy savings). This literature review aims at defining the optimal conditions for gas communicating meter deployment in the residential sector in particular in terms of energy supply & demand management. It gathers recommendations on what kind of information device should be tested, or which method should be used to support households in managing their gas consumption in a sustainable manner.