27 – Specific guide on evaluating outcome of Energy Efficiency Plans (EEP) using voluntary agreements in industry using the energy indicator method
This guide can be applied to evaluate the savings due to a subsidy scheme for a voluntary agreement in the sector industry using the method of engineering estimate. It includes guidance and explanations specific to this combination of types of policy measure, sector and method, as well as links to general guidance and explanations that can also apply to this combination.
The primary audience for this guide is energy efficiency programme designers, implementers or supervisors, and evaluators looking for guidance on the evaluation process of energy savings in the scope of this guide.
Although the application of the guide will generally concern the (sub) national level, account will be taken of issues at EU level when relevant (e.g. the specific format of saving figures for the EED).
This guide is not about the preceding step in the evaluation process, the choice of the method. About this previous step in the evaluation process, see the guidance provided here. However, after presenting the capabilities and limitations of the guide at hand, the user will be offered alternatives for the method within this guide (see section 6).
The objective of this guide is to provide:
- Information on the scope of the guide that enables the user to decide whether this guide is suited to his/her needs, and whether complementary or additional method(s) could be needed or useful (section 2);
- Guidance about specifying the evaluation objectives and requirements (section 3);
- Guidance about key methodological choices to calculate energy savings (section 4);
- Guidance about the inputs (data requirements) and outputs of the method (energy savings metrics) (section 5);
- Possible alternative methods (with pros and cons) (section 6)
- Background about evaluation results other than energy savings (section 7);
- Relevant examples, case studies and/or good practices (section 8);
- Relevant references for further reading (section 9).
The guide is intended for assessing realised (ex-post) energy savings. However, account is taken of earlier (ex-ante) evaluations of expected savings, if available (see section 4).
The focus of the guide is on impact evaluation, i.e. determining the energy savings, but not on how this has been reached through a step by step process with intermediate results (process evaluation).
Readers looking for the basic and general principles of energy efficiency evaluation may find the following link useful.
2.1 About the policy measure: voluntary agreements in industry using energy indicators
More detailed information on the evaluation of grants and subsidies can be found here.
The focus of this guide is restricted to voluntary agreements in industry, using energy indicators as method for determining energy savings. The policy measure of voluntary agreements in industry is typically taken up by a government ministry, for example the ministry of Economic affairs. Rather than confronting the sector with for example, a normative standard or an obligation, this measure is based on formulating , with the relevant stakeholders in the industry sector, common goals of energy savings in a certain time period, typically 5-10 years ahead. In most cases the Ministry acts as a process facilitator. One of the reasons of this voluntary approach is the diverse character of the industry sector. For example, in the Netherlands, on one hand there is a large chemical and petrochemical industry, on the other hand there are a number of much smaller SMEs.
The evaluation is usually focused on a program of energy efficiency measures, undertaken by the industry sector on the basis of voluntary EEP’s (Energy Efficiency Plans). The elements of this evaluation include the achieved energy savings in terms of intended savings and realized savings; but also interaction with other policy instruments; the evaluation of side effects (international market playing field, the execution of the measure; the costs of the measure).
Energy efficiency indicators are, in general, indicators that will help to demon¬strate if one thing is more energy efficient than another. For instance, is one car more energy efficient than another one in terms of fuel use per 100 kilometres? Is the energy heating consumption per floor area of the residential sector lower or higher this year than last year, or ten years ago? Is the energy con-sumption per tonne of cement lower or higher in country A compared with country B?
Energy efficiency indicators can be very aggregated (for instance, total appliances energy consumption per appliance) or disaggregated (for instance, average heating consumption per floor area of single houses using natural gas for heating). They are usually composed of energy consumption as numerator and an activity data as denominator. There are some exceptions, such as the “energy” consumption of cars, which can be expressed in volume (liters, gallons) and not converted into energy units.
Energy consumption can be expressed in various units (kWh, joule, tonnes of oil equivalent, etc.), while activity data cover a wide range of activities: production of cement, floor area, passenger-kilometres, employees, etc., expressed in as many units as activities (tonnes, square metres, kilometres, number of employees, etc.).
2.2 Evaluation for a combination of policy measure types
In case the policy measure of voluntary agreement is combined with a second policy measure, for example a government subsidy, it will turn out that it is difficult to ascribe part of the overall savings to each policy measure type.
A practical approach for combined policy measure types is to perform the evaluation for the set of policy measures as such. The combination is characterised by the policy measure which is seen as most important as to realize savings. A drawback of this approach is that the effectiveness of each separate policy measure can hardly be assessed.
2.3 Evaluation when combined with energy taxes
The calculated savings effect of a voluntary agreement may overlap with that of energy taxes. The guide is not capable of attributing part of the (overall) calculated savings to either the policy measures at hand or the energy tax. For dealing with this overlap see section 4 on Gross to Net savings.
2.4 About the industry sector
Information on (sub) sectors defined in the Toolbox can be found here, chapter 2, p.17
The sector industry that we do cover in this guide is the sector in the economy which is characterized by production and use of material goods, applying a high degree of automation and mechanization.
Although the guide is meant to cover all end-use sectors in industry it will mostly be applied to a selection of sectors within industry. In every country, the industry is divided in different sectors (usually defined by the national statistics office), like food & beverage industry, chemical and pharmaceutical industry, paper industry, rubber and plastic industry electrotechnical industry. For example, a dedicated voluntary agreement could be made with all companies that produce chemicals and pharmaceuticals , in such a specific case, next to individual companies, also a professional organization (e.g. in the Netherlands KNCV – Royal Netherlands Chemical Society) can be involved as stakeholder in a voluntary agreement.
Although the guide is meant to cover all end-use sectors it can also be applied to a selection of sectors with each a specific tax.
2.5 Evaluation for cross-sector saving actions
This guide is also applicable to evaluate cross-sector savings, provided that the data needed are available for all relevant sectors.
2.6 About the energy indicator method
Information about the various evaluation methods can be found here, table 1 and 2. This source also covers the combination of the method at hand with other methods, which will be dealt with below.
Energy consumption and related energy indicators can be developed at different levels of aggregation depending on the purpose for which they will be used and the level of information available. The aggregation level is very important as it determines the extent to which structural differences affect the results observed (figure 1). For the use of indicators on energy evaluation in a sector of industry in a particular country, the indicators on process/appliance level are mostly used. The relevant indicator is in this case the energy consumption per unit of physical output. This figure, in combination with the output of product per year, gives the final energy use for a specific process or appliance.
The data on energy consumption may be available in national energy balances per sector (national statistics office) or in publications from IEA on energy balances.
* Total Final Consumption per Gross Domestic Product
Figure 1 Energy indicators and aggregation level (source : IEA)
The most aggregated indicators include, for instance, the share of residential consumption in TFC and the overall residential consumption per capita, per dwelling or per floor area. If these indicators allow very rough comparisons (however often misleading) among countries and evolution over time, they cannot be assimilated to indicators of energy efficiency as such.
There are also aggregated indicators that can be used for specific purposes. For example the electrification rate of households in a country (total or broken down between urban and rural areas) can be used for feeding studies on electrification programmes.
2.7 Complementary methods to determine total savings
Complementary methods are methods that are required, in addition to the primary selected method, to calculate energy savings.
The method at hand provides directly the savings, instead of combining unitary savings (per action) with the number of actions. Hence no complementary method is needed. For further information see this link, table 2 and 3.
2.8 Additional methods to increase reliability of the results
An additional method can be applied on top of the energy indicator method to improve the reliability and accurateness of the evaluation results and/or the cost-effectiveness of the evaluation approach.
The method of energy indicator is a method that is based on statistical data on (for example) energy use per unit produced. In order to verify the energy indicator for a particular product or service the method may be combined by an additional method like measurement (direct measurement or billing analysis) or engineering method (for this method in combination with a voluntary agreement, please see the corresponding guide). The combination of methods can increase the reliability of the savings figures in a cost-effective way.
For possible combinations with an additional method see chapter 6 in this link.
3.1 Meeting evaluation goals and ambition
The table shows whether this guide can be used to report on general evaluation goals or criteria. See also this document.
|General types of evaluation goals or criteria||Level of ambition||Remarks|
|Calculation of realized energy savings from saving actions||Fair||Saving actions should be well documented (e.g. action of better heat integration in a chemical process)|
|Calculation of energy savings attributed to the policy measure(s)||Fair||Policy measure should be well documented|
|Cost-effectiveness of saving action (for end-users)||Fair||Required investment can be compared with savings in final energy use|
|Cost-effectiveness of policy (government spending)||Limited||May be combined with cost-effectiveness of saving action|
|CO2-emission reduction from saving actions||Fair||Can be integrated in engineering method|
|CO2-emission reduction attributed to the policy measure(s)||Limited||Can be combined with CO2 emission reduction from saving actions|
For more information on verification of actual energy savings and attribution/baseline/corrections, see section 4 of this guide. For cost-effectiveness and emission reduction see section 7 of this guide.
The evaluation goals could be framed as providing a targeted energy savings of X PJ in a number of years (usually a target year that is 5 or 10 years ahead is used). This timeframe has also to be included in the framing of the voluntary agreement.
It is important that in a voluntary agreement, all stakeholders have a similar view and starting point with regard to the goals and ambition of the program.
3.2 Reporting expectations
The reporting of the evaluation results should also be defined in an early stage when setting up the program around the voluntary agreement. This will result in a similar expectation among the stakeholders of the voluntary agreement. The recipients of the results will be the participating companies, the agency or ministry that has initiated the voluntary agreement program and also the public at large, as the reporting will usually be a public document.
Several indicators and metrics to assess the results/impacts may be used, also taking in consideration the differences in the various types of possible recipients of the evaluation, etc.; these include:
- Gross and Net savings
- Yearly or cumulative savings
- Final energy savings or savings in Primary energy,
3.3 Time frame for evaluation
Relevant information regarding the time frame of the evaluation can include: the period for which the evaluated policy measure is active, the planning of the different activities necessary for the evaluation (the timely start of monitoring the “before” development, frequency and timing of data gathering, final monitoring after end of policy, etc.) and estimating roughly the time needed to make an evaluation depending on the type of method used]
The length of the period under evaluation is dependent on the active period of the policy measure, the need to monitor developments before the implementation of savings actions (in case of methods based on before/after saving actions), and the time needed to present (reliable enough) results or impacts that fit into the decision making process. In some cases, the periodicity of evaluation can be set by law.
The planning of evaluation activities concerns regular monitoring of energy consumption and factors that define consumption, intermediate check of (ex-ante) estimated (unitary) savings through measuring or surveys, intermediate evaluations to improve the policy implementation and the final evaluation and reporting.
See also planning of evaluation in the link here.
Regarding a voluntary agreement in industry, it typically takes several years for implementation. An evaluation therefore can only be done after the program has ended, so a typical timeframe when an evaluation is useful is typically 5 years; the time frame of the actual evaluation itself is a couple of months, for example ½ year.
3.4 Expertise needed for chosen method
Expertise required for the chosen method of energy indicators include some knowledge on access to energy statistical data per country or sector within a country. Dependent on the eventual complementary method, additional expertise may be needed. The expertise required has to be addressed in the overall evaluation protocol of this guide, at the start of the evaluation process.
3.5 Boundaries for the evaluation
For this guide boundaries that are present are usually linked to the different sectors(s) that take place in the evaluation.
This section deals with key methodological choices to be considered when calculating energy savings: consistency between ex-ante and ex-post evaluation, baseline, normalization and adjustment factors. These choices are important to document when reporting energy savings, to ensure the transparency of the results.
The method of energy indicator is strongly based on statistical data available for geographic regions, specific sectors or applications. The Odyssey Efficiency Indicator is a good reference for efficiency indicators that should be well defined in a program regarding the application of energy efficiency indicators).
4.1 Matching method with earlier ex-ante evaluation
The method of energy indicators to assess energy savings may be matched with possible ex-ante estimates for a particular sector or a specific industrial process.
From the viewpoint of methodological consistency and data availability using the same method in the ex-ante evaluation and in this guide on ex-post evaluation might be an obvious choice. However, for ex-ante evaluation only a few methods are usually considered, namely deemed savings, engineering estimate and stock modeling.
A different method than the one(s) used for the ex-ante evaluation can be applied for the ex-post evaluation, depending on the evaluation objectives, timeframe and data available for the situation after implementing the actions. For possible combinations of methods applied ex-ante and ex-post, see chapter 7, calculation approaches in this document.
If the energy indicator method does not provide an acceptable combination with the earlier applied ex-ante method it might be useful to select another method (see examples of alternatives in section 6).
In practice, ex-ante and ex-post evaluations are applied consecutively. The ex-post evaluation builds on an ex-post evaluation that makes use of data coming from previous ex-post evaluation or studies (e.g. about previous periods of the same policy measure, or about the same types of energy saving actions as the ones promoted by the new policy measure). These previous ex-post studies could have used another type of method as well.
4.2 Calculation baselines
Energy savings are defined in general as the difference between the actual situation and a reference situation without the saving actions (and without the policy measures that influence these saving actions). In case of saving actions the reference situation can be defined using various calculation baselines. For further background, see further here.
For the calculation of energy savings the baseline options concern Before/after, With/without and Target/control group. The Before/after calculation can be applied for existing energy using systems (e.g. dwellings). For new energy using systems, without a “before” situation, the With/without calculation can be applied. See also this link.
In the application of this guide using the energy indicator method, only the baseline option Before/after can be used. The applicable baseline(s) regard(s) the Before/After option, where the “before” is the baseline for the actual development in the voluntary scheme, so the energy indicator before start of the measure. The energy indicator after implementation of the measure represents the difference between the two which is used to calculate the energy savings.
4.3 Normalization factors
The calculation with the before/after baseline considered in the previous section provides a change in energy consumption that should be corrected for influences on energy consumption other than the saving actions. These so-called normalization factors can be weather (with effect on consumption), the rebound effect and changes in energy using activities, such as production (industry), occupation rate (buildings) or car usage (transport).
Normalization factors that also comprise performance gap and the pre-bound effect (see this link to doc on normalization and adjustment factors, gross and net savings, and in this link and here or in this article here).
For the Specific Guidance at hand the normalization factors can concern performance gap. See table 1 in this link.
Energy consumption can be corrected for differences in production volume and composition for the baseline situation and the situation after the saving action.
4.4 Adjustment factors
Adjustment factors define which part of the calculated energy savings can be attributed to a policy measure or meets the definition of savings specified in the evaluation objectives or reporting requirements (see next section on “Calculating Gross and Net energy savings).
In case of bottom-up methods, see this link, table 1.
Adjustment factors can concern the free rider effect, the spill-over/multiplier effect, additionality and non-compliance. In case of top-down methods, adjustment factors concern autonomous savings (or technological progress) and price-induced energy efficiency progress.
In case of another policy focusing on the same saving actions as evaluated with the guide at hand the adjustment factor double counting might be relevant. If the other policy is not covered in the guide at hand, double counting can only be accounted for at a higher level than the individual guides.
For the specific guide of energy indicators in the sector of industry the adjustment factors free rider and spill-over effect are not relevant.
See link to note on “Saving calculation methods and their application in the Epatee Toolbox”, Calculating Gross and Net energy savings.
Gross savings concern the calculated savings from saving actions using a chosen baseline and normalization factors. Net savings concern the savings attributed to policy measures or to a stakeholder (e.g. an energy company with an obligation to realise savings at their customers). Net energy savings can be evaluated either directly (when using a control or comparison group) or from gross savings by applying further adjustment (or gross-to-net) factors.
The savings should be corrected for the double counting effect, i.e. the overlap between the savings due to the government subsidy with voluntary agreement and savings due to other policy measures. The overlap in the calculated savings of both policy measures cannot be processed at the level of a Specific Guidance but must be corrected at the level of savings due to overall policy portfolios. For addressing double counting see here and in this link.
See also section 8 on concrete examples.
5.1 Main data requirements and data sources and collection techniques
In the table below, the main data requirements for the engineering method and for the measurement method are given.
|Type of method||Main data requirements||Possible data sources and collection techniques|
|Energy indicator method||Acces to data, software to use data to calculate energy indicators||National statistics office (in NL: CBS), Eurostat, IEA|
|Access to simulation model||Plant information data|
Data issues when evaluating net energy savings
The data requirements and the choice of methods require a thorough participation of the industry concerned. At the start of the programme, these data requirements should be discussed and agreed. In a lot of cases in industry, the data sources and collection techniques are already available.
For more details about the evaluation of net energy savings, see this topical case study.
For possible other methods with different data demands see section 6 on alternatives for the chosen method.
5.2 Energy savings in final terms or in primary terms
Energy savings can be expressed in final terms or in primary terms. See definitions about primary and final energy here.
The econometric method can calculate savings in final terms. It can also calculate savings in primary terms provided that savings at end-users are calculated for each energy carrier apart, and primary factors are available to convert the savings in final terms to savings in primary terms.
5.3 Energy savings over time
Implemented saving actions in a year lead to savings over a number of consecutive years. E.g. a more efficient boiler can save gas over its lifetime of about 15 years, insulation over up to 60 years and more efficient computers up to 5 years. For savings from behavioral changes due to a media campaign the life time might be not much longer than that of the campaign. Energy savings can be calculated in different metrics in terms of time reference, for example: year-to-year, annual, cumulated annual, cumulative. See the definitions here.
The calculated yearly savings concern the savings of all new saving actions in that year. In this approach only data for the savings in the chosen year are needed.
Adding up the yearly savings over a period, provided that earlier saving actions are still delivering savings, leads to cumulative savings. For the cumulative savings data are needed for the whole period.
Another cumulative approach, to be applied for the Energy Efficiency Directive, is to multiply the (new) savings in a year with the number of years up to a target year and sum this result with that for all other years up to the target year. This cumulative approach stimulates early saving actions, as these count more times to the target than later actions.
Finally, savings from a saving action can be discounted and summed up over the lifetime of the action See link here.
The econometric method can provide yearly savings of new saving actions in that year. It can also provide cumulative savings provided that data are available over a period.
Cumulative savings according to the Energy Efficiency Directive can be provided when the lifetimes of savings actions are known. Finally, the guide can provide discounted cumulative savings when discount factors have been defined for yearly savings over time. In the econometric model, several discounted savings can be included, for example economic discounting when energy savings are a commodity (white certificates), technical discounting when taking into account that the performance of the saving action can decrease over time.
6.1 Alternatives for the chosen method
This guide offers one method to calculate the savings, the energy indicator method as primary method, with the engineering method and measurement as additional methods.
But often other methods can be applied as well, although they will all have pros and cons regarding various aspects dealt with in preceding sections. In the table below the pros and cons per aspect are provided for the guide at hand. The same has been done for the most relevant alternative methods.
The energy indicator method is a recommended method, but in case a less accurate method that is less expensive, the method of deemed savings could be used. This will result in a less accurate estimate of the energy savings. The deemed savings method could make use of the data in literature on energy consumption of current technologies and of BAT (Best Available Technologies). This gives a window of possible deemed energy savings.
|Type of method||Pros||Cons|
|Energy indicator method||Available data in statistics offices||Expensive, requires experts|
|Alternative method Deemed savings||Inaccurate||Relatively cheap|
7.1 Calculating avoided CO2 emissions
Avoided CO2 emissions can be evaluated from the energy savings by applying emission factors. Four key aspects are to be taken into account when choosing the emission factor(s):
- Emission factors vary according to the energy type, so the data about energy savings need to be available per energy type.
- Emission factors for a given type of energy can vary over time (especially for electricity).
- Emission factors can take into account:
- Direct emission factors: that take into account the emissions generated when producing the energy used;
- Lifecycle emission factors: which take into account all the emissions generated from the extraction of the energy resources up to the dismantling of the energy plant.
Due to the differences that the choice of emission factor(s) can induce, it is important to document what emission factor(s) has (have) been used.
The avoided emission of other greenhouse gasses due to energy savings are not taken into account here, as these emissions (and more specifically their reductions) are generally negligible compared to CO2 (apart from policy measures targeting the agriculture sector).
IPCC (Intergovernmental Panel on Climate Change) provides a detailed database of peer-reviewed emission factors.
7.2 Calculating cost-effectiveness
Cost-effectiveness is the ratio between costs to achieve energy savings and the amount of savings and possibly other benefits.
A distinction can be made according to the point of view adopted to assess cost-effectiveness:
- Cost-effectiveness for the end-user or participant
- Cost-effectiveness for society at large
- Cost-effectiveness for the party that takes responsibility for saving targets (government or actor with an Energy Efficiency Obligation)
- For more details about the different perspectives, see for example (Breitschopf et al., 2018).
The calculation of cost-effectiveness for end-users demands, next to the savings, data on investments made, subsidies on investments, interest rates, lifetimes of the saving actions, energy prices (including taxes) per type of end-user and discount factors per type of end-user. The calculation will involve the calculation of additional cash flows over time (typically the years over which the energy saving action will be present) and calculating this cashflows, with the appropriate discount rate to a net present value (NPV).
For cost-effectiveness from a societal viewpoint no account is taken of subsidies and taxes, energy prices concern world market price, and a lower value of the discount factors is valid. The calculation proceeds as above, but with a modified discount factor.
7.3 Calculating other Co-benefits
Possible co-benefits from saving energy concern:
- Extra employment
- Reduction of energy poverty
- Other emission reductions (NOx, SO2, fine particles, etc.)
- Better indoor climate
- Reduced dependency on (insecure) energy import
The following co-benefits are regarded as most relevant and/or feasible to calculate in conjunction to this guide (see table)
The calculation of the co-benefits is explained in the references.
|Type of co-benefits||Why it can be relevant (and for whom)||References where more details can be found|
|Lower CO2 emissions||For climate targets||IEA, IPCC|
A good overview of the evaluation methodology in relation to voluntary agreements in industry can be found in: https://www.sciencedirect.com/science/article/pii/S095965260100035X?via%3Dihub
Here, the evaluation of voluntary agreements in the Netherlands is reviewed. The methodologies described in this reference conclude that 25-50 % of the energy savings obtained in a 10 year period (1989-1998) can be attributed to the policy mix of long term voluntary agreements and supporting (financial) measures. This translates in a rate of energy efficiency improvement of 33 – 100 % in comparison to a situation in which there are no agreements (the BAU scenario).
More detailed information on the implementation of the voluntary EE agreement in industry, using the energy indicator method can be found in the Dutch reference here:
|Examples||Point of interest|
|Dutch voluntary agreement (LTA, Long Term Agreement)||Broad platform of stakeholders, see this link, and also this link|
Relevant case studies:
Terminology in case studies:
- A study of the comparability of energy audit programme evaluations by Andersson et al., Study type: meta-evaluation, Geographical scope: Australia, Germany, Sweden, USA,
- Ex-post impact and process evaluation of the Swedish energy audit policy programme for small and medium-sized enterprises. https://doi.org/10.1016/j.jclepro.2016.06.139