16 – Evaluating government subsidy scheme for voluntary agreement on energy efficiency in industry, using engineering estimate method

The primary audience for this guide is energy efficiency programme designers, implementers or supervisors, and evaluators looking for guidance on the evaluation process of energy savings in the scope of this guide.

Although the application of the guide will generally concern the (sub)national level, account will be taken of issues at EU level when relevant (e.g. the specific format of saving figures for the EED).

This guide is not about the preceding step in the evaluation process, the choice of the method. About this previous step in the evaluation process, see the guidance provided here. However, after presenting the capabilities and limitations of the guide at hand, the user will be offered alternatives for the method within this guide (see section 6).

However, after presenting the capabilities and limitations of the guide at hand, the user will be offered alternatives for the method within this guide (see section 6).

The objective of this guide is to provide:

• Information on the scope of the guide that enables the user to decide whether this tool is suited to his/her needs, and whether complementary or additional method(s) could be needed or useful (section 2);
• Guidance about specifying the evaluation objectives and requirements (section 3);
• Guidance about key methodological choices to calculate energy savings (section 4);
• Guidance about the inputs (data requirements) and outputs of the method (energy savings metrics) (section 5);
• Possible alternative methods (with pros and cons) (section 6)
• Background about evaluation results other than energy savings (section 7);
• Relevant examples, case studies and/or good practices (section 8);
• Relevant references for further reading (section 9).

The tool is intended for assessing realised (ex-post) energy savings. However, account is taken of earlier (ex-ante) evaluations of expected savings, if available (see section 4).

The focus of the guide is on impact evaluation, i.e. determining the energy savings, but not on how this has been reached through a step by step process with intermediate results (process evaluation).

Readers looking for the basic and general principles of energy efficiency evaluation may find the following link useful.

2.1 About the policy measure: government subsidy with a voluntary agreement

More information and examples on the different subtypes residing under the main type “Co-operative Measures”, including government subsidy with a voluntary agreement can be found here and here.

More detailed information on the evaluation of a government subsidy with a voluntary agreement can be found here.

The policy measure of a government subsidy for voluntary agreements related to energy efficiency is typically taken up by a government ministry, for example the Ministry of Economic affairs. Rather than confronting the sector with, for example, a normative standard or an obligation, this measure is based on formulating , with the relevant stakeholders in the industry sector, common goals of energy savings in a certain time period ahead, typically 3-5 year. The voluntary measure is often accompanied by a government subsidy (for example as a tax reduction) as incentive to achieve the energy savings targets. In most cases the Ministry acts as a process facilitator or mandates a public agency or hires an external firm to act as a process facilitator and moderator.

One of the reasons of this voluntary approach is the diverse character of the industry sector. For example, in the Netherlands, on one hand there is a large chemical & petrochemical industry, on the other hand there are a number of much smaller SME’s. The use of the engineering method in this guide is based on the fact that most companies with production sites and installations, will have a dedicated process information system that is also needed to monitor the safe and efficient operation of plants and facilities. Hence, this method is suitable and equipped for assessing energy savings, as the relevant data on energy consumption are part of this process information system. In some cases engineering estimates are used because the voluntary agreement requires the companies to do an energy audit.

The evaluation is usually focused on a program of energy efficiency measures, undertaken by the industry sector on the basis of voluntary EEP’s (Energy Efficiency Plans). The elements of the evaluation include the achieved energy savings in terms of intended savings and realized savings; but also interaction with other policy instruments; the evaluation of side effects (international market playing field (for example the effects of the voluntary agreement on the competitiveness of companies), the execution of the measure; the costs of the measure).

The verification of the achievement of the targets set in the voluntary agreements (at overall level or company level) is often a key evaluation objective. Because advantages or compensations (e.g. investment subsidies, tax deductions) given to the companies that joined the agreements can be conditional upon the target achievement.

Process evaluations of voluntary agreements often look at the administration costs and the satisfaction of the companies that joined the agreements. One reason for policy makers to choose voluntary agreements can indeed be the assumption that they have smaller administration costs or are simpler compared to other policy options. Process evaluations can thus be used to verify this assumption.

2.2 Evaluation for a combination of policy measure types

This guide addresses a combination of policy measure types, being a voluntary agreement with energy users in conjunction with subsidies for saving actions. With two or more policy measures stimulating the same saving actions it turns out that it is difficult to ascribe part of the overall savings to each policy measure type.

A practical approach for combined policy measure types is to perform the evaluation for the set of policy measures as such. The energy savings are thus evaluated overall for the whole policy package. Then it can be decided to attribute the energy savings to a particular policy measure, usually the one seen as most important as to realised savings. The choice of the policy measure to which energy savings will be attributed is most often a political decision. It can also be based on preliminary analysis to identify what policy measure has the larger scope (i.e. encompasses more actions). For example, in case of a combination of a voluntary agreement with subsidies:

• If the subsidies are eligible only to the participants to the voluntary agreement, then it could be decided to attribute the energy savings to the voluntary agreement, because participants might also report actions that they have done without subsidy to reach their goal (larger scope = voluntary agreement).
• If the subsidies are eligible to all companies of a sub-sector or branch, not requiring to commit to the voluntary agreement, then it could be decided to attribute the energy savings to the subsidy scheme. As it is likely that subsidised actions will be done by companies outside the voluntary agreement (larger scope = subsidy scheme).

A drawback of this approach is that the effectiveness of each separate policy measure can hardly be assessed.

A possible approach to tackle this issue is to use a theory-based evaluation (see this link). This makes it possible to analyze the mechanisms of each policy measure. Intermediate indicators or surveys can then enable to characterize the respective effects of each policy measure.

Another approach can be to use econometric methods, when the data available make possible to include variables in the econometric model for distinguishing the effects of the different policy measures (see for example Bjørner & Jensen 2002).

2.3 Evaluation when combined with energy taxes

The calculated savings effect for the government subsidy of a voluntary agreement may overlap with that of the energy tax. The guide is not capable of attributing part of the (overall) calculated savings to either the voluntary agreements or the energy tax. Moreover, in some cases the difficulty to disentangle the effects of both policy measures can be increased by the fact that the voluntary agreements include deductions on energy taxes as an incentive to join the agreements.

A possible approach to tackle this issue can be to use econometric methods, when the data available make it possible (see for example Bjørner & Jensen 2002).

About the issue of double counting, see also the section on Gross to Net savings.

2.4 About the industry sector

Information on (sub)sectors defined in the Toolbox can be found here, chapter 2, p.17.

The sector industry that we do cover in this guide is the sector in the economy which is characterized by production and use of material goods. In practice, the scope of the industry sector can vary according to the countries (e.g. due to specificities in the national economy). For example, agriculture and fisheries are sometimes also included in the industry sector when dealing with energy issues, for example due to the way national energy statistics are available. However, when setting up voluntary agreements, their scope usually makes the distinction between industry and agriculture (or fisheries), and even between the main industry sub-sectors in the country.

Although the guide is meant to cover all end-use sectors in industry it will mostly be applied to a selection of sub-sectors within industry. In every country, the industry is divided in different sub-sectors (usually defined by the national statistics office), like food & beverage industry, chemical and pharmaceutical industry, paper industry, rubber and plastic industry, electrotechnical industry. For example, a dedicated voluntary agreement could be made with all companies that produce chemicals and pharmaceuticals , in such a specific case, next to individual companies, also a professional organization (e.g. in the Netherlands KNCV – Royal Netherlands Chemical Society) can be involved as stakeholder in a voluntary agreement. Voluntary agreements can thus include rules specific to each sub-sector, including in terms of rules to calculate the energy savings.

Voluntary agreements are most often used to engage private companies, which can encompass services and agriculture in addition to the industry sector covered by this guide. Likewise, voluntary agreements can also be concluded with the public sector, and particularly local authorities (as the case in Finland for example). The general approach presented in this guide can apply to these other sectors (e.g. baseline options). However most of the practical aspects (e.g. data and expertise required, normalization factors to take into account) are specific to the industry sector.

2.5 Evaluation for cross-sector saving actions

This guide is also applicable to evaluate cross-sector savings, provided that the data needed are available for all relevant sectors.

2.6 About the engineering estimate method

Information about the various evaluation methods can be found here, table 1 and 2. This source also covers the combination of the method at hand with other methods, which will be dealt with below.

The Energy Efficiency Directive (2018(2002), Annex V(1.a)) defines the results of engineering estimates as “scaled savings”, also mentioning that “those estimates are carried out on the basis of nationally established methodologies and benchmarks by qualified or accredited experts that are independent of the obligated, participating or entrusted parties involved”.

In practice, engineering estimates correspond to energy savings calculations or modelling that take into account values specific to the energy saving actions or projects implemented.

In the case of energy saving actions in the industry, simulation models (simulation software tools like Aspen Plus and PROII) will frequently be used, especially when dealing with complex industrial processes. These models can be available at industry partners that participate in the voluntary agreement, or can also be used by an independent party, for example the company or consultant that acts as mediator in the execution (both in terms of assisting the companies to define their action plans, as well as adhering to the elements of the voluntary agreement) of the voluntary agreement. The selected plants or production processes are represented in an engineering simulation model (usually a so called steady state model) that includes all material streams of the respective process with their composition, temperature, pressure and enthalpy values. With this, steam, heat and material balance and required fuels, electricity and other utilities (e.g. compressed air or steam) can be calculated.

The size and complexity of energy saving actions or projects evaluated with engineering methods can vary widely. The requirements can be adapted according to criteria to take into account this diversity. More measurements (or longer measurement periods) can be required for projects with expected energy savings above a given threshold, for example when a model is used to estimate the savings and that this model needs to be calibrated.

When dealing with simpler action types (e.g. about lighting), the engineering method can be a calculation formula with a limited number of parameters (e.g. power of the equipment and duration of use), that can be used without specific software. The implementing body for the voluntary agreements (e.g. national energy agency) may define standardized calculation formula to be used by the companies or energy auditors for common action types. This approach is often chosen when aiming at ensuring consistency in the savings calculations from one company to the other.

Another approach commonly used to ensure consistency is to define guidelines for energy audits or standard templates for energy audit reports. Many voluntary agreements indeed require the companies joining the scheme to perform an energy audit as a basis to define an action plan. The energy savings calculations done in the energy audits are mostly based on engineering methods, sometimes complemented with direct measurements (either on energy consumption, or on specific parameters needed for the savings calculation).

2.7 Complementary methods to determine total savings

Complementary methods are methods that are required, in addition to the primary selected method, to calculate total energy savings.

Engineering methods provide the savings at the level of an energy saving action or project (unitary energy savings). A monitoring system is then often used to aggregate the results from the different actions or projects.

First, each company has to report the different actions/projects they have implemented (or plan to implement). This makes possible to sum these estimates and obtain the total savings at the company level.

Second, the monitoring or implementing body of the voluntary agreement (e.g. national energy agency) will gather the reports from all the companies to sum their results and obtain the total savings at the policy level.

This aggregation of data and results is usually managed by the monitoring system of the voluntary agreement, and often facilitated by an online platform that companies can use to report their data. The evaluation can then be used for complementary analysis. For example, to verify the energy savings reported by the companies, to make surveys to investigate the additionality of the actions/savings, or to perform targeted in-depth analysis. For more details about the links between monitoring and evaluation, including practical examples, see (Maric et al. 2018). The voluntary agreement can be applied either at the level of an action or a project combining different actions, and more rarely at the level of an industrial site. Then there is a need to aggregate the results. First each company will report the different actions/projects they have implemented (or plan to implement), and sum these estimates to obtain the total savings at the company level.

Second, the monitoring/implementing body of the voluntary agreement will gather the reports from all the companies to sum their results and obtain the total savings at the policy level.

When no systematic monitoring of the actions done by the companies is in place, then the evaluation often needs to use surveys of the companies (or samples of companies) to assess the type of actions implemented and related energy savings.

For further information about complementary methods to obtain total savings from unitary savings, see see this link.

2.8 Additional methods to increase reliability of the results

An additional method can be applied on top of the engineering estimate method to improve the reliability and accurateness of the evaluation results and/or the cost-effectiveness of the evaluation approach.

The engineering estimates method is very suitable for calculating the savings for a complex energy using system like a building or specific industrial processes. Engineering calculations include assumptions (e.g. about behaviors) that can create uncertainties in the results. It can therefore be useful to check the engineering estimates with an additional method like measurement (direct measurement or billing analysis). The combination can increase the reliability of the savings figures in a cost-effective way.

Another additional method, energy consumption indicators (or energy efficiency indicators) can be used to cross-check the total energy savings at the level of the voluntary agreements. As both methods (engineering method and energy consumption indicators) use a different methodology (bottom-up vs. top-down approach), the energy savings calculated with each method cannot be compared directly. However, the total savings obtained from aggregating the engineering estimates of all energy saving actions or projects can be compared with the total savings obtained from the analysis of the energy consumption indicators, in order to see if both are in the same order of magnitude. This can provide a kind of plausibility check. For more details about the use of energy consumption indicator to evaluate voluntary agreements, see Specific Guidance 27 .

For a practical example and discussion about the comparison between energy indicator methods and engineering calculations, see the EPATEE case study (Veum 2018), or see (Abeelen et al. 2016).

For possible combinations with an additional method see chapter 6 in this link.

3.1 Meeting evaluation goals and ambition

The table shows whether this guide can be used to report on general evaluation goals or criteria. See also this document.

General types of evaluation goals or criteria	Level of ambition	Remarks
Calculation of realized energy savings from saving actions	Fair	Saving actions should be well documented (e.g. action of better heat integration in a chemical process) Engineering methods provide a theoretical estimate of the energy savings. The accuracy depends on the level of details of the data and the formula or model used. Energy audits done by qualified energy auditors can usually be considered reliable. Actual energy savings can differ from the engineering estimates for example due to differences in the actual operating conditions (vs. the ones assumed in the calculations).
Calculation of energy savings attributed to the policy measure(s)	Fair	Engineering methods usually provide results in terms of gross energy savings, unless the baseline is chosen to calculate additional savings (see section 4). In most cases, a complementary method (e.g. survey or statistical analysis) is needed to assess the share of savings that can be attributed to the voluntary agreements.
Cost-effectiveness of saving action (for end-users)	Fair	Required investment can be compared with savings in final energy use. The rules of the voluntary agreements can require energy audit reports to include the payback time or other financial indicators (e.g. Net Present Value) for each action recommended. (for more details, see section 7)
Cost-effectiveness of policy (government spending)	Limited	See comment above about energy savings that can be attributed to the policy measure (that represent the main source of uncertainty for this indicator). Government spending include the budget for subsidies (or other type of incentives), the administration cost for public authorities (e.g. for setting up the scheme, communication, monitoring and evaluation), and possibly other costs related to technical support provided to the companies. (for more details, see section 7)
CO2-emission reduction from saving actions	Fair	Can be integrated in engineering method, provided that energy savings are calculated or registered per energy type. (for more details, see section 7)
CO2-emission reduction attributed to the policy measure(s)	Limited	See comment above about energy savings that can be attributed to the policy measure (that represent the main source of uncertainty for this indicator). (for more details, see section 7)

For more information on verification of actual energy savings and attribution/baseline/corrections, see section 4, and for cost-effectiveness and emission reduction see section 7.

3.2 Reporting expectations

Several indicators and metrics to assess the results/impacts may be used, also taking in consideration the differences in the various types of possible recipients of the evaluation, etc. These include:

• Gross and Net savings
• Yearly or cumulative savings
• Final energy savings or savings in Primary energy, etc. (for more details, see section 5)

The metrics to be used to calculate the energy savings usually depend on the metrics used to set the target(s) of the voluntary agreements (for consistency).

The reporting expectations should be defined in an early stage when setting up the voluntary agreement, and agreed with the companies (or their representative organizations) targeted by the voluntary agreements. This will result in shared expectations among the stakeholders of the voluntary agreement.

In the case of voluntary agreements, there can be several steps or levels of reporting:

• Reporting of the companies to the implementing body: this provides the main data used to monitor the scheme. The scheme can include rules to ensure the quality and consistency of the data reported. For example, it can require energy audits done by qualified energy auditors, use of standard templates for energy audit reports, use of standardised calculation formula for the most common action types.
• Reporting of the implementing body to the public authorities or wider audience: this provides the main results of the scheme based on the review of the companies’ reports. It is often a public report (available online). Therefore the data are usually presented in an aggregated way (cf. confidentiality of data at company level). The main indicators and analysis included in these reports depend on their priority audience (e.g. Ministry of Economy, Parliament, Court of Auditors, companies that joined the scheme, general public). Different reports (with different levels of details) can be prepared according to the audience targeted.

The reporting is usually done on an annual basis to provide a regular feedback and to make possible adaptations of the scheme when needed. While in-depth evaluations are often done once per period of agreement (see below).

3.3 Time frame for evaluation

Relevant information regarding the time frame of the evaluation can include: the period for which the evaluated policy measure is active, the planning of the different activities necessary for the evaluation (the timely start of monitoring the “before” development, frequency and timing of data gathering, final monitoring after end of policy, etc.) and estimating roughly the time needed to make an evaluation depending on the type of method used.

The length of the period under evaluation is dependent on the active period of the policy measure, the need to monitor developments before the implementation of savings actions (when the formula or model needs to be calibrated), and the time needed to present (reliable enough) results or impacts that fit into the decision making process. In some cases, the periodicity of evaluation can be set by law.

An advantage of using engineering methods is that they can provide savings estimates before the actions are implemented. This is why it is a common practice for companies to prepare their action plans as part of the scheme (often based on energy audits or energy management systems).

Different time frames can be distinguished:

• Time frame for the companies to comply with their commitments to the scheme: the rules of the voluntary agreements usually define when companies have to perform energy audits, prepare action plans and report about the actions they implemented and the results achieved. The use of engineering methods enables to report data along each step of the process without time lag (e.g. no need to wait for energy bills or other measured data).
• Time frame for the implementing body to review the progress of the companies: it depends on the reporting from the companies. Then the review can include different types of analysis and verifications that will influence the time lag between the reporting of the companies and the finalisation of the review by the implementing body. Typically, the objective is often to keep this time lag within a few months.
• Time frame for the overall evaluation of the scheme: it usually depends on the length of the agreements’ periods. When the agreements are set for 3 to 5-year periods, there may be one overall evaluation close to the end of the period, to assess the impacts of the scheme and provide a detailed feedback in view of preparing the next period. When the agreements are set for longer periods, intermediate evaluations can be decided according to the needs (e.g. based on issues identified with the regular monitoring of the scheme).

When an impact evaluation is decided, it will usually include verifications of the energy savings as reported by the companies. These verifications can be based on a review of the documentation sent by the companies, benchmarking or comparing results from similar action types (and other plausibility checks), surveys of companies to get complementary data, on-site inspections of samples of actions reported. Additional methods can also be used to improve the reliability of the savings estimates: direct measurements (at action level), billing analysis (at site level) or energy consumption indicator (at company level or at the level of sub-sectors) (see end of section 2).

It is important to define beforehand what type(s) of verification or analysis should be done, as it specifies the data that will be needed, and thereby the needs in data collection (see also section 5). Early planning of data collection will optimize the monitoring and evaluation costs.

It is thus strongly recommended to think about the evaluation (preparation and planning) at the same time as the voluntary agreement is designed.

3.4 Expertise needed for chosen method

Expertise required for the chosen method of engineering method, are experts in simulation software methods, when the calculations require the use of a software (e.g. for complex industrial processes). This expertise should include experience with the calibration of the model used, as it is an important condition for the reliability of results from modelling. In addition, technical experts with regard to the data acquisition may be required.

More generally, the use of engineering methods to assess energy savings in the industry requires a technical expertise of the sub-sectors for which the methods are applied. This is for example important to know the key variables and normalization factors to take into account. The voluntary agreements can thus include technical guidelines for the savings calculations (or energy audits) that are defined by main sub-sectors or categories of action (e.g. industrial processes, heat and cooling, ventilation, compressed air, electric motors, lighting).

3.5 Boundaries for the evaluation

The boundaries for the evaluation usually depend on the scope of the voluntary agreements in terms of sub-sectors and periods. According to the evaluation objectives, the evaluation can then include a focus on particular sub-sectors, action types or given years.

General principles of calculating realized savings using different methods can be found here and here.

This section deals with key methodological choices to be considered when calculating energy savings: consistency between ex-ante and ex-post evaluation, baseline, normalization and adjustment factors. These choices are important to document when reporting energy savings, to ensure the transparency of the results. About guidance for the documentation of energy savings, see for example the standard ISO 50046.

The engineering estimate method is based on a mathematical representation of the system considered. Frequently, simulation models will be used, especially when dealing with complex industrial processes (see “About the engineering estimate method” in section 2). This can also be directly a calculation formula in simpler cases.

4.1 Matching method with earlier ex-ante evaluation

The method of engineering estimate to assess energy savings may very well match with possible ex-ante estimates for a particular sector or a specific industrial process.

From the viewpoint of methodological consistency and data availability using the same method in the ex-ante evaluation and in this guide on ex-post evaluation might be an obvious choice. However, for ex-ante evaluation only a few methods are usually considered, namely deemed savings, engineering estimate and stock modeling.

A different method than the one(s) used for the ex-ante evaluation can be applied for the ex-post evaluation, depending on the evaluation objectives, timeframe and data available for the situation after implementing the actions. For possible combinations of methods applied ex-ante and ex-post, see chapter 7 in this document.

If the engineering estimate method does not provide an acceptable combination with the earlier applied ex-ante method it might be useful to select another method (see examples of alternatives in section 6).

In practice, ex-ante and ex-post evaluations are applied consecutively. The ex-post evaluation builds on an ex-ante evaluation that can make use of data coming from previous ex-post evaluation or studies (e.g. about previous periods of the same policy measure, or about the same types of energy saving actions as the ones promoted by the new policy measure). These previous ex-post studies could have used another type of method as well.

It can be useful to distinguish two levels of energy savings calculation:

• At action or project level: the energy savings are often estimated ex-ante with engineering methods (e.g. as part of an energy audit), before the companies decide to implement them. The ex-post evaluation will then aim at verifying what actions (from the ones recommended in the energy audits or planned in the companies’ action plans) have eventually been implemented. These verifications can also include the use of additional methods, such as direct measurement or billing analysis (see section 2).
• At the scheme level: the ex-ante evaluation is usually done to provide a basis to set the target(s) for the voluntary agreements. It can for example be based on an impact assessment or an analysis of the energy savings potentials in each sub-sector to be included in the scheme. These ex-ante studies can be based on deemed savings per action type combined with an assessment of the number of actions that could be implemented, taking into account cost-effectiveness thresholds or other constraints (e.g. annual investment rates, availability of qualified professionals to install the actions). The ex-ante studies can also be based on analysis of previous energy efficiency trends in the different sub-sectors, in order to estimate rates of energy efficiency improvement.

When the ex-ante evaluation is based on deemed savings, then it can more easily be matched with an ex-post evaluation based on engineering estimates. If the ex-ante evaluation is based on rates of energy efficiency improvement, then it could be relevant to use an additional method (energy consumption indicator) when comparing the achievements (ex-post) with the target (ex-ante).

4.2 Calculation baselines

Energy savings are defined in general as the difference between the actual situation and a reference situation without the saving actions (and without the policy measures that influence these saving actions). In case of saving actions the reference situation can be defined using various calculation baselines. For further see further here.

Energy savings resulting from of saving actions due to policy measures can be calculated from the following options:

• Before/after: analysis of changes in energy consumption before/after the saving action, where “before” acts as the baseline for “after”. This option can be applied for replacement of existing energy using systems. The results can be corrected for other influencing factors than savings (see Normalization factors) providing Gross energy savings. By applying adjustment factors (e.g. free-rider effect), the “before/after” comparison can take into account what share of the savings can be attributed to the policy measure, the Net energy savings.
• With/without: analysis of changes in energy consumption with/without the saving action, where the “without” situation acts as the baseline for the “with” situation. This option is for example suited for new efficient systems where no “before” situation is available. Results can be corrected for adjustment factors, providing Net savings. The With/without baseline can also be applied more generally for evaluation of policy measures, especially to evaluate energy savings additional to a predefined scenario (e.g., business-as-usual scenario or a scenario including other policy measures). In this case, this predefined scenario will be used to specify the “without” situation.
• Comparing with trend: analysis of the difference between actual development of energy consumption over time and development without the influence of new policy; e.g. consumer behaviour, autonomous technical progress, “business as usual” scenario. The “trend analysis” option is a sub-category of the “with/without” option, where the “without” situation is defined from the trend analysis.
• Target/control group: analysis of differences in energy consumption for the target group (or participants) of the policy measure and a control group not subject to it (or a comparison group of non-participants). The influence of other factors than the policy measure is assumed to be directly taken into account by the comparison. The “target/control group” comparison thus directly gives net energy savings (see below Calculating Gross and Net energy savings). The “target/control group” option is a sub-category of the “with/without” option, where the “without” situation is defined as the changes observed in the control (or comparison) group.
• Minimum efficiency standards: if a standard is in place, e.g. for appliances or cars, this baseline can be chosen so that only the above-standard savings count. The standard acts as a baseline for attributing savings due to other policy measures. Therefore, this baseline can be combined with the other baselines. The “minimum efficiency standards” option is a sub-category of the “with/without” option, where the “without” situation is defined as energy performance level required by the standards.
• A special combination of baseline approaches is the so-called Difference-in-Difference analysis, using the combination of Before/after approach and Target/control group approach.

This guide considers the baseline option Before/after as the basic option when using engineering method to estimate energy savings from voluntary agreements. It is indeed an option commonly used in energy audits when screening the possible actions, and assessing the corresponding energy savings and financial indicators. The voluntary agreements can thus decide to use this option as well for the companies to report their results, to make it easier (avoiding the need to re-calculate the savings with another baseline). An advantage of the “before/after” option is that it makes possible to calibrate the calculation model with data measured or metered for the “before” situation.

However, it is also possible for the scheme to use other baseline options, which is needed anyway for the case of actions about new situations (e.g., new industrial site or production line) for which no “before” situation exists. In doing so, an approach to avoid the re-calculation of savings and ensure consistency is to set in the rules of the voluntary agreements when different options of baselines should be used.

A baseline option different from the “before/after” option can for example be recommended for the following cases:

• Absence of “before” situation, for new site, production line, etc. (as mentioned above): in this case, a usual practice is to choose the option “minimum efficiency standard” when a regulation including minimum energy performance requirements applies to the type of action considered. When the action type is not covered by such regulation, then the market average is often taken as baseline option (“Comparing with trend”). However, if the action type is too specific, there might be no real market to compare with. In this case, the baseline can be defined from the comparison of the different solutions that the company could use, taking as baseline the solution that has the lowest investment cost.
• A regulation set minimum energy performance requirements for the type of action considered: in this case, it can be relevant to compare the energy efficient solution considered with a solution that would only meet the minimum requirements (option “minimum efficiency standard”).
• The voluntary agreements include additionality criteria (i.e. the energy savings reported by the companies need to be additional versus a reference situation or business-as-usual scenario): in this case, the baseline can be defined for example from scenarios used in an ex-ante evaluation or impact assessment of the scheme. In practice, this often means that different baseline options will be used depending on the action type. For example, when it is assumed that an action type is occurring rarely so far (based on current market trends), then this action type can be assessed with a “before/after” baseline. At the opposite, when the replacement of a type of equipment is assumed to be done on a regular basis and is covered by an Ecodesign regulation, then this action type can be assessed with a “minimum efficiency standard” baseline.

When using engineering methods, two main approaches are possible to calculate additional energy savings:

1. Take into account additionality criteria in the definition of the baseline (option discussed above).
2. Apply adjustment factors to gross energy savings (option presented later on, see Adjustment factors). In this case, the baseline option “before/after” is used in the calculations that give first gross energy savings, and then additional or net energy savings when applying the adjustment factors.

See also paragraph below on Calculating Gross and Net savings.

About baseline options, see also this link. And for general guidance, see this document.

4.3 Normalization factors

The calculation with the before/after baseline considered in previous section provides a change in energy consumption that should be corrected for influences on energy consumption other than the saving actions. When dealing with the industry sector, these so-called normalization factors can be weather (with effect on consumption) and changes in energy using activities, such as production (volumes and production mix).

Normalization factors can also comprise Performance gap, for example when the actual energy performance of an equipment differs from the performance stated by its manufacturer

For a more general discussion about normalization factors see this link on document the normalization and adjustment factors, gross and net savings, etc. and in this link or here.

In case the engineering estimates are based on measured or metered data (e.g. metered load or flow data), the calculation methodology should include requirements about the representativeness of the metering or measurement period. If so, no further normalization is needed (nor factor about performance gap, depending on the type of data that is measured or metered). Normalization factor about production can be needed if the after situation is also defined based on measured or metered data (likewise for weather normalization, if the action type is weather-sensitive).

When dealing with energy saving action in industry, energy consumption most often need to be corrected for differences in production volume and composition for the baseline situation and the situation after the saving action. In practice, volumes and types of production can change rapidly and significantly. The rules of the voluntary agreements therefore need to specify what conditions should be taken as basis for the energy savings calculations. This is also needed to provide companies with visibility about the energy savings they are allowed to report and can count for the achievement of their target. One option can for example be to require to calculate the normalized conditions as an average over previous years (e.g. 3 to 5 years).

More details about taking into account production and weather conditions can be found for example in (Kissock & Eger 2008)

A common example of performance gap is about electric motors, when the actual load curve is different from the load curve assumed in the calculations (and that motor efficiency varies with load). More generally, performance gap can be due to defaults in the technologies or inappropriate conditions of use. Enforcing quality requirements can help to minimize the risks of performance gaps. Likewise, if the action is installed in a site where an energy management system is in place, this system should help detect defaults or suboptimal conditions of use. Thereby limiting the risks of performance gaps.

Factors to take into account performance gap can for example be derived from previous studies on samples of actions. It can also be assessed by using additional methods to verify actual energy savings, such as direct measurements or billing analysis (see section 2).

4.4 Adjustment factors

Adjustment factors define which part of the calculated energy savings can be attributed to a policy measure or meets the definition of savings specified in the evaluation objectives or reporting requirements (see next section on “Calculating Gross and Net energy savings).

Adjustment factors are to take into account when evaluating net or additional energy savings, and that the baseline option used for calculating the energy savings was the “before/after” option. Another approach to assess additional energy savings can indeed be to take into account additionality criteria when defining the baseline (see Calculation baseline above).

For general guidance about adjustment factors for bottom-up methods, see here.

Adjustment factors can concern the Free rider effect, the Spill-over/multiplier effect, Additionality and Non-compliance.

In case of another policy focusing on the same saving actions as evaluated for the voluntary agreements, the adjustment factor Double counting might be relevant. (see also Evaluation for a combination of policy measure types in section 2).

For Distinction of energy efficiency improvement measures by type of appropriate evaluation method; see this link and this link and here.

When evaluating savings from voluntary agreements including government subsidies (or other financial incentives), the adjustment factors free rider and spill-over effects can be relevant to take into account.

It should be noted that in most of the available experience (especially in Europe), adjustment factors have been defined to take into account free-rider effect only. Spill-over effects have indeed proven to be more difficult to assess quantitatively. In the case of voluntary agreements, evaluating spill-over effects can be done by looking at the possible impacts of the scheme on companies that have not joined the agreements, or on sub-sectors not directly covered by the agreements.

The rules of the voluntary agreements can define values of free-rider effects to be included in the savings calculations done by the companies, based on previous studies or surveys. Another approach can be that the implementing body complement the gross energy savings as calculated by the companies with a factor for free-rider effects (and possibly spill-over effects) to report the total net or additional energy savings of the scheme.

In this latter case, the factor for free-rider effects has to be evaluated with an additional method (e.g. surveys of the companies, analysis of market trends, statistical analysis comparing data from companies that joined the agreements and companies that did not join the agreements).

For a practical example of evaluation assessing free-rider and spill-over effects, see e.g. the evaluation of the Swedish voluntary agreements (Stenqvist & Nilsson 2012).

More details about methods to assess net energy savings can be found in the dedicated EPATEE topical case study. See also link to note “Saving calculation methods and their application in the EPATEE Toolbox” here.

The savings should be corrected for the Double counting effect, i.e. the overlap between the savings due to the government subsidy with voluntary agreement and savings due to other policy measures. The overlap in the calculated savings of both policy measures cannot be processed at the level of a single policy measure but must be corrected at the level of savings due to overall policy portfolios. For addressing double counting see this link and this link.

4.5 Calculating Gross and Net energy savings

Gross savings concern the calculated savings from saving actions using a chosen baseline and normalization factors. Net savings concern the savings attributed to policy measures, here the voluntary agreements. Net energy savings can be evaluated either directly (when using a control or comparison group) or from gross savings by applying further adjustment (or gross-to-net) factors.

See this link and this link.

The gross savings can be calculated using the baseline option “before/after”. and correcting for the normalization factors about production, and possibly weather conditions and performance gap when relevant.

When the evaluation of energy savings is firstly based on a before/after comparison, the net total savings are calculated from the gross total savings applying adjustment factors (free-rider effect, spill-over effect and double counting, see above). This approach can be used for ex-ante or ex-post evaluations. For ex-ante evaluations, the adjustment factors will be defined from the literature (e.g. results from previous studies on similar policies, or on previous periods of the same policy). Ex-post evaluations will require the use of additional methods (e.g. surveys, market trend analysis).

A second approach to assess net energy savings can be to use statistical methods to compare changes in energy consumption or in rates of implementing energy saving actions between companies who joined the voluntary agreements (participants) and a comparison group (non-participants). This approach can only be used for ex-post evaluations, as it requires data about the implementation period of the voluntary agreements.

In general, it will not be possible to define samples of participants and non-participants on a purely random basis. That is why we speak here of comparison group, and not of control group.

This also means that there can be risk of sampling bias, for example self-selection bias. As the participants can have a different profile compared to the non-participants (e.g. in terms of energy awareness, experience with energy management, importance given to environmental strategy). This can be mitigated by using relevant matching methods. Several options can be used to define the comparison group, for example: non-participants monitored over the same period as the participants, participants monitored in years before they join the voluntary agreements (when it is possible for companies to join at any time during the period of the voluntary agreements). For a practical example of using statistical methods to assess the effects of a policy measure on the adoption of energy saving actions, see e.g. (Schle33ich et al. 2015). This example is about the effects of an energy audit programme, but the approach is also relevant for voluntary agreements.

A third approach is to calculate additional energy savings, by taking into account additionality criteria in the definition of the baseline (see Calculation baseline above). This approach can be used for both, ex-ante or ex-post evaluations. It can make possible to calculate energy savings additional to other policy measures whose effects are included in the baseline. A common example is to choose as baseline the minimum energy performance requirements set in current regulations. This way, there is no double counting of energy savings between the voluntary agreements and these regulations.

Defining baselines taking into account additionality criteria can in particular be chosen when the Member State reports the energy savings of the voluntary agreements for the achievement of its target set for the article 7 of the Energy Efficiency Directive. In this case, the reported energy savings should be complying with the concept of additionality, as defined in the amended Energy Efficiency Directive (EU2018(2002)):

“To determine the savings that can be claimed as additional, Member States shall have regard to how energy use and demand would evolve in the absence of the policy measure in question by taking into account at least the following factors: energy consumption trends, changes in consumer behaviour, technological progress and changes caused by other measures implemented at Union and national level” (Energy Efficiency Directive 2018(2002), Annex V(2) point (a)).

See also section 8 on concrete examples. More particularly about assessing net savings from gross savings and adjustment factors, see the practical example of the evaluation of the Swedish voluntary agreements (Stenqvist & Nilsson 2012).

5.1 Main data requirements and data sources and collection technics

Data requirements specified in the table below correspond to the calculation of energy savings, when using the baseline option [before/after], i.e. a baseline representing the situation before energy saving actions are implemented. Moreover, the table below deals with the case where engineering estimates are the result of calculations that estimate “before” and “after” energy consumption through intermediate parameters.

Calculation subject	Data requirements	Possible data sources and collection technics
Energy consumption “before”	data requirements of the calculation formula /simulation software (usually defined per main type of action) data should be representative of usual operating conditions	Data from the plant information system (e.g. monitoring system of production lines). Data can also be collected or estimated as part of an energy audit or through the implementation of an energy management system
Energy consumption “after”	Justifying the energy efficiency improvement brought by the action (depending on the action type, the calculation model used for the energy consumption “after” can be different from the one for the energy consumption “before”)	Data from the energy audits (ex-ante). Data from an energy management system or from other monitoring systems (ex-post).
Normalization factors affecting energy consumption   (production, weather conditions when relevant)	Defining the normalized operating conditions	Time series from the plant information system Standard assumptions defined in the guidelines of the voluntary agreements (e.g. about normalized weather conditions)
Normalization factors affecting energy consumption (performance gap)	Comparing theoretical and actual energy performance of the energy-using equipment or system	Data from energy audits (or feasibility studies or alike) for the theoretical energy performance. Data from sub-metering, energy management system or other monitoring system, for the actual energy performance.
Adjustment factors [free-rider effect, spill-over effect, double counting]	These factors can include high levels of uncertainties. Sensitivity analysis can then be useful.	Surveys of participants and non-participants companies Market data about the energy saving actions

One common approach to apply normalization for production is to estimate unitary energy consumption (e.g. in kWh per ton of product). However, it should be noted that energy consumption is not necessarily linearly proportional to production. For example, part of the consumption can be fixed, another part depending on the production. The corresponding formula can either be based on the engineering analysis of the process (when the influencing variables and their relations to energy consumption are well established by physics or other scientific laws), or based on statistical analysis of previous periods (e.g. for complex processes).

About performance gap, this issue can be dealt with quality requirements, as an alternative to apply a factor for performance gap in the energy savings calculations.

It should also be noted that a complementary method is needed to aggregate the results, first at company level and then at the level of the voluntary agreements. This is usually done directly by the monitoring system of the voluntary agreements (see section 2).

Depending on the rules of the voluntary agreements and the conditions of the government subsidy, complementary information or data can be required from the participants’ companies about the actions they have implemented. For example, cost data can be reported to calculate payback time (see also in section 7 Calculating cost-effectiveness).

Data issues when evaluating net energy savings

The main good practice to ensure the feasibility and reliability of the evaluation of net energy savings is to think about the method to be used when designing (or revising) the voluntary agreements.

Experience indeed shows that unless the data collection has been planned in advance, it will be very costly, time-consuming or even impossible to collect the data required to apply most of the methods that can be used to evaluate net energy savings. Which makes that in practice, using surveys will remain the only option possible (or considered feasible).

The main challenges when using surveys are:

• to achieve a high answer rate, in order to limit sampling bias;
• to use question phrasing that can limit the risk of bias in the answers.

For more details about the evaluation of net energy savings, see the topical case study.

See also Calculation baseline in section 4, about the alternative approach to define baselines taking into account additionality criteria (when aiming at calculating additional energy savings).

For possible other methods with different data demands, see section 6 on alternative methods.

5.2 Energy savings in final terms or in primary terms

Energy savings can be expressed in final terms or in primary terms. See definitions about primary and final energy here.

The savings in primary energy terms provide savings that represent the reduction in primary energy consumption (before conversion in energy carriers for end-users). Engineering methods usually calculate savings in final terms. It can also be used to calculate savings in primary terms provided that savings at end-users are calculated for each energy carrier apart, and primary factors are available to convert the savings in final terms to savings in primary terms.

For consistency, the metrics should be the same for setting the target(s) of the voluntary agreements and counting the savings reported by the participants’ companies. The energy savings results can then be expressed in other metrics for other purposes (e.g. reporting in the context of the EED article 7), provided that the data needed to convert from one metric to the other is available. It is thus important to identify the needs to express the results in different metrics, so that data used in the energy savings calculations are documented enough to enable future conversions.

5.3 Energy savings over time

Implemented saving actions in a year lead to savings over a number of consecutive years. E.g. a more efficient boiler can save gas over its lifetime of about 15 years and more efficient computers up to 5 years. Energy savings can be calculated in different metrics in terms of time reference, for example: year-to-year, annual, cumulated annual, cumulative. See the definitions here.

The calculated yearly savings concern the savings of all new saving actions in that year. In this approach only data for the savings in the chosen year are needed.

Adding up the yearly savings over a period, provided that earlier saving actions are still delivering savings, leads to cumulative savings. For the cumulative savings data are needed for the whole period.

Another cumulative approach, to be applied for the EE directive, is to multiply the (new) savings in a year with the number of years up to a target year and sum this result with that for all other years up to the target year. This cumulative approach stimulates early saving actions, as these count more times to the target than later actions.

Finally, savings from a saving action can be discounted and summed up over the lifetime of the action See link here.

The engineering estimate method can provide yearly savings of new saving actions in that year. It can also provide cumulative savings provided that data are available over a period.

Cumulative savings according to the Energy Efficiency Directive can be provided when the lifetimes of savings actions are known. The lifetime of energy savings from actions in the industry can differ from the technical lifetime of the actions. For example, due to maintenance practices, business cycles, closing or relocating of industrial sites or production lines. The guidelines of the voluntary agreements may define how savings lifetime should be dealt with in the savings calculation, to ensure consistency in the data reported by the different companies. One approach can for example be to define standard lifetime values per action type or sub-sector.

Finally, the guide can provide discounted cumulative savings when discount factors have been defined for yearly savings over time

6.1 Alternatives for the chosen method

Often other savings calculation methods can be applied as well, although they will all have pros and cons regarding various aspects dealt with in preceding sections.

Engineering estimates are often chosen when the objective is to evaluate energy savings from actions or projects that can be specific to their context and conditions of implementation, while providing stakeholders with visibility for investment decisions or reporting purposes.

Engineering estimates are appropriate to evaluate savings from specific actions or cross-cutting actions whose characteristics can vary significantly from one project to the other. They are then a cost-effective option when case-by-case calculations are needed, or when a feasibility study or energy audit would be done anyway (e.g. to prepare an investment plan). When the engineering estimates are based on standardised calculation formulas, they can also be used to process data in a partial or fully automatized way, thereby minimizing administration costs (especially when the data needed for the formula can be easily collected and reported by the stakeholders).

In case the action types can be standardised (e.g. standardised lighting actions) and the number of actions can be very large, deemed savings can be a more cost-effective evaluation approach (especially if the savings per action or project remain small). See Specific Guidance 28 about the use of deemed savings for the evaluation of energy savings from Energy Efficiency Obligation scheme, that can also be relevant for the case of voluntary agreements.

Direct measurements can also be relevant, especially for large projects in the industry. The rules of the voluntary agreements can for example require higher accuracy for projects representing energy savings above a given threshold. Using direct measurements can be a cost-effective option in these cases, as it can for example help to optimize the design of the energy saving project and provide results reliable enough for deciding large investments. For more details about direct measurements on actions in the industry sector, see for example (Kissock & Eger 2008).

Billing analysis (or other types of econometric analysis) or the use of energy consumption (or energy efficiency) indicators are common alternative methods when the evaluation objective is more specifically to verify the energy savings actually achieved or to assess the cost-effectiveness or efficiency of the scheme. See the corresponding Specific Guidance 17 (Evaluating government subsidy scheme for voluntary agreement on energy efficiency in industry, using econometric estimate method) or Specific Guidance 27 (Specific guide on evaluating outcome of Energy Efficiency Plans (EEP) using voluntary agreements in industry using the energy indicator method). For a discussion about the pros and cons of engineering methods vs. energy efficiency indicators, see for example (Abeelen et al. 2016) or the EPATEE case study on the Dutch voluntary agreements (Veum 2018).

The table below presents the pros and cons of the engineering estimate method and for alternative methods commonly used to evaluate energy savings from voluntary agreements including government subsidies in the industry sector.

Type of method	Pros	Cons
Method at hand: Engineering estimate method	Relatively accurate (take into account the specificities of the actions or projects evaluated) Can provide savings estimates before the action is implemented (no or small time lag for reporting energy savings) Can be based on energy audits or feasibility studies done for other purposes (e.g. defining action plans or decision-making about investments) Can enable to automatize energy savings calculations (through standardised formula for simple cases)	Level of complexity depending on the complexity of the action type (usually, the more complex the action, the more data needed) Possible gaps between engineering estimates and measured savings (see the corresponding topical case study), https://epatee.eu/sites/default/files/files/epatee_topical_case_study_linkage_between_monitoring_and_evaluation.pdf Complementary method needed to aggregate the results of the whole voluntary agreements (see section 2) Additional method needed to evaluate ex-post net or additional savings (see section 4)
Alternative method: Deemed savings (see specific guidance 28)	Relatively simple and quick method (no time lag for reporting energy savings) Low running cost (once a catalogue of deemed savings has been defined) Provide visibility to stakeholders	Higher level of uncertainties (depending on the quality of data used to define the deemed savings, and the extent to which the actions can be standardized), possible gaps between deemed savings and actual savings Use limited to action types that can be described in a standardised way Do not reflect the energy savings achieved for a given situation, but an average result for a population of actions Can require significant preliminary efforts (if many action types to be included in the catalogue) Complementary method needed to aggregate the results of the whole voluntary agreements (see section 2) Additional method needed to evaluate ex-post net or additional savings (see section 4)

 

7.1 Calculating avoided CO2 emissions

Depending on the objectives of the voluntary agreements or on the needs of reporting, scaled savings from engineering estimates can be expressed in CO2 savings (i.e. avoided CO2 emissions).

In practice, scaled savings are first calculated in terms of energy savings. Avoided CO2 emissions can be evaluated from the energy savings by applying emission factors. Four key aspects are to be taken into account when choosing the emission factor(s):

1. Emission factors vary according to the energy type, so the data about energy savings need to be available per energy type.
2. Emission factors for a given type of energy can vary over time (especially for electricity).
3. Emission factors can take into account:
   1. Direct emission factors: that take into account the emissions generated when producing the energy used;
   2. Lifecycle emission factors: that take into account all the emissions generated from the extraction of the energy resources up to the dismantling of the energy plant.

Due to the differences that the choice of emission factor(s) can induce, it is important to document what emission factor(s) has(have) been used.

Emission factors used for the needs of the voluntary agreements can for example be based on official national emission factors used for the national inventory of emissions of greenhouse gases.

The conversion of electricity savings into CO2 savings is a special case. Depending on the national mix for electricity production, several choices are indeed possible, for example:

• Average emission factor: calculated from the total annual emissions from electricity production (possibly taking into account national imports and exports) divided by the annual amount of electricity consumed: this is a simple approach, but that might not reflect the fact that end-uses can have different times of use and thus correspond to different load profiles (while the emission factor for electricity can vary significantly between base load and peak load).
• Emission factors per type of end-use: this requires more sophisticated calculations (e.g. by decomposing the national load curves per type of end-use) that will be meant to use emission factors reflecting the differences in time of use (e.g. daily, seasonally).
• Specific emission factor: in case the industrial site has its own electricity generation capacities (e.g. CHP units). In that case, the definition of the emission factor needs to be explained by the stakeholders reporting the project.

If the engineering estimates cumulate energy savings over time, it can also be needed to define a scenario about the evolution of the national electricity mix over the period of calculation (e.g. taking into account the objectives of shares of electricity produced from renewable energy sources).

The avoided emission of other greenhouse gasses (GHG) due to energy savings are not taken into account here, as these emissions (and more specifically their reductions) are generally negligible compared to CO2.

When dealing with certain sub-sectors, other GHG emissions can be significant. For example, some sub-sectors of the chemical industry can generate significant amount of emissions of N2O (e.g. for production of nitric acid), SF6 (e.g. industry related to electricity networks) or HFC. In some particular cases, it can thus be relevant to analyze whether the energy saving actions also impact GHG emissions not due to the consumption of energy. For example, when actions are done on cooling processes, it could be relevant to assess emissions of HFC (Hydrofluorocarbon), as HFC has a GWP (Global Warming Potential) about 14 000 times higher than CO2 (so even if the emissions are smaller in quantity, their impact can be significant).

IPCC (Intergovernmental Panel on Climate Change) provides a detailed database of peer-reviewed emission factors.

The section 2 of this database deals specifically with emission factors for industrial processes, which can be a source of information to see if gases other than CO2 should be considered. Likewise, the section 3 deals with emission factors for agriculture, in case the voluntary agreement deals with this sector as well.

7.2 Calculating cost-effectiveness

Cost-effectiveness is the ratio between costs to achieve energy savings and the amount of savings and possibly other benefits.

A distinction can be made according to the point of view adopted to assess cost-effectiveness:

• Cost-effectiveness for the participant companies
• Cost-effectiveness for the public authority responsible for the voluntary agreements
• Cost-effectiveness for society at large

See for example (Breitschopf et al., 2018).
The calculation of cost-effectiveness for end-users demands, next to the savings, data on investments made, subsidies on investments, interest rates, lifetimes of the saving actions, energy prices (including taxes) per type of end-user and discount factors per type of end-user. The calculation will involve the calculation of additional cash flows over time (typically the years over which the energy saving action will be present) and calculating this cashflows, with the appropriate discount rate to a Net present value (NPV).

For cost-effectiveness from a societal viewpoint no account is taken of subsidies and taxes, energy prices concern world market price, and a lower value of the discount factors is valid. The calculation proceeds as follows as above, but with a modified discount factor.

In the case of voluntary agreements including government subsidies, the calculation of cost-effectiveness requires to collect other data on top of the ones used to calculate energy savings, as summarized in the table below.

Point of view	Costs	Benefits
Participant companies	·         Part of the investments paid for the energy saving actions implemented ·         Administration and monitoring costs due to the voluntary agreements (e.g. reporting costs)	·         Gross energy savings (reduced energy cost) ·         Government subsidies received ·         Other co-benefits (e.g. increased competitiveness, see below)
Public authorities	·         Administration costs for the scheme ·         Government subsidies ·         Losses in tax revenues (due to additional or net energy savings)	·         Increases in tax revenues (due to additional or net investments made in energy saving actions, and possibly due to increased economic activity)
Society	·         Total investment costs of the energy saving actions ·         Administration costs (for the public authorities and for the participant companies)	·         Additional or net energy savings ·         Reduced GHG emissions ·         Other co-benefits (e.g. increased economic activity, see below)

Data about investment costs can be difficult to collect if the data collection is done afterwards. The rules of the voluntary agreements can require participant companies to report their costs on a regular basis to avoid this difficulty. Such requirements usually need to take into account the confidentiality of data reported by the companies.

The calculation of cost-effectiveness indicators will usually require to consider scenarios of energy prices over given periods (e.g. lifetime of the energy savings). The assumptions about trends in energy prices should be documented (like the discount rates). Whenever possible, it is recommended to make a sensitivity analysis (testing several scenarios of energy prices).

For a more detailed discussion about the methodology to evaluate cost-effectiveness of voluntary agreements in the context of comparison with other policy measures (e.g. energy taxes), see for example (Henriksson & Söderholm 2009) about the evaluation of the Swedish voluntary agreements.

7.3 Calculating other Co-benefits

Possible co-benefits from saving energy in the industry sector concern:

• Extra employment
• Other emission reductions (NOx, SO2, fine particles, etc.)
• Increased competitiveness of companies
• Increased economic activity
• Reduced dependency on (insecure) energy import

The following co-benefits are regarded as most relevant and/or feasible to calculate in conjunction to the case of voluntary agreements in the industry sector (see table below)

Type of co-benefits	Why it can be relevant (and for whom)	References where more details can be found
Other emission reductions	In relation to GHG reduction targets of sector or country	Numerous text books on engineering
Increased competitiveness of companies	This is usually one of the objectives of the voluntary agreements.

For more discussions and examples about co-benefits of energy efficiency in industry, see for example the resources developed by the H2020 project M-Benefits: https://www.mbenefits.eu/

A good overview of the evaluation methodology in relation to voluntary agreements in industry can be found in:

• Rietbergen, M. G., Farla, J. C., & Blok, K. (2002). Do agreements enhance energy efficiency improvement?: Analysing the actual outcome of long-term agreements on industrial energy efficiency improvement in The Netherlands. Journal of Cleaner Production, 10(2), 153-163. https://doi.org/10.1016/S0959-6526(01)00035-X
  This paper presents the evaluation of voluntary agreements in the Netherlands. The methodologies described in this reference conclude that 25-50 % of the energy savings obtained in a 10 year period (1989-1998) can be attributed to the policy mix of long term voluntary agreements and supporting (financial) measures. This translates in a rate of energy efficiency improvement of 33 – 100 % in comparison to a situation in which there are no agreements (the BAU scenario).

A summary of the evaluation practices related to the Dutch voluntary agreements in the industry sector can also be found in the corresponding EPATEE case study:

• Veum, K.C., 2018. Long-Term Agreements on Energy Efficiency for the non-ETS sector (LTA3, the Netherlands). Case study prepared by ECN-TNO for the EPATEE project, funded by the European Union’s Horizon 2020 programme.
• More detailed information on the implementation of the subsidy scheme for voluntary EE agreement in industry, using engineering estimate method can be found in the Dutch reference here.

A second practical example of the use of engineering methods to estimate energy savings from voluntary agreements in industry is the case of Finland, summarized in the following EPATEE case study:

• Gynther, L., & Suomi, U., 2017. Energy Efficiency Agreement for Industries in Finland. Case study prepared by Motiva for the EPATEE project, funded by the European Union’s Horizon 2020 programme.
• A particular point of interest of the Finnish experience is the system develop to monitor the energy savings, and more generally the results of the voluntary agreements. For more details, see:
  Suomi, U., Puhakka, P., & Väisänen, H. (2007). Comprehensive monitoring system – essential tool to show the results of the energy audit and voluntary agreement programmes. Proceedings of the 2007 ECEEE Summer Study 2007. https://www.eceee.org/library/conference_proceedings/eceee_Summer_Studies/2007/Panel_4/4.166/

A third practical example is the case of Sweden, as described in the following paper:

• Stenqvist, C., & Nilsson, L. J. (2012). Energy efficiency in energy-intensive industries—an evaluation of the Swedish voluntary agreement PFE. Energy Efficiency, 5(2), 225-241. http://portal.research.lu.se/portal/files/2026257/2026522.pdf

Examples	Point of interest
Dutch voluntary agreement (LTA, Long Term Agreement)	Broad platform of stakeholders (in Dutch)

General guidance on evaluations

• ISO 50046:2019. General methods for predicting energy savings. March 2019. https://www.iso.org/standard/67790.html
• Kissock, J. K., & Eger, C. (2008). Measuring industrial energy savings. Applied Energy, 85(5), 347-361. http://academic.udayton.edu/kissock/http/publications/measindsav_sae2006_draft_.pdf
• Maric, L., Thenius, G., Gynther, L., and C. Guermont (2018). Linkage between M&V tools (data collection) and evaluation (complementary analysis). Topical case study of the EPATEE project, funded by the European Union’s Horizon 2020 programme. https://www.epatee-toolbox.eu/?page_id=428

About voluntary agreements

• Price, L. (2005). Voluntary agreements for energy efficiency or ghg emissions reduction in industry: An assessment of programs around the world. Proceedings of the 2005 ACEEE Summer Study. https://cloudfront.escholarship.org/dist/prd/content/qt67c4x06h/qt67c4x06h.pdf
• Rezessy, S., & Bertoldi, P. (2011). Voluntary agreements in the field of energy efficiency and emission reduction: Review and analysis of experiences in the European Union. Energy Policy, 39(11), 7121-7129. https://doi.org/10.1016/j.enpol.2011.08.030
• Storey, M., Boyd, G., & Dowd, J. (1999). Voluntary agreements with industry. In Voluntary approaches in environmental policy (pp. 187-207). Springer, Dordrecht. https://www.econstor.eu/bitstream/10419/154790/1/NDL1997-026.pdf
• Suomi, U., Puhakka, P., and Väisänen, H. (2009). New board energy efficiency agreement. Proceedings of the ECEEE 2009 Summer Study 2009. https://www.eceee.org/library/conference_proceedings/eceee_Summer_Studies/2009/Panel_3/3.266/

About the comparison of different methods to assess the energy savings from voluntary agreements (changes in energy efficiency indicators vs. engineering estimates at project level):

• Abeelen, C., Harmsen, R. & Worrell, E. (2016). Counting project savings—an alternative way to monitor the results of a voluntary agreement on industrial energy savings’, Energy Efficiency, 9(3), 755–770. doi: 10.1007/s12053-015-9398-3

Example of using statistical methods to assess effects of a policy measure on the implementation of energy saving actions in SMEs (about energy audits programme, but approach also relevant for voluntary agreements):

• Schleich, J. , Fleiter, T., Hirzel, S., Schlomann, B., Mai, M., & Gruber, E. (2015). Effect of energy audits on the adoption of energy-efficiency measures by small companies. Proceedings of the ECEEE 2015 Summer Study, 1827-1836. https://www.eceee.org/library/conference_proceedings/eceee_Summer_Studies/2015/8-monitoring-and-evaluation-building-confidence-and-enhancing-practices/effect-of-energy-audits-on-the-adoption-of-energy-efficiency-measures-by-small-companies/

Example of approach to distinguish the effects of combinations of policy measures (including voluntary agreements) for energy efficiency in the industry:

• Bjørner, T. B., & Jensen, H. H. (2002). Energy taxes, voluntary agreements and investment subsidies—a micro-panel analysis of the effect on Danish industrial companies’ energy demand. Resource and energy economics, 24(3), 229-249. https://doi.org/10.1016/S0928-7655(01)00049-5

Example of evaluation of cost-effectiveness of a voluntary agreement:

• Henriksson, E., & Söderholm, P. (2009). The cost-effectiveness of voluntary energy efficiency programs. Energy for Sustainable Development, 13(4), 235-243. http://www.diva-portal.org/smash/get/diva2:1005208/FULLTEXT01.pdf