25 – Specific guide on evaluating energy audit scheme for energy savings in industry using econometric estimate method

The primary audience for this specific guide is energy efficiency programme designers, implementers or supervisors, and evaluators looking for guidance on the evaluation process of energy savings in the scope of this guide.

Although the application of the guide will generally concern the (sub) national level, account will be taken of issues at EU level when relevant (e.g. the specific format of saving figures for the EED).

This guide is not about the preceding step in the evaluation process, the choice of the method. About this previous step in the evaluation process, see the guidance provided here.

However, after presenting the capabilities and limitations of the specific guide at hand, the user will be offered alternatives for the method within this guide (see section 6 of this guide).

The objective of this specific guide is to provide:

• Information on the scope of the guide that enables the user to decide whether this guide is suited to his/her needs, and whether complementary or additional method(s) could be needed or useful (section 2);
• Guidance about specifying the evaluation objectives and requirements (section 3);
• Guidance about key methodological choices to calculate energy savings (section 4);
• Guidance about the inputs (data requirements) and outputs of the method (energy savings metrics) (section 5);
• Possible alternative methods (with pros and cons) (section 6)
• Background about evaluation results other than energy savings (section 7);
• Relevant examples, case studies and/or good practices (section 8);
• Relevant references for further reading (section 9).

The guide is intended for assessing realised (ex-post) energy savings. However, account is taken of earlier (ex-ante) evaluations of expected savings, if available (see section 4).

The focus of the guide is on impact evaluation, i.e. determining the energy savings, but not on how this has been reached through a step by step process with intermediate results (process evaluation).

Readers looking for the basic and general principles of energy efficiency evaluation may find the following link useful.

2.1 About the energy audit measure

Information and examples on the different subtypes residing under the main type of energy audit measure can be found here and here.

The policy measure of an energy audit scheme involves an inspection, survey and analysis of energy flows for energy conservation to reduce the amount of energy input into the system without negatively affecting the output (E.A. Abdelaziz et al, Renewable and Sustainable Energy Reviews 115 (2011) 150-168). The energy audit is the key for decision-making in the area of energy management. Energy audit is a systematic approach in the industrial sector. It helps an organization to analyze its energy use and discover areas where energy use can be reduced and waste can occur, plan and practice feasible energy conservation methods that will enhance their energy efficiency, serve to identify all the energy streams in a facility, quantify energy usage, in an attempt to balance the total energy input with its use.

Energy audit requires a systematic approach-from the formation of a suitable team, to achieving and maintaining energy savings. A typical process is outlined in Fig. 1.

Figure 1 Typical Audit process (taken from E.A. Abdelaziz et al, Renewable and Sustainable Energy Reviews 115 (2011) 150-168.

More detailed information on the evaluation of energy audit schemes can be found in this link.

2.2 Evaluation for a combination of policy measure types

The guide addresses one possible policy measure type, but sometimes a combination of policy measure types is implemented, e.g. an energy audit scheme in conjunction with subsidies for saving actions. With two or more policy measures stimulating the same saving actions it is difficult to ascribe part of the overall savings to each policy measure type.

When an energy audit scheme is combined with other policy measures types (for example a government subsidy), it is assumed that the overall savings are mainly resulting from the policy measure at hand. However, the evaluation concerns the combined savings effect of both policy measures. The guide is not capable of attributing part of the (overall) calculated savings to each of the policy measures (see also Double counting in section 4 on Gross to Net savings).

2.3 Evaluation when combined with energy taxes

The calculated savings effect for energy audit scheme will overlap with that of energy taxes. This specific guide is not capable of attributing part of the (overall) calculated savings to either the policy measures at hand or the energy tax. For dealing with this overlap see the section 4 on Gross to Net savings.

2.4 About the industry sector

Information on (sub) sectors defined in the Toolbox can be found here, chapter 2, p.17

The sector industry that we do cover in this guide is the sector in the economy which is characterized by production and use of material goods, applying a high degree of automation and mechanization.

Although the guide is meant to cover all end-use sectors in industry, it will mostly be applied to a selection of sectors within industry. In every country, the industry is divided in different sectors (usually defined by the national statistics office), like food & beverage industry, chemical and pharmaceutical industry, paper industry, rubber and plastic industry electrotechnical industry. For example, a dedicated energy audit protocol could be made with all companies that produce chemicals and pharmaceuticals , in such a specific case also a professional organization (e.g. in the Netherlands KNCV – Royal Netherlands Chemical Society) can be involved as stakeholder in a voluntary agreement.

Although the guide is meant to cover all end-use sectors it can also be applied to a selection of sectors with each a specific tax.

The guide is also applicable to cross-sector savings (see next section).

2.5 About the econometric method

Information about the various evaluation methods can be found here, table 1 and 2. This source also covers the combination of the method at hand with other methods, which will be dealt with below.

The econometric method is a top-down simulation with econometric models. An econometric model specifies the statistical relationship that is believed to exist between the various economic quantities pertaining to a particular economic phenomenon, in this particular case energy savings. In the case of the econometric method, a detailed econometric model with access to an update database is required and is also a prerequisite for the use of this method, as setting up such a detailed economic model is a significant effort. However, in several countries such a model and database will be available. The model provides a regression analysis of energy use over time.

For example, in the Netherlands, the EXIOMOD model is a model with the world economic relations, making use of the EXIOBASE database, was used for an evaluation in industry.

EXIOMOD (Extended Input Output Model) is an economic model able to measure the environmental impact of economic activities. As a multisector model, it accounts for the economic dependency between sectors. It is also a global and multi-country model with a consistent trade linking between countries at the commodity level. Based on national account data, it can provide compressive scenarios regarding the evolution of key economic variables such as GDP, value-added, turn-over, (intermediary and final) consumption, investment, employment, trade (exports and imports), public spending or taxes. Thanks to its environmental extensions, it makes the link between the economic activities of various agents (sectors, consumers) and the use of a large number of resources (energy, mineral, biomass, land, water) and negative externalities (greenhouse gases, wastes).

An econometric model is in particular helpful when different industrial sectors take part in the energy audit scheme. For a single industrial sector, the engineering method, described in guide 16, may be more appropriate.

The estimates that will be produced with the econometric models have to be checked regularly with measurement (for example using measured plant data) as additional method. This combination can increase the reliability of the savings figures in a cost-effective way.

The calculation of energy savings will, in the case of a mandatory auditing scheme be based on a (ex-ante) agreed auditing scheme by the industrial companies that take part in the auditing scheme. The auditing scheme should also be attached to agreed objectives in terms of energy savings or energy efficiency in a specific target year.

2.6 Complementary methods to determine total savings

Complementary methods are methods that are required, in addition to the primary selected method, to calculate energy savings.

This guide does not require complementary methods.

2.7 Additional methods to increase reliability of the results

An additional method can be applied on top of the econometric method to improve the reliability of the evaluation results and/or the cost-effectiveness of the evaluation approach. The econometric method is very suitable for calculating the savings when a large database on the economic performance of industry sectors is available. However, the estimates with the econometric method have to be checked regularly with measurement as additional method. The combination can increase the reliability of the savings figures in a cost-effective way.

For possible combinations with an additional method see chapter 6 in See this link, table 2 and 3.

3.1 Meeting evaluation goals and ambition

The table shows whether this guide can be used to report on general evaluation goals or criteria. See also this document.

General types of evaluation goals or criteria	Level of ambition	Remarks
Calculation of realized energy savings from saving actions	Fair	Good bookkeeping required
Calculation of energy savings attributed to the policy measure(s)	Depends	When combined with additional measure like subsidies, this may be difficult
Cost-effectiveness of saving action (for end-users)	Fair	To be included in auditing plan
Cost-effectiveness of policy (government spending)	Fair
CO2-emission reduction from saving actions	Fair	Emission reduction can be calculated from savings
CO2-emission reduction attributed to the policy measure(s)	Depends	Check how this depends on what?

For more information on verification of actual energy savings and attribution/baseline/corrections, see section 4, and for cost-effectiveness and emission reduction see section 7.

3.2 Reporting expectations

The reporting of the results and impacts of a program around a mandatory auditing scheme in industry has to be defined in the protocol that will have to be set up. Reporting will typically include the choice of indicators and metrics (e.g. yearly or cumulative savings, savings in final energy or in primary energy terms, etc.), objectives and expected results, any additional methods to be used.

3.3 Time frame for evaluation

The length of the evaluation period is dependent on the active period of the policy measure (without major changes), the need to monitor developments before the implementation of savings actions and the time after terminating the policy while still actions are implemented. The policy measure of a mandatory audit scheme, together with an econometric calculation model is typically set up for a period of at least 3 years, and often for a 10 year period, with intermittent monitoring moments. Therefore, the planning of evaluation activities concerns regular monitoring of energy consumption and factors that define consumption, intermediate check of (ex-ante) estimated (unitary) savings through measuring or surveys, intermediate evaluations to improve the policy implementation and the final evaluation and reporting. The process for implementation and subsequent monitoring of the results of a mandatory auditing scheme is typically executed over a number of years, as implementation in the production sector cannot be done overnight.

The planning of evaluation activities concerns regular monitoring of energy consumption and factors that define consumption, intermediate check of (ex-ante) estimated (unitary) savings through measuring or surveys, intermediate evaluations to improve the policy implementation and the final evaluation and reporting. See also planning of evaluation in the link here.

3.4 Expertise needed for chosen method

For the evaluation of the energy audit scheme using an econometric method, a suitable econometric model is required and expert(s) are needed to operate this model. The model further needs to be fed with the evaluation experts may be dependable on technical experts with regard to the data acquisition. This point has to be addressed in the overall evaluation protocol of this guide.

3.5 Boundaries for the evaluation

The spatial boundary for the industrial installations will typically be the so called battery limits (physical boundaries of the installation or factory) of the plants or installations that are part of mandatory auditing scheme. The econometric model may contain several sectors and subsectors. These boundaries are to be included the auditing plan.

This section deals with key methodological choices to be considered when calculating energy savings: consistency between ex-ante and ex-post evaluation, baseline, normalization and adjustment factors. These choices are important to document when reporting energy savings, to ensure the transparency of the results.

The method of econometric analysis is described in section 2 above and can be tailored to various methodological choices as described in this section.

General principles of calculating realized savings using different methods can be found here and here.

4.1 Matching method with earlier ex-ante evaluation

From the viewpoint of methodological consistency and data availability using the same method in the ex-ante evaluation and in this guide on ex-post evaluation might be an obvious choice. However, for ex-ante evaluation only a few methods are usually considered, namely deemed savings, engineering estimate and stock modeling.

A different method than the one(s) used for the ex-ante evaluation can be applied for the ex-post evaluation, depending on the evaluation objectives, timeframe and data available for the situation after implementing the actions. For possible combinations of methods applied ex-ante and ex-post, see chapter 7 of this guide, calculation approaches in this link.

If the econometric method does not provide an acceptable combination with the earlier applied ex-ante method it might be useful to select another method (see examples of alternatives in section 6).
In practice, ex-ante and ex-post evaluations are applied consecutively. The ex-post evaluation builds on an ex-ante evaluation that makes use of data coming from previous ex-post evaluation or studies (e.g. about previous periods of the same policy measure, or about the same types of energy saving actions as the ones promoted by the new policy measure). These previous ex-post studies could have used another type of method as well.

4.2 Calculation baselines

Energy savings are defined in general as the difference between the actual situation and a reference situation without the saving actions (and without the policy measures that influence these saving actions). In case of saving actions the reference situation can be defined using various calculation baselines. For further background see this link.

The applicable baseline option for this guide concerns “With/without”, where the “without energy audit scheme” is the baseline for the actual development in the “with energy audit scheme” case. The difference between the two is the energy audit target level (input) and the calculated savings (output).

4.3 Normalization factors

The calculation with the econometric model in the regression calculation over time, incudes any influences on energy consumption other than the saving actions. These so-called normalization factors can be weather (with effect on consumption), the rebound effect and changes in energy using activities, such as production (industry), occupation rate (buildings) or car usage (transport).

4.4 Adjustment factors

Adjustment factors define which part of the calculated energy savings can be attributed to a policy measure or meet the definition of savings specified in the evaluation objectives or reporting requirements (see next section on “Calculating Gross and net savings).

For the guide on energy audit using the econometric method, the adjustment factors free rider effect, spill-over effect and additionality are not relevant.

See link to note “Saving calculation methods and their application in the EPATEE Toolbox”.

4.5 Calculating Gross and Net energy savings

Gross energy savings concern the calculated savings from saving actions using a chosen baseline and normalization factors. Net energy savings concern the savings attributed to policy measures or to a stakeholder (e.g. an energy company with an obligation to realise savings at their customers). Net savings can be evaluated either directly (when using a control or comparison group) or from gross savings by applying further adjustment (or gross-to-net) factors.

See here and here and here.

The savings should be corrected for the Double counting effect, i.e. the overlap between the savings due to the energy audit scheme and savings due to other policy measures, like subsidies. The overlap in the calculated savings of both policy measures cannot be processed at the level of a specific guide but must be corrected at the level of savings due to overall policy portfolios. For addressing double counting see here and here.

See also the section 8 on concrete examples in this report.

5.1 Main data requirements and data sources and collection techniques

Data requirements specified in the table below correspond to the calculation of energy savings, when using the baseline option With/Without.

Calculation subject	Data requirements	Possible data sources and collection techniques
Energy consumption (in selected energy audit sectors)	Yearly values per type of energy carrier, corrected for variations in heating degree days	National statistics
Energy price per sector or subsector	Mean prices per type of energy carrier delivered to end-users	EU and national statistics
Relevant activity levels as to energy consumption	Yearly activity level	EU and national statistics or other sources
Other discrete variables (e.g. other policy measures)	Introduction year of policy measure (value 0 > 1)	Policy measure overview

Data issues when evaluating net energy savings

The main good practice to ensure the feasibility and reliability of the evaluation of net energy savings is to think about the method to be used when designing (or revising) the energy audit scheme.

Experience shows that unless the data collection has been planned in advance, it will be very costly, time-consuming or even impossible to collect the data required to apply most of the methods that can be used to evaluate net energy savings. Which makes that in practice, using surveys will remain the only option possible (or considered feasible).

The main challenges when using surveys are:

• to achieve a high answer rate, in order to limit sampling bias
• to use question phrasing that can limit the risk of bias in the answers

For more details about the evaluation of net energy savings, see this topical case study.

For possible other methods with different data demands see the section 6 on alternatives for the chosen method.

5.2 Energy savings in final terms or in primary terms

Energy savings can be expressed in final terms or in primary terms. See definitions about primary and final energy here.

The econometric method can calculate savings in final terms. It can also calculate savings in primary terms provided that savings at end-users are calculated for each energy carrier apart, and primary factors are available to convert the savings in final terms to savings in primary terms.

5.3 Energy savings over time

Implemented saving actions in a year lead to savings over a number of consecutive years. E.g. a more efficient boiler can save gas over its lifetime of about 15 years, insulation over up to 60 years and more efficient computers up to 5 years. For savings from behavioral changes due to a media campaign the life time might be not much longer than that of the campaign. Energy savings can be calculated in different metrics in terms of time reference, for example: year-to-year, annual, cumulated annual, cumulative. See the definitions here.

The calculated yearly savings concern the savings of all new saving actions in that year. In this approach only data for the savings in the chosen year are needed.

Adding up the yearly savings over a period, provided that earlier saving actions are still delivering savings, leads to cumulative savings. For the cumulative savings data are needed for the whole period.

Another cumulative approach, to be applied for the Energy Efficiency Directive, is to multiply the (new) savings in a year with the number of years up to a target year and sum this result with that for all other years up to the target year. This cumulative approach stimulates early saving actions, as these count more times to the target than later actions.

Finally, savings from a saving action can be discounted and summed up over the lifetime of the action See link here.

The econometric method can provide yearly savings of new saving actions in that year. It can also provide cumulative savings provided that data are available over a period.

Cumulative savings according to the Energy Efficiency Directive can be provided when the lifetimes of savings actions are known. Finally, the guide can provide discounted cumulative savings when discount factors have been defined for yearly savings over time. In the econometric model, several discounted savings can be include, for example economic discounting when energy savings are a commodity (white certificates), technical discounting when taking into account that the performance of the saving action can decrease over time.

6.1 Alternatives for the chosen method

Often other savings calculation methods can be applied as well, although they will all have pros and cons regarding various aspects dealt with in preceding sections.

The table below presents the pros and cons of the method for evaluating purposes, and for commonly used alternative methods for the same combination of policy measure and sector.

Type of method	Pros	Cons
Econometric model	Top-down simulation, once the model is available, relatively fast results	Need econometric model and – experts
Alternative method : Measurement Surveys	For example per sector, in specific production plants, relatively accurate Relatively cheap	May be time consuming   May be inaccurate

7.1 Calculating avoided CO2 emissions

Avoided CO2 emissions can be evaluated from the energy savings by applying emission factors. Four key aspects are to be taken into account when choosing the emission factor(s):

1. Emission factors vary according to the energy type, so the data about energy savings need to be available per energy type.
2. Emission factors for a given type of energy can vary over time (especially for electricity).
3. Emission factors can take into account:
   1. Direct emission factors: that take into account the emissions generated when producing the energy used;
   2. Lifecycle emission factors: which take into account all the emissions generated from the extraction of the energy resources up to the dismantling of the energy plant.

Due to the differences that the choice of emission factor(s) can induce, it is important to document what emission factor(s) has (have) been used.

If the savings concern one fuel only (e.g. for gas boilers), the reduction in CO2-emissions can be calculated from the savings with an emission factor for the fuel at stake. This can be included in the econometric model.

If the savings concern electricity only (e.g. appliances), the reduction in CO2-emissions can be calculated from the savings with an emission factor for electricity that takes into account the different inputs of power production. The actual factor to be applied can vary, depending on saving action(s) and sector, year of implementation, policy considerations, etcetera (see this link).

If the savings concern different energy carriers, the reduction in CO2-emissions can only be calculated when savings are calculated per relevant energy carrier and a specific emission factor is available for each energy carrier

The avoided emission of other greenhouse gasses due to energy savings are not taken into account here, as these emissions (and more specifically their reductions) are generally negligible compared to CO2 (apart from policy measures targeting the agriculture sector).

IPCC (Intergovernmental Panel on Climate Change) provides a detailed database of peer-reviewed emission factors.

7.2 Calculating cost-effectiveness

Cost-effectiveness is the ratio between costs to achieve energy savings and the amount of savings and possibly other benefits.

A distinction can be made according to the point of view adopted to assess cost-effectiveness:

• Cost-effectiveness for the end-user or participant
• Cost-effectiveness for society at large
• Cost-effectiveness for the party that takes responsibility for saving targets (government or actor with an Energy Efficiency Obligation)

For more details about the different perspectives, see for example (Breitschopf et al., 2018).

The calculation of cost-effectiveness for end-users demands, next to the savings, data on investments made, subsidies on investments, interest rates, lifetimes of the saving actions, energy prices (including taxes) per type of end-user and discount factors per type of end-user. If required, these data can be included in the econometric model.

7.3 Calculating other Co-benefits

The method of econometric analysis does usually not include data on other possible co-benefits. In case required, these co-benefits may be estimated by a separate analysis, making use of the results of the econometric model,

Possible co-benefits from saving energy concern:

• Employment effects
• Reduction of energy poverty
• Other emission reductions (NOx, SO2, fine particles, etc.)
• Better indoor climate
• Reduced dependency on (insecure) energy import

7.4 Other aspects of importance

An energy audit can also include significant environmental advantages.

For the method of econometric analysis the following references are useful.

The effect of energy audits in Danish industry – Evaluation of a DSM programme by Larsen et al., in: Energy Studies Review Vol. 14. No, 2. 2006 pp 30-41; This paper presents an evaluation of the impact on electricity consumption of the free-of-charge energy audit provided to all Danish enterprises by the 60 electricity network operators. The scheme has existed since the early 1990′ s. Approximately 0.045 EUR-Cent per kWh consumption is being spent annually on this activity. The evaluation is based on an econometric analysis of a large panel dataset. The econometric analysis is complemented with ten case studies that allow us to get a deeper understanding of the causa l processes. Overall, we find that the scheme has had limited impact on the electricity consumption among the enterprises that have been audited.

CAG Consultants, in partnership with Carbon Trust, Databuild and the Imperial College Business School, were commissioned by DECC to undertake research on the CRC energy efficiency scheme, available here and focusing on phase 1 of the scheme which ran from April 2010 to the end of March 2014. The research in this paper also examined a number of wider themes including: how energy use is managed by organizations, what steps they may have taken or considered to improve energy efficiency in recent years and which factors influenced their decision making processes. This final synthesis report brings together findings from three workstreams: a quantitative survey, qualitative research and econometric research, as well as supporting desk research. The detailed findings of the three workstreams are documented in separate appendices.

About Econometric analysis

• Free-riding on tax credits for home insulation in France: An econometric assessment using panel data, Marie-Laure Nauleau, CIRED, France (regression on all factors as to implementing saving actions on dwellings, including CIDD program with tax credits); https://www.sciencedirect.com/science/article/pii/S0140988314001923
• CRC Energy Efficiency Scheme Evaluation – Final synthesis report, CAG Consultants, July 2015

About Energy audits

• The effect of energy audits in Danish industry – Evaluation of a DSM programme, Larsen et al, 2006;

Relevant case studies

• No topical case studies are available on this subject