19 – Evaluating subsidies on energy efficiency in freight transport using engineering estimates

The primary audience for this guide concerns energy efficiency programme designers, implementers or supervisors, and evaluators looking for guidance on the evaluation process of energy savings in the scope of this tool.

Although the application of the Specific guide will generally concern the (sub)national level, account will be taken of issues at EU level when relevant (e.g. the specific format of saving figures for the EED).

This guide is not about the preceding step in the evaluation process, the choice of the method. About this previous step in the evaluation process, see the guidance provided here. However, after presenting the capabilities and limitations of the guide at hand, the user will be offered alternatives for the method within this guide (see section 6).

The objective of this guide is to provide:

• Information on the scope of the guide that enables the user to decide whether this tool is suited to his/her needs, and whether complementary or additional method(s) could be needed or useful (section 2);
• Guidance about specifying the evaluation objectives and requirements (section 3);
• Guidance about key methodological choices to calculate energy savings (section 4);
• Guidance about the inputs (data requirements) and outputs of the method (energy savings metrics) (section 5);
• Possible alternative methods (with pros and cons) (section 6)
• Background about evaluation results other than energy savings (section 7);
• Relevant examples, case studies and/or good practices (section 8);
• Relevant references for further reading (section 9).

The guide is intended for assessing realised (ex-post) energy savings. However, account is taken of earlier (ex-ante) evaluations of expected savings, if available (see section 4).

The focus of the guide is on impact evaluation, i.e. determining the energy savings, but not on how this has been reached through a step by step process with intermediate results (process evaluation).

Readers looking for the basic and general principles of energy efficiency evaluation may find the following link useful.

2.1 About subsidies

More information and examples on the different subtypes residing under the main type Grants & Subsidies (GS) can be found here and here. The focus of this specific guide is restricted to subsidies.

Financial incentives are an important instrument for spurring investment in energy efficient technologies and services as they can address various barriers to energy efficiency technology deployment. These financial incentives can make energy efficiency investments more attractive for private and public entities, particularly by lowering inhibitive upfront costs.

Financial incentives may be used to complement other energy efficiency policies such as minimum performance standards and buildings codes, overcoming market barriers for cost-effective technologies.

More detailed information on the evaluation of a government subsidy with a voluntary agreement can be found here.

2.2 Evaluation for a combination of policy measure types

When subsidies are combined with other policy measure types it is assumed that the overall savings are mainly resulting from the policy measure at hand. However, the evaluation concerns the combined savings effect of both policy measures.

The specific guide is not capable of attributing part of the (overall) calculated savings to each of the policy measures (see also Double counting in section 4 on Gross to Net savings).

2.3 Evaluation when combined with energy taxes

The calculated savings effect for subsidies will overlap with that of the energy tax. The tool is not capable of attributing part of the (overall) calculated savings to either the policy measures at hand or the energy tax. For dealing with this overlap see section 4 on Gross to Net savings.

2.4 About freight transport

Information on (sub)sectors defined in the Toolbox can be found here, chapter 2, p.17 .

Freight transport is the part of the overall transport sector that serves the movement of commodities and merchandise goods and cargo, mainly on land (road or rail) but also by water and by air. For these modes different vehicle types (heavy and light duty trucks, inland vessels, etc.) with different drive technologies (diesel engines, electric motors etc.) are relevant.

2.5 About engineering estimates

Information about the various evaluation methods can be found in this link, table 1 and 2. This source also covers the combination of the method at hand with other methods, which will be dealt with below.

Detailed engineering estimates (e.g. through calibrated simulation) include more or less complex modelling of individual units (e.g. an energy balance of an individual building) or a truck type with a certain load, at different speeds on various road types.

2.6 Complementary methods to determine total savings

Complementary methods are methods that are required, in addition to the primary selected method, to calculate energy savings.

The method at hand is meant to calculate unitary savings for one freight transport vehicle. These unitary savings should be multiplied by the number of actions or participants (trucks or drivers targeted by the policy) in order to have the calculated total savings. The number of actions or participants can be obtained in various ways. See this link for more background information, in particular table 2 and 3. The number of actions is sometimes directly available from the monitoring of policy measures, such as awarded subsidies in case of a subsidy scheme.

The unitary savings calculated with the method at hand may vary due to the non-uniform nature of the saving action. In that case the total savings are calculated by summing up the unitary savings for all saving actions (instead of the multiplication of average unitary savings with the number of actions).

2.7 Additional methods to increase reliability of the results

An additional method can be applied on top of engineering estimates to improve the reliability of the evaluation results and/or the cost-effectiveness of the evaluation approach.

Engineering calculations include assumptions (e.g. about behavior) that can create uncertainties in the results. It can therefore be useful to check the engineering estimates with direct measurement or billing analysis as additional methods. The combination can increase the reliability of the savings figures in a cost-effective way.

For possible combinations with an additional method see chapter 6 in this link.

3.1 Meeting evaluation goals and ambition

The table shows whether this guide can be used to report on general evaluation goals or criteria. See also this document.

General types of evaluation goals or criteria	Level of ambition	Remarks
Calculation of realized energy savings from saving actions	Fair	Based on adequate data exact estimates of energy savings can be produced
Calculation of energy savings attributed to the policy measure(s)	Medium	Depending on the quality of estimation of actions
Cost-effectiveness of saving action (for end-users)	Medium	Technologies used are known
Cost-effectiveness of policy (government spending)	Low	The exact costs (per vehicle) are known
CO2-emission reduction from saving actions	Fair	Very limited number of well-defined energy carriers
CO2-emission reduction attributed to the policy measure(s)	Medium	See energy savings

For more information on verification of actual energy savings and attribution/baseline/corrections, see section 4, and for cost-effectiveness and emission reduction see section 7.

3.2 Reporting expectations

The main results will be:

• Gross and Net savings
• Annual or cumulative savings

3.3 Time frame for evaluation

The length of the period under evaluation is dependent on the active period of the policy measure, the need to monitor developments before the implementation of savings actions (in case of methods based on before/after saving actions), and the time needed to present (reliable enough) results or impacts that fit into the decision making process.

In some cases, the periodicity of evaluation can be set by law. For financial measures, such as subsidies, the funding period play a decisive role. These are determined by the budgeting of the funds provided in the government’s financial planning.

The planning of evaluation activities concerns regular monitoring of energy consumption and factors that define consumption, intermediate check of (ex-ante) estimated (unitary) savings through measuring or surveys, intermediate evaluations to improve the policy implementation and the final evaluation and reporting.

3.4 Expertise needed for chosen method

Knowledge on modelling the systems concerning the targeted vehicle type from a technological and behavioral viewpoint are necessary to establish a working engineering estimate.

This section deals with key methodological choices to be considered when calculating energy savings: consistency between ex-ante and ex-post evaluation, baseline, normalization and adjustment factors. These choices are important to document when reporting energy savings, to ensure the transparency of the results.

The method engineering estimates uses models or simulations to calculate expected energy savings from a savings action. Depending on the choice of model, it is capable to either do so for unitary savings, for example for one single vehicle, or on more complex units, larger systems, e.g. fleet of vehicles (or just single types of vehicles). The modelling approach is more sophisticated than the unitary savings approach; however, results can be more precise. The methodological choices laid out in the following sections either can be part of model calibration or employed for later adjustment of results.

General principles of calculating realized savings using different methods can be found here and here.

4.1 Matching method with earlier ex-ante evaluation

From the viewpoint of methodological consistency and data availability using the same method in the ex-ante evaluation and in this specific guide on ex-post evaluation might be an obvious choice.

This is indeed possible as one of the (few) applicable ex-ante evaluation methods concerns engineering estimates.

The ex-post method engineering estimate can be combined with other methods for ex-ante evaluation, such as deemed savings and stock modeling, depending on the evaluation objectives, timeframe and data available for the situation after implementing the actions.

Here a combination is possible with the deemed savings method. See chapter 7, calculation approaches in this (internal EPATEE) document.

If the engineering estimate does not provide an acceptable combination with the earlier applied ex-ante method it might be useful to select another method (see examples of alternatives in section 6).

In practice, ex-ante and ex-post evaluations are applied consecutively. The ex-post evaluation builds on an ex-post evaluation that makes use of data coming from previous ex-post evaluation or studies (e.g. about previous periods of the same policy measure, or about the same types of energy saving actions as the ones promoted by the new policy measure). These previous ex-post studies could have used another type of method as well.

4.2 Calculation baselines

Energy savings are defined in general as the difference between the actual situation and a reference situation without the saving actions (and without the policy measures that influence these saving actions). In case of saving actions the reference situation can be defined using various calculation baselines: Before/after, With/without, Trend, Target/control group and Minimum efficiency standards. For further information, see here.

The applicable baseline regarding engineering estimate is usually the before/after baseline.

The so-called Difference-in-Difference analysis constitutes the combination of Before/after approach and Target/control group approach. For further information, see this link.

Depending on the requirements in section 3 preference should be given to before/after or with/without baseline.

See further information in this document.

4.3 Normalization factors

The calculation with the chosen baseline provides a change in energy consumption that should be corrected for influences on energy consumption other than the saving actions. These so-called normalization factors can be weather (with effect on consumption), the rebound effect and changes in energy using activities, such as production (industry), occupation rate (buildings) or car usage (transport).

Normalization factors can also comprise Performance gap and the Pre-bound effect (see this link to internal document on normalization and adjustment factors, gross and net savings, etc).

For the Specific Guidance at hand, the normalization factors can concern the intensity of vehicle use, e.g. load factor and km driven. See table 1 in this link.

Energy consumption can be corrected for differences in transport use (distance, weight) for the baseline situation and the situation after the saving action. This correction should not cover driving more km due to lower driving costs of more efficient cars (see Rebound-effect below).

See also in this link or here.

4.4 Adjustment factors

Adjustment factors define which part of the calculated energy savings can be attributed to a policy measure or meets the definition of savings specified in the evaluation objectives or reporting requirements (see next section on “Calculating Gross and net savings).

Adjustment factors can concern the Free rider effect, the Spill-over/multiplier effect, Additionality and Non-compliance, see this link.

Additionality and non-compliance are connected to unitary savings, while free riders and spill-over work through the number of actions (see next section on “Calculating Gross and net savings).

Double counting might be relevant in case of another policy focusing on the same saving action as evaluated with the Specific Guidance at hand. Double counting can only be accounted for at a higher level than the individual guides. See Distinction of energy efficiency improvement measures by type of appropriate evaluation method, see here for more information and also in this link and here.

For correction of the gross unitary savings non-compliance can be applied as adjustment factor. See table 1 in this link.

For correction of the gross number of saving actions free-riders are applied as the adjustment factors, see chapter 8 in this link for more information.

For free riders a distinction must be made between saving actions due to the policy measure and actions which would have been taken anyway. The method engineering estimate does not provide directly this information, thus other ways must be found, such as a survey among participants to the policy measure about their motivation, or application of Randomized Controlled Trial (RCT) or Quasi-experimental design (see further in this topical case study).

4.5 Calculating Gross and Net energy savings

Gross savings concern the calculated savings from saving actions using a chosen baseline and normalization factors. Net savings concern the savings attributed to policy measures or to a stakeholder (e.g. an energy company with an obligation to realise savings at their customers).

When calculating the savings a distinction must be made for most tools to the unitary savings and number of actions, see this link.

Net total savings can be determined from gross savings applying the relevant adjustment (see section on adjustment factors).

The gross unitary savings can be calculated using the baseline before/after and correcting for relevant normalization factor intensity of vehicle use (load factor en km driven).

The gross number of actions is determined with the complementary method (see chapter 2), using the number of subsidies awarded to more efficient trucks, drivers or transport companies.

Total gross savings are equal to gross unitary savings times gross number of actions.

Net unitary savings can be determined from gross unitary savings applying the relevant adjustment factors, here non-compliance. The Net number of actions is determined from the gross number applying the relevant adjustment factors, in this case free riders.

Net total savings are equal to net unitary savings times net number of actions.

See this link and this link.

Gross savings can be corrected for non-compliance due to inappropriate implementation of saving actions. To this end data should be available of sample-wise checks on the implementation.

The savings should be corrected for the Double counting effect, i.e. the overlap between the savings due to subsidies and savings due to other policy measures. The overlap in the calculated savings of both policy measures cannot be processed at the level of a specific guide but must be corrected at the level of savings due to overall policy portfolios. For addressing double counting see this link and this link.

See section 8 on Concrete examples (Karner et al).

5.1 Main data requirements, sources and collection techniques

Data requirements specified in the table below correspond to the calculation of energy savings, when using the baseline option before/after.

Calculation subject	Data requirements	Possible data sources and collection techniques
Energy consumption / unitary savings	Data on all factors when modeling fuel use: vehicle type, engine efficiency as to load, drag and speed,  technologies and data on differences in efficiency	Vehicle technology data from manufacturers and measurement / surveys,
Normalization factors affecting energy consumption	Data on usage of vehicles	Prior survey or monitoring
Adjustment factors	Compliance as to subsidized efficiency measures	Surveys and sample checks by subsidy program
Primary energy factors applied (for conversion from final to primary savings)	Factors per energy carrier used per vehicle	(prior) technical Measurements
Number of actions	Number of vehicles, drivers or companies targeted by the subsidy scheme	Monitoring of awarded subsidies for various actions

Data issues when evaluating net energy savings

The main good practice to ensure the feasibility and reliability of the evaluation of net energy savings is to think about the method to be used when designing (or revising) the financial measure.

Experience indeed shows that unless the data collection has been planned in advance, it will be very costly, time-consuming or even impossible to collect the data required to apply most of the methods that can be used to evaluate net energy savings. Which results in the fact that in practice, using surveys will remain the only option possible (or considered feasible).

The main challenges when using surveys (e.g. on free riders) are:

• to achieve a high answer rate, in order to limit sampling bias
• to use question phrasing that can limit the risk of bias in the answers

For more details about the evaluation of net energy savings, see the topical case study.

For possible other methods with different data demands see section 6 on alternatives for the chosen method.

5.2 Energy savings in final terms or in primary terms

Energy savings can be expressed in final terms or in primary terms. See definitions about primary and final energy here.

This specific guide with the method Engineering estimates can calculate savings in final terms. It can also calculate savings in primary terms provided that savings at end-users are calculated for each energy carrier separately, and factors are available to convert the savings in final terms to savings in primary terms.

5.3 Energy savings over time

Implemented saving actions in a year lead to savings over a number of consecutive years. E.g. a more efficient boiler can save gas over its lifetime of about 15 years, insulation over up to 60 years and more efficient computers up to 5 years. For savings from behavioral changes due to a media campaign the life time might be not much longer than that of the campaign. Energy savings can be calculated in different metrics in terms of time reference, for example: year-to-year, annual, cumulated annual, cumulative. See the definitions here.

The calculated annual savings concern the savings of all new saving actions in that year. In this approach only data for the savings in the chosen year are needed.

Adding up the annual savings over a period, provided that earlier saving actions are still delivering savings, leads to cumulative savings. For the cumulative savings data are needed for the whole period.

Another cumulative approach, to be applied for the EED, is to multiply the (new) savings in a year with the number of years up to a target year and sum this result with that for all other years up to the target year. This cumulative approach stimulates early saving actions, as these count more to the target than later actions.

Finally, savings from a saving action can be discounted and summed up over the lifetime of the action See link here.

Engineering estimates can provide annual savings of new saving actions in that year. It can also provide cumulative savings provided that data are available over a period.

Cumulative savings according to the Energy Efficiency Directive can be provided when the lifetimes of savings actions are known. Finally, the tool can provide discounted cumulative savings when discount factors have been defined for yearly savings over time.

6.1 Alternatives for the chosen method

Often other savings calculation methods can be applied as well, although they will all have pros and cons regarding various aspects dealt with in preceding sections.

The table below presents the pros and cons of the method for evaluating subsidies in freight transport, and for commonly used alternative methods for the same combination of policy measure and sector.

Type of method	Pros	Cons
Engineering estimates	Also applicable ex-ante, and able to estimate the effects of all kinds of efficiency actions	Much data needed for modeling energy consumption. Potentially inaccurate is not checked through measurement
Direct measurement	No data needed for modeling energy consumption, but measurement of individual vehicles and actions costly and time consuming	Not possible to attribute the observed savings to autonomous or policy savings and to different actions at the same time

7.1 Calculating avoided CO2 emissions

Avoided CO2 emissions can be evaluated from the energy savings by applying emission factors. Four key aspects are to be taken into account when choosing the emission factor(s):

1. Emission factors vary according to the energy type, so the data about energy savings need to be available per energy type.
2. Emission factors for a given type of energy can vary over time (especially for electricity).
3. Emission factors can take into account:
   1. Direct emission factors: that take into account the emissions generated when producing the energy used;
   2. Lifecycle emission factors: which take into account all the emissions generated from the extraction of the energy resources up to the dismantling of the energy plant.

Due to the differences that the choice of emission factor(s) can induce, it is important to document what emission factor(s) has (have) been used.

The reduction in CO2-emissions can only be calculated when savings are calculated per relevant energy carrier and a specific emission factor is available for each energy carrier.

The avoided emission of other greenhouse gases due to energy savings are not taken into account here, as these emissions (and more specifically their reductions) are generally negligible compared to CO2 (apart from policy measures targeting the agriculture sector).

IPCC (Intergovernmental Panel on Climate Change) provides a detailed database of peer-reviewed emission factors.

7.2 Calculating cost-effectiveness

Cost-effectiveness is the ratio between costs to achieve energy savings and the amount of savings and possibly other benefits.

A distinction can be made according to the point of view adopted to assess cost-effectiveness:

• Cost-effectiveness for the end-user or participant
• Cost-effectiveness for society at large
• Cost-effectiveness for the party that takes responsibility for saving targets (government or actor with an Energy Efficiency Obligation)

See further here, report on Evaluation into Practice: lessons learnt from 23 evaluations of energy efficiency policies, section 13.1.

The calculation of cost-effectiveness for end-users demands, next to the savings, data on investments made, subsidies on investments, interest rates, lifetimes of the saving actions, energy prices (including taxes) per type of end-user and discount factors per type of end-user.

For cost-effectiveness from a societal viewpoint no account is taken of subsidies and taxes, energy prices concern world market price, and a lower value of the discount factors is valid.

7.3 Calculating other co-benefits

Possible co-benefits from saving energy can concern:

• Reduction of energy poverty
• Better indoor climate
• Extra employment
• Impact on economic activity
• Reduced dependency on (insecure) energy import

It should be noted that the impacts from incentive schemes on each of these aspects are usually positive, but can also be negative (e.g. on State budget or distributional effects). Therefore, it is in general more appropriate to speak about non-energy impacts.

For a general background about non-energy impacts, see here.

Analysis about how non-energy impacts can have an influence on the political support or rationale for an incentive scheme can be found for example in (Kerr et al., 2017; Rosenow, 2013).

Type of co-benefits	Why it can be relevant (and for whom)	References where more details can be found
Reduction of local emissions (NOx etc.)	Health effects for residents near high traffic roads	https://www.eea.europa.eu/

Karner et al.: Evaluierung der Umweltförderungen des Bundes 2011 – 2013. In German. Available online here:

https://www.umweltfoerderung.at/fileadmin/user_upload/media/publicconsulting/Evaluierung_der_Umweltfoerderungen_des_Bundes.pdf

General guidance on evaluations

• Baumgartner, R. (2017). Chapter 12: Survey Design and Implementation for Estimating Gross Savings Cross-Cutting Protocol. The Uniform Methods Project: Methods for Determining Energy-Efficiency Savings for Specific Measures. Prepared for NREL (National Renewable Energy Laboratory), September 2017. http://www.nrel.gov/docs/fy17osti/68568.pdf
• Hoffman, I., Schiller, S., Todd, A., Billingsley, M., Goldman, C., Schwartz, L., 2015. Energy Savings Lifetimes and Persistence: Practices, Issues and Data. Technical Brief, Lawrence Berkeley National Laboratory, May 2015. https://emp.lbl.gov/publications/energy-savings-lifetimes-and
• Eichhammer et al., 2008. Distinction of energy efficiency improvement measures by type of appropriate evaluation method. Final Report on Evaluation and Monitoring for the EU Directive on Energy End-Use Efficiency and Energy Services. EMEEES Project report. https://www.epatee-lib.eu/media/docs/EMEEES_WP3_Report_Final.pdf

Specific guidance on engineering estimate method

• Evaluation good practice: is ‘good enough’ better than ‘perfect’? Joanne Wade & Nick Eyre, Environmental Change Institute, Oxford, ECEEE Summer study proceedings 2015

Relevant case studies

• Not available