Deemed savingsFreight transportGUIDEInformation/EducationPassenger transport

26 – Evaluating information policies on energy efficiency in transport using deemed savings

By October 24, 2018 No Comments

26 – Evaluating information policies on energy efficiency in transport using deemed savings

This guide can be applied to evaluate the savings due to information policies in the transport sector using the deemed savings method. It includes guidance and explanations specific to this combination of types of policy measure, sector and method, as well as links to general guidance and explanations that can also apply to this combination.

1. USE OF THE GUIDE – AUDIENCE, OBJECTIVES AND FOCUS

The primary audience for this guide is energy efficiency programme designers, implementers or supervisors, and evaluators looking for guidance on the evaluation process of energy savings in the scope of this guide.

Although the application of the guide will generally concern the (sub)national level, account will be taken of issues at EU level when relevant (e.g. the specific format of saving figures for the EED).

This guide is not about the preceding step in the evaluation process, the choice of the method. About this previous step in the evaluation process, see the guidance provided here. However, after presenting the capabilities and limitations of the guide at hand, the user will be offered alternatives for the method within this guide (see section 6).

The objective of this guide is to provide:

  • Information on the scope of the guide that enables the user to decide whether this guide is suited to his/her needs, and whether complementary or additional method(s) could be needed or useful (section 2);
  • Guidance about specifying the evaluation objectives and requirements (section 3);
  • Guidance about key methodological choices to calculate energy savings (section 4);
  • Guidance about the inputs (data requirements) and outputs of the method (energy savings metrics) (section 5);
  • Possible alternative methods (with pros and cons) (section 6)
  • Background about evaluation results other than energy savings (section 7);
  • Relevant examples, case studies and/or good practices (section 8);
  • Relevant references for further reading (section 9).

The guide is intended for assessing realised (ex-post) energy savings. However, account is taken of earlier (ex-ante) evaluations of expected savings, if available (see section 4).

The focus of the guide is on impact evaluation, i.e. determining the energy savings, but not on how this has been reached through a step by step process with intermediate results (process evaluation).

Readers looking for the basic and general principles of energy efficiency evaluation may find the following link useful.

2. SCOPE OF THE GUIDE – POLICY, SECTOR and METHOD

2.1 About information policies

Information policies, like for example information campaigns, try to make people aware of the energy saving effects of certain type of products and behaviour. When dealing with energy consumption in transports, three main categories of information policies can be distinguished (as regards the way they can be evaluated):

  1. Information policies targeting the purchase of vehicles: an example is energy or CO2 labelling of cars, as required by the Directive 1999/94/EC. For more details, see for example (Gibson et al., 2016; Haq and Weiss, 2016). And about discussions on the design of the label or on the type of information that can influence purchasing decisions, see for example (Brazil et al., 2019; Brocklehurst, 2015; Codagnone et al., 2016).
  2. Information policies targeting driving behaviours: examples can be the various types of eco-driving programmes (e.g. training, feedback devices). For more details about this type of programme, see for example (Alam and McNabola, 2014; Barkenbus, 2010). And about discussions about the design of eco-driving programmes or devices, see for example (Allison and Stanton, 2019 ; Stanton and Allison, 2019).
  3. Information policies targeting modal shift: examples can be awareness campaigns about the benefits of biking or walking, local information programmes about alternatives to cars, the development of online facilities to make it easy to choose various transport modes according to the journey and time, etc. For practical cases, see for example (Brög et al., 2009; Griffin, 2019; Moser et al., 2016; Shaheen et al., 2013; Shaheen et al., 2013; Wall et al., 2017).

While the two first types of policies (on purchase of vehicles, and on driving behaviours) can be evaluated with deemed savings, the use of deemed savings would be more challenging for the third type (modal shift). The effects on modal shift are indeed often specific to the local context, especially for passengers’ transport (cf. transport options available, importance of congestion, etc.). However deemed savings can be used for the case of modal shift in freight transports, for which standardized indicators can more easily be defined (as done for example for some standardized actions of the French white certificates scheme).

In this specific guidance, the scope is therefore restricted to policies related to purchase of vehicles or driving behavior. The effects of information policies (or soft measures) are notoriously hard to evaluate:

  • The causality between information and changes in behaviours (purchasing behaviours, driving behaviours, choices of transport mode) is more difficult to establish, compared to policies using financial incentives or regulations.
  • Changes in behaviours can be difficult to maintain over time (particularly for driving behaviours or modal shifts).

The causality and lasting effects thus require a particular attention when evaluating this type of policies. For more resources about the design, challenges, monitoring, etc. of policies and programmes aimed at behaviour changes, see for example the outputs from the phase I and phase II of the Task 25 of the IEA-DSM programme.

More information and examples on the different subtypes residing under the main type information policies can be found here and here. More detailed information on the evaluation of information policies can be found here.

2.2 Evaluation for a combination of policy measure types

When information policies are combined with other types of policy measures, different cases can be distinguished:

  1. The information policy comes as a support measure to another policy (e.g. energy labelling as a support to tax incentive for new cars): usually the energy savings will be attributed to the other policy, as it is difficult to disentangle the effects of both policies, and that the causality of the financial incentives is assumed to be stronger.
  2. The information policy and the other policies are complementary, each one targeting a different effect (e.g. a tax incentive on the purchase of new cars, and an eco-driving programme on driving behaviours): in that case, the respective effects of each policy can be separated. However the interaction should be taken into account in the savings calculation (e.g. the tax incentive will make that the baseline consumption for the eco-driving programme is lower due to more efficient new cars).

In general, when there is an overlap between an information policy and another type of policy, a usual practice is to evaluate the savings only from the other type of policy, assuming that the information policy will reinforce its effects and therefore be encompassed in the evaluation of the other policy.

Therefore, this guidance is mostly meant for the situation when there is no overlap. (see also Double counting in the section on Gross to Net savings).

2.3 Evaluation when combined with energy taxes

The calculated savings effect for information policies will overlap with that of the energy tax. The energy tax will indeed increase the costs of transport (and particularly of using an individual car), thereby creating an incentive to consider more energy efficient vehicles, transport modes or behaviours. The guide is not capable of attributing part of the (overall) calculated savings to either the information policy or the energy tax. For dealing with this overlap see section on Gross to Net savings. However, if the energy tax was already in place at the start of the information campaign and its level has not changed, all of the observed effects can be attributed to the information campaign.

Similarly, it can be difficult to separate the effects from the information policy and from changes in fuel prices. Fuel prices are subject to frequent changes. Therefore, unless the effects of the information policy are evaluated for a very short period (e.g. a few weeks), it is important to consider how the effects due to changes in fuel prices are taken into account.

2.4 About the transport sector

Information on (sub)sectors defined in the Toolbox can be found here, chapter 2, p.17

In practice, most policies for the transport sector target road transport, which is responsible for the largest share of energy consumption by far in transport. The objectives of the policies are therefore most likely to be about promoting more energy efficient vehicles, more energy efficient driving behaviours or modal shifts to more energy efficient transport modes.

As explained about information policies above, the scope of this guidance is focused on policies aiming at promoting more efficient vehicles or driving behaviours.

2.5 About deemed savings

The Energy Efficiency Directive (2018(2002), Annex V(1.a)) defines “deemed savings” as “results of previous independently monitored energy improvements in similar installations”. They are also often called ex-ante or stipulated savings.

Deemed savings are the savings that result from an estimate of the effect of a single action taken as a result of a policy. Deemed savings are thus unitary savings. In the case of information policies the number of actions needs to be estimated as well to calculate the total savings of all actions attributed to the policy (see next section about Complementary methods).

For the cases considered in this guidance (about information policies for transport), there can mainly be two approaches to define deemed savings:

  • Using the results from monitoring, measurements or surveys done previously for the same policy (e.g. during a pilot phase), or for a similar policy (in this latter case, the conditions to use results from another policy should be analysed carefully). About evaluations done with monitoring, measurements or surveys, see for example (Garcia et al., 2017 ; Jeffreys et al., 2018; Savkovic et al., 2019; Sullman et al., 2015; Wilbers and Wardenaar, 2007).
  • Using the results from modelling or simulating the expected effects of the policy. See for example (Hiraoka et al., 2009; Dib et al., 2012).
  • Using standard fuel consumption and the corresponding differences between energy classes of vehicles (when evaluating savings from purchase of new cars). See for example the EPATEE case study on purchasing tax in the Netherlands (Van den Oosterkamp and de Wilde, 2018) (even if this case study deals with financial incentives, and not with information policy, the same evaluation approach can be used for information policies targeting the purchase of more efficient vehicles).

When the evaluation of the energy savings use as input data standard fuel consumption ratios from manufacturers, this might increase the uncertainties on the energy savings as there can be significant differences between the standard fuel consumption and the actual fuel consumption, as pointed for example by (Fontaras et al., 2017).

A set of deemed savings can also be defined for the same action type, to take into account variations due to parameters that can easily be monitored. For example, different values for deemed savings according to categories of vehicles. Deemed savings can also be defined as an energy saving ratio (e.g. savings per km travelled). In that case, the monitoring of these parameters should be ensured, in addition to the number of actions. For example the categories of vehicles can be collected by a survey of participants or through market data analysis. The distances travelled can be available from the freight companies or public transportation companies, when they monitor them for other purposes.

As the main objective of using deemed savings is to keep monitoring and evaluation requirements as simple as possible, the number of data to monitor is usually limited to a few parameters that can be documented easily or is needed to be registered anyway.

However, this does not mean that the calculation made to define the deemed savings is simplistic. Depending on the data available from general references and other studies, and on the means and time available, deemed savings can be defined from simple experts’ estimates up to sophisticated modelling or detailed measurements.

Due to the way they are defined, deemed savings do not reflect the energy savings that are achieved for a given situation, but an average result for a typical situation or a large population of similar actions.

Whenever possible, it is recommended to test and compare different approaches to define the deemed savings, to assess the robustness of the energy savings ratios and their sensitivity to key parameters. Sensitivity analysis can for example be used to select the most relevant parameters to differentiate deemed savings.

Likewise, it is recommended to update the deemed savings on a regular basis, for example to take into account new data available (e.g. updated trends on sales of new cars), changes in regulations (especially when used to set the baseline), results from ex-post studies, etc.

Deemed savings can be used for example as part of an Energy Efficiency Obligation scheme, for which obligated parties may implement programmes such as eco-driving training (see for example the standardised actions eligible to the French white certificates scheme). It can also be used to provide a basis for an on-going monitoring of a policy or programme, as the results can be estimated with deemed savings without delay.

Information about the various evaluation methods can be found here , table 1 and 2. This source also covers the combination of the method at hand with other methods, which will be dealt with below.

2.6 Complementary methods to determine total savings

Complementary methods are methods that are required, in addition to the primary selected method, to calculate energy savings.

Deemed savings are meant to calculate unitary savings for one vehicle or another unit of action (e.g. passenger, ton.kilometer). These unitary savings should be multiplied by the number of actions or participants in order to have the calculated total savings. The number of actions or participants can be obtained in various ways. See here, table 2 and 3. The number of actions is sometimes directly available from the monitoring of policy measures, such as in case of a training programme (cf. number of trainees). In the case of information campaigns to change the driving behaviour, surveys can be used to determine the effect of the information on a single driver as well as on the number of people reached by the campaign. This last number can be used for the number of actions taken.

Stock modelling can also be used, when evaluating policies targeted at the purchase of new cars (see the EPATEE case study on purchasing tax in the Netherlands (Van den Oosterkamp and de Wilde, 2018).

2.7 Additional methods to increase reliability of the results

An additional method can be applied on top of the deemed savings approach to improve the reliability of the evaluation results and/or the cost-effectiveness of the evaluation approach.

As deemed savings are based on data available from other sources than the actions implemented and evaluated, their reliability is unknown and depends on the quality of the data used and whether the data and assumptions used to calculate the deemed savings represent the actions and their conditions of implementation well. Measurement of savings (e.g. monitoring fuel consumption with on-board IT devices or registering of fuel invoices) for a sample group of the saving actions can be an additional method to check the deemed savings. The combination can increase the reliability of the savings figures and is more cost-effective than measurement for all vehicles. For possible combinations with an additional method see chapter 6 here.

3. EVALUATION OBJECTIVES and REQUIREMENTS

Meeting evaluation goals and ambition

The table below shows whether this guide can be used to report on general evaluation goals or criteria. See also in this link to EMEEES project.

General types of evaluation goals or criteria Level of ambition Remarks
Calculation of realized energy savings from saving actions Limited to medium Depending on the quality of the deemed savings. In practice, both the individual effect (unitary savings) and number of people reached are often uncertain.

In any case, complementing deemed savings with a method to verify actual energy savings is recommended for this objective (see above Additional methods to increase reliability of the results in section 2)

Deemed savings should not be used to estimate savings for only one or a small number of actions. Deemed savings should be used to estimate savings for a large number of actions or participants.

Calculation of energy savings attributed to the policy measure(s) limited It is recommended for this objective to complement deemed savings with further ex-post analysis (see below Calculating Gross and Net energy savings in section 4)
Cost-effectiveness of saving action (for end-users) limited to medium Deemed savings do not necessarily reflect energy savings as experienced by a given end-user. They represent average values. So they should only be used to estimate cost-effectiveness indicators on average.

 

Cost-effectiveness of policy (government spending)  limited See comment above about energy savings attributed to policy measure(s)
CO2-emission reduction from saving actions Limited to medium Depending on the quality of the deemed savings (see above). And see section 7 about issues related to emission factors.
CO2-emission reduction attributed to the policy measure(s) limited See comment above about energy savings attributed to policy measure(s)

3.1 Reporting expectations

The topics that can be reported about are the estimated savings achieved and the number of people surveyed that say they are applying the recommendations from the information policy.

Deemed savings are mostly used to evaluate energy savings at the policy level. As explained above in section 2 (see About deemed savings), deemed savings are not appropriate to provide the result specific to a given participant or action.

Deemed savings can be defined in a variety of energy savings (or even CO2 savings) metrics. The choice of the metric(s) will mostly depend on the evaluation objectives and the data available to define the deemed savings.

The table below discusses possible options of metrics.

Criteria Common options Remarks
Nature of the objective Energy savings

CO2 savings

Bill savings

Choice mostly depending on the primary objective of the policy. The results can be expressed in several metrics when needed.

(for more details about CO2 savings see section 7)

Duration for which the results are counted Annual (or first-year)

Lifetime cumulated

Cumulative over the obligation period

Choice depending on the objectives of the policy, and on the type of action (cf. purchase of new vehicle, eco-driving). A “lifetime cumulated” unit can be chosen to value multi-year actions (e.g. purchase of new vehicle). Assessing the effects on driving behaviours over time can be challenging (for more details, see Energy savings over time in section 5).
Energy basis (if nature = energy savings) Primary energy savings

Final energy savings

Choice depending on the objectives and scope of the policy. For example, primary energy savings can be chosen if the scope includes actions dealing with energy switching (change in the energy source). Final energy savings can be chosen for example if the focus is on reducing end-users’ transport expenses (e.g. to alleviate energy poverty).

About specific data needs related to assessing final or primary energy savings, see section 5.

Energy unit (if nature = energy savings) PJ / ktoe / TWh / … Choice usually depending on the energy unit commonly used in the country (e.g. for the national energy balance), or on the objectives or scope of the policy.
Evaluation perspective Gross / Additional / Net For more explanations, see Calculating Gross and Net energy savings in section 4.

Deemed savings can be first expressed in the metric chosen according to the main policy objective, and then converted in other metrics for other reporting purposes (e.g. reporting to European Commission for the article 7 of the EED), or for communicating to different audiences (e.g. expense savings might be more explicit when communicating to the general public).

Whatever the energy savings metrics used, it is important to keep the documentation of the deemed savings, as well as the distribution of the total energy savings according to the different types of actions (and other key parameters, when relevant). These details will indeed be needed to convert the results from one metric to the other. Guidance and examples of template to document deemed savings can be found in the standard ISO 50046.

3.2 Time frame for evaluation

The length of the period under evaluation is dependent on the evaluation objectives, the active period of the policy measure and the time needed to present (reliable enough) results or impacts that fit into the decision making process. In some cases, the periodicity of evaluation can be set by law. In the case of information campaigns, the time the campaign has been running has to be long enough to get reliable results about its effect.

One of the main advantages of deemed savings is that it enables to assess energy savings as soon as the data about the numbers (or other units) of actions per action type are available from the monitoring system or complementary method (e.g. survey). This is one of the reasons why deemed savings are often chosen when the objective is to evaluate results on a regular basis (e.g. each year) or to get a quick feedback loop to detect if adaptations are needed.

It should be noted however that the definition of deemed savings can require time, as well as updates. The time and resources needed to develop (or update) deemed savings depend on the evaluation objectives (about the accuracy of the results), the number and diversity of the action types promoted by the information policy, data availability, whether a consultation is included in the process, the level of consensus (or disagreement) about the data and calculation formula, etc.

In practice, a pilot phase might often be needed to define the deemed savings based on changes in monitored fuel consumption, comparing participants and a control group (see section 4). This pilot phase can also be used to test different settings for the information measures.

The update of deemed savings can be done on a regular or case-by-case basis. It is usually made to take into account changes in the context, for example affecting the definition of the baseline. An update can also be needed if the settings of the information policy are changed (e.g. revised training contents for eco-driving).

The use of deemed savings can be complemented by other methods to verify their reliability, and possibly update them (see Additional methods to increase reliability of the results in section 2). These additional methods have different timeframes. It is thus recommended to consider how the combination of evaluation activities should be planned, especially to ensure the feasibility of the corresponding data collection and to optimize the use of resources (time and budget). (See also planning of evaluation in the link here)

3.3 Expertise needed for chosen method

When the deemed savings method is combined with information campaigns, expertise on behaviour and communication is indispensable.

A specific expertise can be needed according to the method / approach used to define the deemed savings. For example, if a pilot phase is used, then expertise in experimental methods (such as Randomised Controlled Trials) would be required.

If the information policy deals with the purchase of efficient vehicles, then expertise about the markets of vehicles and market transformation would be needed.

If the information policy deals with promoting efficient driving behaviours, then expertise about behaviour change and about the impacts of driving behaviours on fuel consumption would be needed.

3.4 Boundaries for the evaluation

As information campaigns are almost always using media channels with a limited reach, it only makes sense to evaluate the effects of the information policy within the geographical region that has been reached by the campaign. This range will often be a country. It can also be more focused, for example in case of eco-driving programmes for bus drivers. In that case, the evaluation will be focused on the bus companies targeted by the programme.

4. KEY METHODOLOGICAL CHOICES FOR CALCULATION OF ENERGY SAVINGS

General principles of calculating realized savings using different methods can be found here and here.

This section deals with key methodological choices to be considered when calculating energy savings: consistency between ex-ante and ex-post evaluation, baseline, normalization and adjustment factors. These choices are important to document when reporting energy savings, to ensure the transparency of the results.

The deemed savings method in combination with information policies is based on combining information about the number of people reached, the fraction of people that have been reached and also change their behaviour (either purchasing behaviour or driving behaviour), and finally the actual change in energy consumption for individual drivers due to the behavioural change.

4.1 Matching method with earlier ex-ante evaluation

From the viewpoint of methodological consistency and data availability using the same method in the ex-ante evaluation and in this guide on ex-post evaluation might be an obvious choice. However, for ex-ante evaluation only a few methods are usually considered, namely deemed savings, engineering estimate and stock modelling.

A different method than the one(s) used for the ex-ante evaluation can be applied for the ex-post evaluation, depending on the evaluation objectives, timeframe and data available for the situation after implementing the actions. For possible combinations of methods applied ex-ante and ex-post, see chapter 7, calculation approaches in this document.

In practice, ex-ante and ex-post evaluations are applied consecutively. The ex-post evaluation builds on an ex-ante evaluation that makes use of data coming from previous ex-post evaluation or studies (e.g. about a pilot phase or previous periods of the same policy measure). These previous ex-post studies could have used another type of method as well.

4.2 Calculation baselines

Energy savings are defined in general as the difference between the actual situation and a reference situation without the saving actions (and without the policy measures that influence these saving actions). In case of saving actions the reference situation can be defined using various calculation baselines. For further background, see here.

The choice of the baseline will depend on the evaluation objectives, the data available and the target of the information policy.

When promoting efficient vehicles, the following baselines can for example be considered:

  • stock average: average specific fuel consumption (e.g. in l/km) of the current stock of vehicles, combined with average distances travelled per year, both defined per category of vehicles. This baseline can be used to assess gross energy savings (from the point of view of the end-users).
  • market average (or business-as-usual trend in sales of vehicles): average specific fuel consumption (e.g. in l/km) of new vehicles sold on the market (or estimated in business-as-usual scenarios), combined with average distances travelled per year, both defined per category of vehicles. This baseline can be used to assess net energy savings (from the point of view of the policy makers).
  • minimum performance standards: maximum specific fuel consumption (e.g. in l/km) set in regulations for new vehicles, combined with average distances travelled per year, both defined per category of vehicles. This baseline can be used to assess additional energy savings (i.e. additional to the effects of current regulations).

When promoting efficient driving behaviours, the deemed savings can be either defined as an absolute amount of energy savings per driver (assuming average baseline specific fuel consumption, average change in specific fuel consumption due to eco-driving, and average distance travelled per year), or as a relative energy saving effect (i.e. an average % of reduction in fuel consumption). The latter case can for example be applied in case of eco-driving programmes for professional drivers. In that case, the monitoring system can collect data about initial fuel consumption per driver (i.e. before taking part in the eco-driving program).

Similarly, the deemed savings can be defined as relative energy savings ratio in terms of fuel saved per km travelled. Then the monitoring system or a survey can be used to assess the distances travelled by the participants.

When dealing with driving behaviours, deemed savings are commonly defined based on a before/after comparison. Whenever possible, it is recommended to enhance this comparison with a comparison between participants and a control group, using a Difference-in-Differences (DiD method) as part of an experimental or quasi-experimental approach. For more details about this type of approach, see for example (Stewart and Todd, 2017). See also examples of evaluations that used this type of approach for eco-driving programmes: (Garcia et al., 2017; Jeffreys et al., 2018; Savkovic et al., 2019; Sullman et al., 2015).

4.3 Normalization factors

The calculation with before/after baselines provides a change in energy consumption that should be corrected for influences on energy consumption other than the saving actions. These so-called normalization factors can be the rebound effect and changes in energy using activities (e.g. changes in car usage due to other factors than the information policy).

Normalization is normally not needed in case of comparison between participants and a control group, as the factors will be the same for the participants and the Control group. However, this should be checked and possibly the factors should be applied nevertheless. For example, if the time series available for both groups are limited (which could thus mean that the energy savings would be calculated for a particular period, not necessarily representative of typical conditions).

Energy consumption can be corrected for the rebound effect, e.g. longer distances travelled after buying a more efficient vehicle (due to reduction in the cost per kilometre).

Energy consumption can be corrected for differences in transport use (distance, weight) for the baseline situation and the situation after the saving action. This correction should not cover driving more km due to lower driving costs of more efficient cars (see Rebound-effect above).

4.4 Adjustment factors

Adjustment factors define which part of the calculated energy savings can be attributed to a policy measure or meets the definition of savings specified in the evaluation objectives or reporting requirements (see next section on “Calculating gross and net savings).

In case of bottom-up methods (see here, table 1). Adjustment factors can concern the Free rider effect, the Spill-over/multiplier effect, Additionality and Non-compliance. In case of top-down methods, adjustment factors concern autonomous savings (or technological progress) and price-induced energy efficiency progress.

In case of another policy focusing on the same saving actions as evaluated with the PSMC at hand the adjustment factor Double counting might be relevant. If the other policy is not covered in the PSMC at hand double counting can only be accounted for at a higher level than individual PSMCs (see Distinction of energy efficiency improvement measures by type of appropriate evaluation method, see this document and here and here).

For information policy, free-rider effects can be difficult to define, as the participants do not receive a direct financial benefit. Still, free-rider effects can be relevant to consider, as public budget is spent to implement the information activities. Free-rider effects can for example be a household who would have bought the same vehicle in the absence of a communication campaign related to energy labelling. Or a bus company who was already planning to offer eco-driving training to their bus drivers.

Spill-over effects are usually very difficult to capture. An example of study looking at spill-over effects of eco-driving programmes can be found in (Ando and Nishihori, 2011).

See also this link to note “Saving calculation methods and their application in the Epatee Toolbox”.

4.5 Calculating Gross and Net energy savings

Gross savings concern the calculated savings from saving actions using a chosen baseline and normalization factors. Net savings concern the savings attributed to policy measures or to a stakeholder (e.g. an energy company with an obligation to realise savings at their customers). Net savings can be evaluated either directly (when using a control or comparison group) or from gross savings by applying further adjustment (or gross-to-net) factors.

See here and here.

The savings should be corrected for the Double counting effect, i.e. the overlap between the savings due to information policies and savings due to other policy measures. The overlap in the calculated savings of both policy measures cannot be processed at the level of an PSMC but must be corrected at the level of savings due to overall policy portfolios. For addressing double counting see here and here.

See also section on Concrete examples.

5. INPUT AND OUTPUT

5.1 Main data requirements and data sources and collection techniques

Data requirements specified in the table below correspond to the calculation of energy savings, when using the baseline option with/without.

Calculation subject Data requirements Possible data sources and collection technics
Energy savings Information on efficiency of cars bought, on regularity of tire pressure checks and adjustments, on more efficient driving behaviour Self-reported information during surveys
Normalization factors affecting energy consumption: N.A.
Adjustment factors: additionality Information on the effect of other measures: vehicle and road tax depending on car efficiency, other information that affects behaviour Estimated effects of vehicle taxes, self-reported influence of other information on behaviour
Primary energy factors applied (for conversion from final to primary savings): N.A.
Number of actions Number of people reached, fraction of people reached acting on it Surveys
Representativeness of people surveyed

Data issues when evaluating net energy savings

The main good practice to ensure the feasibility and reliability of the evaluation of net energy savings is to think about the method to be used when designing (or revising) the information policy.

Experience indeed shows that unless the data collection has been planned in advance, it will be very costly, time-consuming or even impossible to collect the data required to apply most of the methods that can be used to evaluate net energy savings. Which makes that in practice, using surveys will remain the only option possible (or considered feasible).

The main challenges when using surveys are:

  • to achieve a high answer rate, in order to limit sampling bias;
  • to use question phrasing that can limit the risk of bias in the answers.

For more details about the evaluation of net energy savings, see the topical case study.

For possible other methods with different data demands, see the section on alternatives for the chosen method.

5.2 Energy savings in final terms or in primary terms

Energy savings can be expressed in final terms or in primary terms. See definitions about primary and final energy here.

5.3 Energy savings over time

Implemented saving actions in a year lead to savings over a number of consecutive years. A more efficient boiler for example can save gas over its lifetime of about 15 years, insulation over up to 60 years and more efficient computers up to 5 years. For savings from behavioural changes due to a media campaign the life time might be not much longer than that of the campaign. Energy savings can be calculated in different metrics in terms of time reference, for example: year-to-year, annual, cumulated annual, cumulative. See the definitions in the EPATEE terminology.

The calculated yearly savings concern the savings of all new saving actions in that year. In this approach only data for the savings in the chosen year are needed.

Adding up the yearly savings over a period, provided that earlier saving actions are still delivering savings, leads to cumulative savings. For the cumulative savings, data are needed for the whole period.

Another cumulative approach, to be applied for the EE directive, is to multiply the (new) savings in a year with the number of years up to a target year and sum this result with that for all other years up to the target year. This cumulative approach stimulates early saving actions, as these count more times to the target than later actions.

Finally, savings from a saving action can be discounted and summed up over the lifetime of the action See here.

Deemed savings can provide yearly savings of new saving actions in that year. It can also provide cumulative savings provided that data are available over a period.

Cumulative savings according to the Energy Efficiency Directive can be provided when the lifetimes of savings actions are known.

6. ALTERNATIVE FOR CHOSEN METHOD

6.1 Alternatives for the chosen method

Deemed savings are often chosen when the objective is to keep the evaluation simple or to get a quick feedback loop (e.g. getting results’ estimates without delay). Other evaluation methods can be more relevant depending on the evaluation objectives, data availability and means available (see table below).

Type of method Pros Cons
Deemed savings ·         Provide visibility to stakeholders

·         No delay in getting results from the monitoring system

·         Low running cost (once the set of deemed savings has been defined)

·         Use limited to action types that can be described in a standardised way (if the information activities change, then the deemed savings need to be revised)

·         Do not reflect the energy savings achieved for a given situation, but an average result for a population of actions

·         Can require significant preliminary efforts (e.g. a pilot phase might be needed to define the deemed savings)

·         Quality depending on the data available to define deemed savings

·         Possible gaps between deemed savings and actual savings (recommended use of additional method to check the reliability of the deemed savings, see section 2)

·         Additional method needed to evaluate ex-post the additionality of the savings

Engineering calculations

(based on modelling on fuel consumption)

·         Can be used to explore more in details the effects of the energy saving actions (e.g. influence of given types of driving behaviours on fuel consumption)

·         Limited delay in getting the results (calculations can be done before the actions are installed)

·         Require sophisticated modelling (to provide reliable results)

·         Possible gaps between engineering estimates and measured savings (the model needs to be well calibrated)

·         Additional method needed to evaluate ex-post the additionality of the savings

Type of method Pros Cons
Direct measurement

(based on monitoring of actual fuel consumption and/or of actual distances travelled)

·         Provide data about actual energy consumption / energy savings (capturing rebound effects, as well as performance gaps)

·         Can be used to evaluate ex-post net savings (depending on the type of comparison group chosen, or whether adjustment factors can be assessed with an additional method)

·         Can only be used for ex-post evaluation (or as basis to define deemed savings)

·         Delays in getting the result (depending on the time needed to collect and analyse the data)

·         Require efforts in data collection (but these data can be collected for other purposes, making it cost-effective to use)

·         Difficulties to get representative samples (cf. sampling bias + data losses along the evaluation process)

·         Difficulties to find relevant control or comparison groups (when assessing net or additional savings)

It should be noted that the development of on-board devices that can monitor actual fuel consumption on real time, and of ICT (e.g. smartphones’ applications) has offered new opportunities for data collection in recent years. This makes that the use of different types of direct measurement are now available, and can be cost-effective to use.

See for example: (Ferreira et al., 2015; Husnjak et al., 2015).

7. ADDITIONAL EVALUATION RESULTS

7.1 Calculating avoided CO2 emissions

Avoided CO2 emissions can be evaluated from the energy savings by applying emission factors. Four key aspects are to be taken into account when choosing the emission factor(s):

  1. Emission factors vary according to the energy type, so the data about energy savings need to be available per energy type.
  2. Emission factors for a given type of energy can vary over time (especially for electricity).
  3. Emission factors can take into account:
    1. Direct emission factors: that take into account the emissions generated when producing the energy used;
    2. Lifecycle emission factors: that take into account all the emissions generated from the extraction of the energy resources up to the dismantling of the energy plant.

Due to the differences that the choice of emission factor(s) can induce, it is important to document what emission factor(s) has(have) been used.

The reduction in CO2-emissions can be calculated from the savings with an emission factor for the fuel in question. For electric vehicles, the reduction in CO2-emissions can be calculated from the savings with an emission factor for electricity that takes into account the different inputs of power production. The actual factor to be applied can vary, depending on saving action(s) and sector, year of implementation, policy considerations, etcetera (see this reference).

The avoided emission of other greenhouse gasses due to energy savings are not taken into account here, as these emissions (and more specifically their reductions) are generally negligible compared to CO2 (apart from policy measures targeting the agriculture sector).

When needed, IPCC (Intergovernmental Panel on Climate Change) provides a detailed database of peer-reviewed emission factors.

7.2 Calculating cost-effectiveness

Cost-effectiveness is the ratio between costs to achieve energy savings and the amount of savings and possibly other benefits.

A distinction can be made according to the point of view adopted to assess cost-effectiveness:

  • Cost-effectiveness for the end-user or participant
  • Cost-effectiveness for society at large
  • Cost-effectiveness for the party that takes responsibility for saving targets (government or actor with an Energy Efficiency Obligation)

For cost-effectiveness from a societal viewpoint no account is taken of subsidies and taxes, energy prices concern world market price, and a lower value of the discount factors is valid. Societal cost-effectiveness can be calculated by comparing the costs of the information campaign and the surveys to the amount of energy saved and CO2-emissions avoided.

7.3 Calculating other co-benefits

Possible co-benefits from saving energy concern:

  • Extra employment
  • Reduction of energy poverty
  • Other emission reductions (NOx, SO2, fine particles, etc.)
  • Better indoor climate
  • Reduced dependency on (insecure) energy import

The following co-benefits are regarded as most relevant and/or feasible to calculate in conjunction to this guide (see table). The calculation of the co-benefits is explained in the references.

Type of co-benefits Why it can be relevant (and for whom) References where more details can be found
Cost reduction For the end user
Less air pollution Less negative health effects (affecting citizens) and related costs (for society) Odyssee-Mure multiple benefits tool, COMBI-project

8. CONCRETE EXAMPLES

  • An example of an information policy in transport is the eco driving campaign in the Netherlands (in Dutch: Het Nieuwe Rijden). It targets not only car drivers but also drivers of commercial vehicles. By promoting the purchasing of more efficient vehicles, providing information on energy efficient driving styles and the right tyre pressures to professional and private drivers, the aim is to reduce CO2 emissions from fuel consumption. The cost effectiveness is an effect indicator as well. To evaluate the results, the number of people reached by the campaign as well as the effect of the campaign on individual driving habits needs to be known. These data are collected yearly by, amongst other things, performing surveys. Because of the uncertainty in both quantities the resulting total energy savings are qualified as deemed savings. The results show that the number of people applying the driving style tips has been increasing from 10% in 2000 to 22% in 2004.
    For more details, see:
    Wilbers, P., & Wardenaar, H. (2007). The Dutch national ecodriving programme Het Nieuwe Rijden: A success story. Proceedings of the ECEEE 2007 Summer Study, 1673-1678.
  • NL eco-driving program
  • EPATEE case study related to transport policies:Van den Oosterkamp, P.F. , de Wilde, H. 2018. Purchase tax reduction for passenger cars (The Netherlands). Case study prepared by ECN, part of TNO for the EPATEE project, funded by the European Union’s Horizon 2020 programme.

    This case study is not related to an information policy. However, the evaluation approach used can be easily adapted to the case of information policies targeting the purchase of new vehicles.