EPATEE topical case study : Evaluating net energy savings
Summary – Key ideas
The aim of energy efficiency policies is to promote a more efficient use of energy through actions that can be technical, organisational or behavioural. These actions are meant to consume less energy, thereby reducing energy costs and most often also GHG emissions. These are reasons, why some energy efficiency actions would also occur without any targeted policy. Hence, one of the frequent issues about energy efficiency policy evaluation is trying to find out what energy savings can be attributed to the policy and what would have occurred anyways. This is where net savings come in. They account only for that part of savings that are due to the policy.
Policymakers need this information to decide upon policy success, especially when looking at the efficiency of the policy. In other words to determine whether the money and other resources spent on the policy are spent well and have a strong impact on savings performance or whether the policy design should be changed to use the money and other resources more effectively. For example, Courts of Auditors frequently consider net savings to scrutinize the use of public money.
The evaluation of net energy savings is not trivial and depends on a series of factors including the type of data collection, access to participants or stakeholders, calculation methodology, baseline, inclusion and exclusion of effects and intended usage of the study results. Experience shows that there is no silver bullet to tackle this issue. Evaluators face the difficulty that results can vary largely depending on the methodology and that evaluation results are not comparable between each other. After all, the interested reader of the evaluation study might face difficulties to interpret the results because of the multitude of methodologies and complexity of data analysis.
On a broad level, net savings can be determined using two ways. The one being to define or assess a baseline that represents the scenario that “would have happened” without the policy. The other one is adjusting the gross savings for a series of effects. Gross savings are those savings that occurred among participants of the policy, independently of whether they would have acted the same or not in the absence of the policy. The most common adjustment effects are free-rider and spill-over effects. Free-riders are those participants who would have invested also without the policy but participated to benefit from a financial incentive or other types of support. Spill-over effects lead to savings in other areas or in the future due to the informative character of the policy, word-of-mouth effects, market transformation effects or other effects beyond the scope of the policy.
This topical case study presents commonly used approaches to calculate net savings. Each of the methods has its strong and weak points, which are discussed in more detail on the following pages. The following table sums up the methods.
| Type of method and related conditions | Pros and Cons |
| Randomised Control Trial (RCT) * Experimental Design * Treatment (exposed) and control group (not exposed to policy) * Random assignment to groups * Measured savings data available |
+ Random assignment reduces bias + Increases reliability and validity + Widely accepted – Expensive in terms of time and money – Needs to be planned – Ethical problems to bar control group from participation |
| Quasi-experimental design * Experimental Design * Treatment (exposed) and comparison group (not exposed to policy) * Matching of treatment and comparison groups by defined properties * Measured savings data available |
+ Reliable method + Limited bias if comparison group well assigned – Assignment of comparison group difficult – No statistical means to determine the adequacy of the comparison group |
| Survey approach * Participants are asked how they would have acted without the policy * Deemed or scaled savings possible * When no access to non-participant group (or not possible to define a control or comparison group) * When budget and time restrictions |
+ Does not require non-participant control group + Flexibility to adjust questions to policy + Relatively low costs – Prone to biases (in questions and answers) – Participants’ inability to know what they would have done – Tendency to rationalise past choices – Responses cannot be validate |
| Deemed or Stipulated Net-to-Gross-Ratios * Literature-based estimation using market averages and secondary data * Good for short studies * Strict budget limitations * Limited data availability |
+ Gives a first rough estimate before exploring further with other methods + Inexpensive – Inexact – Heterogeneity of policies do not allow for default values |
Some lessons learned about evaluation of net effects in practice are summarised below.
Due to issues with limited data availability, budget constraints, privacy laws and policy design and implementation, randomised controlled trials are often not feasible. In many cases, this applies also to quasi-experimental designs. That is because savings data are often based on deemed or scaled savings or there is no access to a non-participant group. The two examples from Germany and Denmark therefore used the survey approach.
Then one of the crucial issues is the exposure to biases in the questions and answers. Questions about the free-rider effect face biases in both directions. The free-rider effect may be overestimated due to hindsight bias and social desirability bias. Hindsight bias means that respondents might tend to rationalise their decision and believe they would have acted the same without the policy after knowing that the action was successful. Social desirability bias means that respondents might tend to give the answers that they consider as socially desirable, e.g., acting environmentally friendly and not only for monetary reasons. In the other directions, participants have an incentive to stress the importance of the policy to be able to obtain financial incentives in the future. It appears probable that the two former effects outweigh the latter and that free-rider effects are overestimated. However, there is no statistical means to verify this argument. While absolute values of net savings are therefore disputed, the results using the survey method can be used to compare the net savings between policies that have been analysed using the same methodology.
Due to the cross-sector nature of the Danish example, differences in net savings could be analysed for businesses as opposed to households stating that the net effects are higher relative to gross effects for businesses than for households. Furthermore, programmes including a combination of informative action and financial incentives showed a better net performance than each of the two intervention types alone. Another issue with the Danish survey method was the small sample size, which made it impossible to form subsamples large enough for statistical significance.
An approach to reduce the issue with biases is to include further question for verification. Implementing discrete choice experiments into the survey can reduce response biases. Furthermore, comparison with less subjective methods like market data analyses for the construction of a baseline can be included.