Use of Quantitative Uncertainty Analysis to Support M&V Decisions in Super ESPCs

August 31, 2005

Introduction

Super Energy Savings Performance Contracts (ESPCs) are a flexible tool to make energy efficiency improvements in federal facilities. While they specify general terms and conditions for the contract between the agency and the energy services company (ESCO), the contract leaves broad latitude to customize specifics such as measurement and verification (M&V) requirements.

M&V is a critical element of an ESPC—without it, there is no way to confirm that the projected savings are in fact being realized. Every FEMP Super ESPC is required to have an M&V plan, which describes how the savings will be verified for each energy conservation measure (ECM), and includes details on the parameters that will be measured, how they will be measured, etc. For any given ECM, there are usually several M&V choices, which will vary in terms of measurement uncertainty, cost, and technical feasibility. At one end of the spectrum, the M&V plan may simply state that most of the parameters that affect a savings estimate be stipulated for the length of the contract, with as little as only one parameter of the savings estimate being measured. At the other end of the spectrum, M&V may involve detailed long-term measurements of most parameters. Typically there is a tradeoff between measurement uncertainty and cost, e.g., a savings calculation method that requires spot measurements will typically cost less than one that requires continuous long-term measurements, but will result in greater uncertainty in the expected savings.

FEMP has developed several tools to aid the M&V decision-making process for Super ESPCs. These include the M&V Guidelines: Measurement and Verification for Federal Energy Management Projects, which are based on the International Measurement and Verification Protocol, the Risk/Responsibility Matrix, and the M&V decision support flow chart. These tools mostly provide qualitative guidance and advocate the use of Quantitative Uncertainty Analysis (QUA) to augment the qualitative guidance. ASHRAE Guideline 14: Measurement of Energy and Demand Savings, Annex B provides some information on how to conduct uncertainty analysis for energy savings. As noted in the standard, "[a] proper uncertainty analysis can be very complex and cumbersome especially if the potential user strives to be very meticulous." In practice, QUA is seen as too complicated and cumbersome, and its use in Super ESPCs has been minimal.

FEMP initiated a pilot project to explore the use of Monte-Carlo simulation to assess savings uncertainty and thereby augment the M&V decision-making process. Monte-Carlo simulation is a flexible QUA technique that has been widely used for risk analysis in various domains. The intent is to use QUA selectively in combination with heuristic knowledge, in order to obtain quantitative estimates of the savings uncertainty without the burden of a comprehensive "bottoms-up" QUA.

Monte-Carlo simulation

Although a full description of Monte-Carlo simulation is beyond the scope of this article, a brief description is provided. Consider a lighting retrofit project in which the lighting energy savings is calculated from the following inputs: total wattage before retrofit (kWpre), total wattage after retrofit (kWpost), and operation hours (hrs). Each of these first-order inputs is in turn determined from second-order inputs. For example, total kWpre is determined from the number and wattage of each fixture type. Typically, point estimates of the inputs are used to calculate the savings. In reality, however, there is uncertainty associated with each input.

In Monte-Carlo simulation, the user applies probability distributions to one or more inputs, reflecting the uncertainty of that input (see figure below). For example, the probability distribution of kWpre indicates that the estimate of 1,800 kilowatts varies from 1,710 kilowatts to 1,890 kilowatts, with a triangular distribution. Probability can similarly be applied to other input parameters. The user then runs the simulation, which yields the probability distribution of energy savings, which in essence describes the savings uncertainty.

The left side of this figure shows a triangular probability distribution for total wattage before retrofit (KWpre), one of the input variables, and the right side shows the probability distribution resulting from the Monte Carlo Simulation, for the Energy Savings (forecase variable).
Conceptual illustration of Monte-Carlo simulation for a hypothetical lighting retrofit project, using Crystal Ball.™

QUA with Monte-Carlo simulation can be as simple or complex as the user wants it to be. For example a "bottoms-up" approach would involve applying probability distributions to all first and second order inputs in order to capture the full range of the uncertainty (e.g. wattages, fixture counts, operating hours, measurement precision, etc.). At the other end of the spectrum, probability distributions may be applied to just a few first order inputs in order to capture the uncertainty resulting from those few inputs (e.g., inputs that the ESCO controls and is responsible for). Another aspect is the source of the input probability distributions. These could be derived from empirical data, standard statistical formulae, or may be simply based on engineering expertise. Thus, the analysis is flexible in two ways—with regard to which inputs probability distributions are applied to, and with regard to the source for the probability distributions.

For this project, the Monte-Carlo analysis was done using Crystal Ball™, which is available as an "add-in" for Excel™. Since many savings calculations are done in Excel™, it is relatively easy to do the Monte-Carlo simulation—it essentially involves adding probability distributions to the input cells (e.g., kWpre) and defining the output cell (e.g., $ savings) for which the uncertainty information is desired.

Case study

While the theoretical basis for QUA is well established and widely used in other domains, the primary objective of this pilot project was to assess the practical implications of applying QUA to Super ESPCs. The QUA project team sought Super ESPC projects that: a) were in the initial stages and in which M&V decisions were not yet made; b) had multiple energy-saving measures involving several M&V choices; and c) had a project facilitator, agency and ESCO who were willing to apply QUA to their Super ESPC.

This approach was used to analyze the savings uncertainty in a Super ESPC at a major federal agency, which included lighting, HVAC, and some cost-avoidance ECMs. QUA was done for each of these ECMs, as well as for the project as a whole. For the individual ECMs, the analysis was done at varying levels of granularity, depending on the size of the ECM. For larger ECMs probability distributions were applied to more inputs. Some examples of the findings from QUA include the following:

  • A more measurement-intensive M&V plan for the lighting ECM would have reduced uncertainty by only $6000, which would not cover the increased M&V costs.

  • Uncertainty analysis on a steam trap replacement ECM suggested that the ESCO may have been more conservative than necessary in discounting the savings estimate and may be "leaving money on the table."

  • While conducting the uncertainty analysis for a cost avoidance ECM, it was discovered that a contractual anomaly could potentially reduce cost-avoidance savings significantly and almost double the portfolio savings risk.

Thus, in some cases the QUA simply confirms intuitive or qualitative information, while in other cases, it provides insight that suggests revisiting the M&V plan.

Besides analysis of the uncertainty for individual ECMs, QUA is also useful for assessing the impact of the portfolio effect in reducing overall savings uncertainty. Many agencies are primarily interested in the savings uncertainty for the whole portfolio of measures, rather than the individual measures. In this particular ECM, the simple arithmetic sum of the savings risk for each of the individual measures results in a total of about $60,000. However, this is a misleading metric, because it does not take into account the risk diversification among measures i.e., a shortfall in one measure may be compensated for by greater than expected savings in another measure (assuming the savings are not correlated). QUA can be used to model the effect of risk diversification, and in this particular ESPC, the analysis showed that the risk was actually only about $23,000.

By providing quantitative uncertainty information, QUA can effectively augment the M&V decision-making process as well as the overall ESPC financial analysis.

Conclusion

QUA can be seamlessly integrated into the current FEMP Super ESPC development process, and the incremental effort is relatively small with user-friendly tools that are commercially available. The input data requirements for QUA are flexible and can be based on empirical or theoretical data, as well as engineering judgment. Furthermore, uncertainty information does not have to be applied to all input parameters, which would be a daunting task. Rather, engineering judgment can be used to determine which parameters are most appropriate to apply uncertainty information to, taking into account factors such as relative impact on savings.

A case study on an ESPC at a large federal agency showed that in some cases the QUA simply confirms intuitive or qualitative information, while in other cases, it provides insight that suggests revisiting the M&V plan. Furthermore, the case study showed that M&V requirements should be informed by the portfolio risk diversification.

Additional case studies are required to better understand and document the optimal mix of QUA and heuristic knowledge in ESPC decision-making. The purpose of QUA is not to deterministically derive M&V requirements. Ultimately, such requirements are a business decision, based on risk analysis and a variety of other factors. QUA simply deepens the information base from which to make those business decisions. The data from QUA can also improve the financial analysis of the ESPC, in that it provides uncertainty data instead of just point estimates. Indeed, uncertainty analysis is critical to bridging the gap between technical and financial analysis in ESPCs.

For more information, please contact Satish Kumar, LBNL, 202-646-7953 or Paul Mathew, LBNL, 202-646-7952