●Would a comparison group that matches the baseline usage pattern of a treatment group start to exhibit bias over time that would negate some of the savings of the treatment group?
●We know that there might be self-selection issues that might need to be addressed during the selection of the comparison group
●But we don’t know what the variability of the bias might be and we don’t know whether or not it might get worse under conditions of large-scale energy changes, such as during COVID

●Simulate the selection of treatment and comparison groups for a program
●Target a subset of high peak load users, similar to what might done for a program that is focused on peak load reduction
●Select a variety of sizes of “treatment” and “comparison” groups from the larger targeted population that can be compared against each other

 Bias is inevitable in comparison groups due to factors beyond our ability to control for

2) Larger groups will reduce bias, but treatment group may still diverge from comparison group over time

3) Need for more systematic approach to determine optimal comparison and treatment group sizes

Stratification is about binning

●What parameters?
●Equal distance or population?
●How many bins?
●How many parameters?
●In what order of priority?
●Should there be an order?
●How do you gauge success?


Stratified Sampling: What are the Constraints?
The only option in resampling is the elimination of meters.

Two main constraints:

1.The size of the comparison pool
2.The number of meters needed in the final sample

Standardizing Stratification: Dealing with Multiple Parameters

Multiple parameters can generate a more representative sample

●Complexity: Stratifying the first parameter and then moving on to the next will inherently change the first
               ○Solution = Simultaneous 2D or nD binning
●How should the individual parameters be prioritized and binned?
               ○Solution = An optimal scheme should be determined via                    algorithm.
●How do you gauge success?


Video of the August 28, 2020 Meeting: Comparison WG VIDEO - 2020-08-28
Slides from the August 28, 2020 Meeting (cumulative): Comparison WG SLIDES - 2020-08-28
Chat Record from the August 28, 2020 Meeting: Comparison WG Chat - 2020-08-28

The “Difference of Differences” Calculation

• Step 1: Create baseline period model for treatment group and project that model as a counterfactual into the reporting period (Counterfactual_Treatment)
• Step 2: Create baseline period model for comparison group and project forward as a counterfactual (Counterfactual_Comparison)
• Step 3: Calculate savings as:
  
  

Savings = 
(Counterfactual_Treatment - Observed_Treatment) - (Counterfactual_Comparison - Observed_Comparison)

Video of the August 14, 2020 Meeting: Comparison WG VIDEO - 2020-08-14

Slides from the August 14, 2020 Meeting (cumulative): Comparison WG SLIDES - 2020-08-14

Chat Record from the August 14, 2020 Meeting: Comparison WG Chat - 2020-08-14

For most comparison pools, stratify on up to 3 parameters

1. Annual Consumption
2. Percent usage change from COVID
3. 3. A parameter of relevance to the program (likely a normalized metric)

Examples of normalized electric features

% Heating kWh

% Baseload kWh

% Summer Peak kWh

Video of the July 31, 2020 Meeting: Comparison WG VIDEO - 2020-07-31
Slides from the July 31, 2020 Meeting (cumulative): Comparison WG SLIDES - 2020-07-31
Chat Record from the July 31, 2020 Meeting: Comparison WG Chat - 2020-07-31

Goal: Develop and Demonstrate successful implementation of stratified sampling

• What parameters can/should be used
• Develop codebase and test/refine
• Multidimensional stratification

Question: What does it take (binning schemes) to create equivalency between treatment and comparison?









Video of the July 17, 2020 Meeting: Comparison WG VIDEO - 2020-07-17

Slides from the July 17, 2020 Meeting (cumulative): Comparison WG SLIDES - 2020-07-17

Chat Record from the July 17, 2020 Meeting: Comparison WG Chat - 2020-07-17

• Assumption: There is a singular event in a building that marks the beginning of a new pattern of energy consumption.
• We want to be able to calculate the difference between this new pattern of energy use and the energy use that would have occurred if the old pattern was still in place.

• Confusing term: baseline
  • Baseline can refer to the chronological time frame corresponding to the historical energy consumption of a building. For example, we might say “12 month baseline” and refer to the time period prior to the new pattern of energy use.
  • Baseline can also refer to the “counterfactual.” This is the estimate of how much energy would have been consumed at any given point if no changes had occurred in the building’s energy use pattern. For example, when we calculate savings, we subtract actual energy use from “baseline” energy use. In this case, we are subtracting from a calculated counterfactual rather than the actual historical consumption.
• Confusing term: normalize
  
  
  • Statistical term versus a weather term
    
    

• In California, the policy term of art used to describe meter-based programs is NMEC - Normalized Metered Energy Consumption.
• Normalized (statistical)
  • Identification of a key predictive factor that correlates well with changes in energy consumption
    • Outdoor weather conditions (dry bulb temp, wet bulb temp, wind, cloud cover)
    • Indoor conditions (occupancy, production, certain end uses)
  • What is the minimum set of data that we have available across all buildings that will allow for consistent, replicable analysis?
    • Outdoor temperature (dry bulb)
    • Indoor building state (occupied/unoccupied by inference)
      
      
  • Normalization means that we use a particular set of exogenous conditions to calculate a counterfactual energy consumption value
    • E.g., When the daily average temperature is 80, this building will use 27 kWh of electricity on average.

Video of the June 05, 2020 Meeting: Comparison WG VIDEO - 2020-06-05

Slides from the June 05, 2020 Meeting (cumulative): Comparison WG SLIDES - 2020-06-05

Chat Record from the June 05, 2020 Meeting: Comparison WG Chat - 2020-06-05

What motivates this work?

-- Overarching goal is to reduce the risk of procuring energy efficiency by using meter-based savings calculations --

Principles:

• Data that is used must be accessible to all parties to the transaction and generally available across all transactions
• Methods must be specified in such a way that they can be reasonably implemented in code and verified by third parties
• Where possible, methodological debates and questions are settled through empirical testing and analysis, the results of which should be published, if possible
• Open Source, by default.

Video of the May 22, 2020 Meeting: Comparison WG VIDEO - 2020-05-22

Slides from the May 22, 2020 Meeting: Comparison WG SLIDES - 2020-05-22

Chat Record from the May 22, 2020 Meeting: Comparison WG Chat - 2020-05-22