Intro to Incremental Measurement: Test & Control
How a basic scientific principle is used to measure marketing
The foundation for incremental measurement is the test & control methodology. You may have encountered this by a variety of different names: treat & control, holdout groups, controlled experimentation, A/B tests, etc. These terms are used in some scenarios to denote differences in application or purpose, and sometimes interchangeably. Ultimately, they are all based on the same core principles of the controlled experiment. Most of us have conducted at least a few simple controlled experiments as early as elementary school, but I offer the below example as a refresher:
You are given two identical potted tomato plants; Same size, same pots, same soil mixture, planted at the same time, from seed of the same parent plant, as identical as you can ensure. In order to measure the impact of plant food you provide one plant, Plant A, with 5 grams of the plant food once every morning and the other, Plant B, is provided no plant food. Aside from plant food the plants are treated the same way every day. You give them the same amount of water and sunshine, you even sing them the same Katy Perry song every morning to help them attack the day with confidence. After two weeks have passed you measure the plants height and compare it to the measurements you made at the beginning of the experiment. Plant A has grown 11in, and Plant B has grown 10in.
In this example the plant food is the independent variable, the plant receiving the food is the test plant, and the plant not receiving the plant food is the control plant. Without our control plant for comparison we may know that our test plant grew 11 inches, but we wouldn’t have any understanding how much of that 11 inches is because of the plant food. Given the growth of the control plant we can estimate the impact of the plant food is 1 inch of incremental growth over the observed period. We can use this same basic setup to analyze the effect of our marketing actions. For purposes of this discussion I’m going to focus on utilizing control group comparisons in order to gauge the impact of a marketing action vs. not taking that marketing action. The same concept is frequently applied using A/B tests in order to optimize specific elements of a marketing action.
Control groups can be created at various levels of any organizational hierarchy for marketing actions by excluding a portion of customers from that activity. Depending on the size of your customer base, frequency of marketing touches, and risk tolerance you may decide to maintain control groups at multiple levels. I find these 3 general levels to be applicable to most marketing organizations.
- All Marketing- As the name implies this level involves keeping all marketing efforts from touching a portion of customers/prospects.
- Strategic Initiatives- This level is much more arbitrary. It is based on grouping marketing activities based on a major strategic characteristic important to your business. An example of this would be holding out a portion of customers from all of your up selling campaigns.
- Campaign/Inbound Interactions- This level is specific to a campaign run or inbound interaction. Outside of the specific touch the treat and control groups should have the same opportunities to interact with marketing efforts.
Generally speaking, the incremental measurements at a lower level can be aggregated to calculate the total incremental impact of a higher level. For example, when control groups are created at the campaign level they can be used to calculate the incremental impact of that specific campaign and then the impact of all your campaigns can be aggregated to assess the total incremental impact of your marketing efforts. Aggregating up is almost always acceptable, but it does introduce some additional possibility of creating overlap or to miss synergies between campaigns. Make sure you understand how you’re attribution model works if doing so.
Using higher level control groups to measure impact at lower levels is something I would avoid. A campaign cannot reliably be measured incrementally based on a control group created at the all marketing level. Due to the control group being held out from all other marketing activity its circumstances both before and after the specific campaign are vastly different than that of the test group being touched by the campaign you wish to measure. This can greatly bias your experiment, especially in noisy marketing environments.
If deciding on a single control process, I would default to the lowest level you want to measure. Typically, this will be the campaign/interaction level.
While the concept of test & control is quite simple, implementation can get complicated fast. Complexities will grow when making considerations for recurring campaigns, inbound customer experience, or other unique factors of your marketing process. When deciding on how to handle these complexities keep in mind the fundamentals of the controlled experiment and always understand what you’re independent variable is.
To conclude, I know two questions have been killing you.
- How can you determine the value of plant food based on only 2 plants split into test and control? — We may have not utilized any statistical analysis for our elementary school experiments, but in the next post I’m going to address that question by introducing statistical significance.
- What Katy Perry song am I playing to those plants to get that kind of growth in just 2 weeks? — If you don’t already know this, you’ve already lost.
Have questions about implementing control groups, want to share how you implement test & control for your marketing? Maybe you just want to let the world know what kind of music you play for your hypothetical tomato plants. That’s why comment sections exist, start typing!
If you can’t get enough marketing and analytics. Follow me here or on LinkedIn to find out when new posts are available.
Check out the next post in this series to understand how significance testing can add context to your incremental calculation!
