Experimentation + Bayesian MMM: The Weird Science of Marketing Measurement
In the classic 80’s comedy movie “Weird Science”, two nerdy, lonely teenage boys use a personal computer, a Barbie doll, a voodoo ceremony, and a lightning strike while wearing brassieres as hats to create their dream woman. “Weird Science” parodies the classic film “Frankenstein” in which a mad scientist unintentionally creates a monster.
Like the boys from “Weird Science” and Dr. Frankenstein, in the world of Marketing Measurement, there are well-intentioned but needy parties trying to compensate for the limits of the real world by creating a strange new alchemy: Experimentation + Bayesian MMM.
What’s Behind the Invention of Experimentation + Bayesian MMM?
As the industry is painfully aware, the deprecation of third-party cookies makes it increasingly difficult to track individual customers, their advertising exposure, and their subsequent purchasing behavior. As attribution becomes more difficult, marketers are shifting their measurement efforts to Media Mix and Marketing Mix Modeling (MMM). Marketing Mix Modeling is the “tried and true” solution to measure performance at market and campaign levels, using aggregated data as opposed to individual-person cookie data.
The strength of MMM is that it uses aggregated data, which eliminates the need for personal identifiable information (PII). Nevertheless, MMM requires abundant first-party and third-party data from a variety of sources. The more sophisticated your MMM, the bigger your datasets get. Therefore, when embarking on an econometric modeling program, it is exceedingly important to get the data requirements and methodology aligned from the get-go.
Given the commitment required for a successful MMM program, it is only human nature that people will seek shortcuts. In addition, advertising platforms want to encourage advertisers to continue to spend on their platforms. Hence, we have the “Weird Science” proposition of Experimentation + Bayesian MMM. The proposed techniques rely on experiments to create the base case to calibrate Bayesian MMM models.
Experimentation + Bayesian MMM: Why is it Weird Science?
The proposed Experimentation + Bayesian MMM methodologies rely on the use of experiment results as the base case for MMM results. This is a fundamentally flawed approach to econometric modeling. Any MMM program should be based on comprehensive econometric modeling, where base assumptions are not forced or derived from partial (or siloed) information. Basing an MMM on experiment results inverts the logical order of analytics. The data-driven econometrics behind an MMM should create the base metrics to understand the marketplace, and then, and only then, should experiments be used to validate insights and expand insights to test new activities. Experiments should be used in the decisioning stage of an analytics program, not in the base data collection or modeling stage of an analytics program.
Test-and-Learn Measurement with Walled Garden Data Can Produce a Siloed View
Another common, flawed practice in Experimentation + Bayesian MMM is using experiments from walled-garden environments to represent an entire marketplace. Walled gardens, by their very nature, are not representative of the world outside their walls. So, even though experiment design methodology is sound, test-and-learn results will be misleading if experiments are limited to information from a siloed ecosystem.
And the only thing worse than extrapolating an unrepresentative experiment to the entire market is using an unrepresentative experiment as your base case for econometric modeling.
Sophisticated advertisers leverage multiple media channels across different platforms. With no single channel or platform dominating, even the biggest channel only accounts for a fraction of total business impact. Experiments limited to a siloed ecosystem will overlook the other media channels/platforms. Due to variations of other media outside the siloed ecosystem, measurement results will swing up and down. Overlooking any portion of an advertiser’s media mix will create misleading and unreliable measurement results. Likewise, measurement results inevitably fluctuate due to variations in experiment time periods and/or geolocations, and therefore risk creating misleading information when generalizing experiment results across different time periods and geolocations.
Experimental Design Scalability Limitations
Advertising platforms promote experiments based on the available information within their own ecosystems, neglecting the plethora of information from other media channels and platforms. Measurement results from siloed experiments are partial, unstable and sensitive to variations in time and geolocations due to other media weights and execution.
These “Weird Science” modified-MMM creations are born of good intentions: How can advertising platforms provide advertisers with performance metrics cheaper and faster, while simultaneously encouraging them to continue advertising? However, accurate MMM intelligence is not something that can be generated fast and cheap, no matter how much someone tries to Frankenstein together disparate methodologies.
About the Analytic Partners Position
We are strong advocates for experiments. Our experienced team and our platform’s Agile Learning capabilities ensure that experiments are conceived, configured and tested in a representative and unbiased manner. Our goal isn’t to sell more advertising on any particular platform, but to partner with our customers to optimize their performance across all channels and platforms.
The statistical and econometric principles that underpin our platform are born of MMM. However, on our 24+ year journey to exceed our customers’ needs, we have evolved beyond marketing and media measurement to create a Commercial Intelligence solution.
We believe that measuring marketing is not enough. We measure all factors that create value for brands. We also believe that measurement is not enough. Measurement without decisioning tools and discipline is just a report card of past performance. Our Commercial Intelligence solution enables our customers to make decisions with confidence, based on a full, commercial view of their business.