I’ve recently revisited Sam Savage’s "The Flaw of Averages: Why We Underestimate Risk in the Face of Uncertainty", a book he published in 2009 building on his original article from 2000. Savage’s concept is that decisions made based on average assumptions are often flawed due to overlooking the variability in data. Savage illustrates how this statistical pitfall can lead to disastrous outcomes across various fields, from finance and business to healthcare and climate change, urging a more nuanced approach to understanding and managing risk.
In one example, Savage describes the path of a drunk walking down the middle of a road. If the drunk's position at any point is an average, it might suggest he is safe in the middle of the road. However, considering the variability and uncertainty of his movements, it's clear that he's in danger of veering into traffic, which the average position fails to predict.
This metaphor illustrates the flaw of averages by showing how focusing on an average (the drunk's position in the middle of the road) can dangerously overlook the variability and uncertainty in data (the drunk's unpredictable movements).
Figure 1 – Jeff Danziger’s original cartoon from the 2000 article © Jeff Danziger
Just as the drunk's average position misleads us about his safety, so too can average marketing ROI mislead us about our marketing effectiveness.
Measuring marketing ROI and the flaw of averages
If your business has achieved even modest scale, the chances are that your marketing activity covers several channels, multiple customer segments, multiple product/service offerings and potentially multiple geographies. You might also be spending time and resources targeting existing or returning customers as well as brand new customers. However, even with this level of complexity, you will still probably focus on an overall measure of marketing effectiveness, whether that is marketing costs as a % of revenue, cost per acquisition (CPA) or return on investment (ROI).
The flaw of averages tells us that focusing on an aggregate, average measure may lead to overlooking the range and diversity of outcomes from marketing efforts, and potentially missing opportunities for increased efficiency and growth. For example:
Misinterpretation of Campaign Performance: A marketing campaign could include several different components, each with different ROI – for example different channels. By looking at the average ROI, marketers might miss understanding which components are working well and which ones aren't. This could lead to misallocation of resources.
Ignoring Variability: The ROI from marketing efforts could be highly variable. Some campaigns might result in very high ROI, while others might yield low or negative returns – or perhaps the variability occurs over time. An average figure could hide this variability, potentially causing surprises in future campaigns. An ROI of 2.5 with low variability should be viewed very differently to one with high variability.
Masking Outliers: There might be certain marketing efforts that result in exceptionally high or low returns. These outliers could significantly impact the average ROI, but they might not represent the typical outcome of a marketing effort.
Discrepancies in Target Audience: Different segments of the target market – customer segments or geographies - may respond differently to the same marketing effort. Using averages may hide these discrepancies and could lead to an inefficient allocation of resources.
The message from these examples is clear – make sure to segment your analysis of marketing performance across a range of dimensions to understand performance, rather than relying on average measures. If you are a business leader or investor – dig into average measures and ask to understand the level of variability in the underlying data.
Setting average targets for marketing ROI
One area where I think the flaw of averages can be particularly troublesome for marketers is when setting targets for marketing ROI. I’m thinking specifically about businesses that spend a significant amount of their budgets on performance digital marketing such as paid search/PPC. Having done an analysis of customer lifetime value (LTV), marketing leaders will often use the LTV:CPA ratio to select an ROI target that will deliver an acceptable balance of growth in customer numbers and profitability. However, the challenge of using an average ROI target on a day-to-day basis is that significant parts of your marketing budget are likely to be loss-making or marginally profitable.
The approach I’d advocate instead is to adopt a minimum ROI on a day-to-day basis. This is much more practical to implement than trying to manage a large portfolio of activity towards an average number. Perhaps you set a minimum ROI 25%-50% lower than the average ROI you are trying to achieve and monitor both closely. I’ve found that adopting this approach reduces variability in performance and focuses front-line marketers on profit & commercial outcomes.
So, the next time you are preparing or reading headline, average measurements of marketing effectiveness, remember the flaw of averages and think about what interesting opportunities and learnings might be hiding within. Be sure to dig into performance at granular campaign level, customer segment, product line, new vs existing customer and over time – all should generate valuable, actionable learnings.
If you’d like to discuss how you can better understand marketing ROI in your business, please Contact Me.
All views expressed in this post are the author's own and should not be relied upon for any reason. Clearly.