Marketing’s misleading metric
Introduction
In 1946, three Bedouin shepherds were walking along the northern shores of the Dead Sea when they noticed a series of rocky inlets. Overcome by curiosity, the trio ventured into the Qumran caves and soon stumbled upon a handful of brittle parchments.
Over the course of the following year, the shepherds discovered a total of six scrolls dating back to the 3rd century BC. The moment was monumental. The shepherds had discovered the oldest surviving manuscripts of biblical books.
News of the discovery soon spread and the potential for further findings captured the attention of the academic world. Archaeologists issued an incentive: anyone who could find additional scrolls, or fragments of the precious manuscripts, would be rewarded.
But the plan backfired.
Here’s Rolf Dobelli in his book, The Art of Thinking Clearly:
“When the Dead Sea scrolls were discovered, archaeologists set a finder’s fee for each new parchment. Instead of lots of extra scrolls being found, they were simply torn apart to increase the reward.”
By tearing the manuscripts in two, people doubled the reward they received. Whilst the scheme set out to recover the precious parchments, it set in motion a series of events that determined their destruction.
A goal was set, perverse incentives were created and unintended consequences followed.
When you’re aware of this, you start to see throughout history.
As the British Mathmetician, Hannah Fry put it in The New Yorker:
“Textile factories were required to produce quantities of fabric that were specified by length, and so looms were adjusted to make long, narrow strips. Uzbek cotton pickers, judged on the weight of their harvest, would soak their cotton in water to make it heavier. Similarly, when America’s first transcontinental railroad was built, in the eighteen-sixties, companies were paid per mile of track. So a section outside Omaha, Nebraska, was laid down in a wide arc, rather than a straight line, adding several unnecessary (yet profitable) miles to the rails.”
In economics this effect is known as Goodhart’s Law (when a measure becomes a target, it ceases to be a good measure) and in the social sciences it’s called Campbell’s Law (the more a metric is used to make decisions, the more it will be manipulated).
And, unfortunately, it is still happening today.
In his book, ‘The Tyranny of Metrics’, the historian Jerry Z. Muller explores many modern-day examples. When you judge teachers by their students’ exam results, they ‘teach to the test’. When you judge surgeons by their patients’ 30-day survival rate, they refuse the most complicated cases. When you judge police officers by the number of arrests they make, they focus all their energy on tackling petty crime.
Muller puts it plainly:
“Anything that can be measured and rewarded will be gamed.”
Perverse incentives, and the unintended consequences that flow from them, can be found on every continent, in every time, and in every industry.
And marketing is no different.
This article argues that a malevolent metric sits at the heart of many marketing discussions and decisions.
I believe that the many marketers who prioritise this metric fall into the same trap as the Dead Sea archaeologists. They seek to capture value, and unintentionally destroy it.
Let’s dive in.
Perverse incentives in marketing
In the summer of 2023, Marketing Week partnered with Kantar to poll more than 1,300 brand-side marketers for the publication’s second annual ‘Language of Effectiveness’ survey.
Among other questions, the respondents were asked which three metrics senior stakeholders deemed most important when assessing the success of a marketing campaign.
More than a third (41.6%) said ‘return on investment’ (ROI). Amongst the respondents who work at businesses with more than 250 employees, that figure rose to over half (52.5%).
ROI, it seems, is the metric that matters most for modern marketers.
And why wouldn’t it be? Of course we should measure the returns delivered by our investments. It seems entirely logical. Obvious. Inarguable, even. But, as you may suspect by now, the story is not quite so simple.
If you dig deeper into the ‘Language of Effectiveness’ survey results, you’ll find a subtler but arguably more interesting finding:
“48.1% of marketers believe their company is too focused on ROI when it comes to examining effectiveness, a slight increase on the 45.7% who reported the same last year, and significantly more than the 23.7% of marketers who believe the opposite.”
So while half of all business leaders prioritise ROI, half of all marketers believe this to be misguided. How can this be?
I believe that many marketers understand that setting ROI as the number one KPI for creative campaigns causes brand building attempts to backfire. Just as we’ve seen elsewhere, it gets set as a goal and it creates perverse incentives.
Or to be more specific, it creates three.
To maximise ROI marketers must prioritise efficiency over effectiveness. They must reduce their media money. And they must focus on the few consumers who are already most likely to convert. The result is an approach to advertising which suppresses a campaign’s profit producing potential.
I realise this may sound unbelievable. Paradoxical, perhaps. So let’s work through these points one at a time.
Optimising ROI incentivises efficiency over effectiveness
The term “ROI” is often used, somewhat sloppily, to refer to the incremental value sales produced by advertising activity. ROI, however, is not an absolute measure but a relative one. It doesn’t relate to the effectiveness of work, but to its efficiency. It doesn’t measure the total returns, but the returns in comparison to the costs.
In their book ‘How Not To Plan’, Les Binet and Sarah Carter provide an illustrative example:
“Suppose you spend £10 million on a campaign, and the profit on extra sales generated is £II million. Then net profit, after subtracting the campaign cost, is £l million. Net profit is a good measure of effectiveness - the more profit, the more effective the campaign. ROl, on the other hand, is a measure of efficiency, not effectiveness. It's calculated by taking the ratio of net profit generated (£1 million) to the cost of the campaign (£10 million). In this case, ROl is 10%.”
So £10 million is spent to generate £11 million in sales. The absolute return is £1 million, and the ROI is 10% (or 1.1:1.0 when expressed as a ratio).
Given this distinction between absolute effectiveness and relative efficiency, it doesn’t take much to imagine a scenario where a higher ROI could also mean a lower profit.
In fact, the marketing effectiveness consultancy Gain Theory provide an example of just this in their report ‘Profit Ability: The Business Case for Advertising’:
“Would you rather have a profit ROI of 2:1 on a spend of £3m (so £3m in profit) or a profit ROI of 3:1 on a spend of £500,000 (a profit of £1.5m)?”
If you were asked to choose between the two options with the goal of maximising the ROI you’d choose the second option (as 3:1 is a greater ROI than 2:1). But if your goal was to maximise profit, you’d choose the first (as £3m is a greater profit than £1.5m).
This is achingly obvious. And yet I see this mistake, this costly confusion, being made constantly.
Here’s Paul Worthington, President of Invencion Inc., writing in his essay ‘Weaponizing the Wanamaker Paradox’:
“ROI has been driven so deeply into the modern marketers’ psyche that few understand that it’s a measure of efficiency rather than effectiveness.”
So ROI is not a measure of advertising’s absolute effectiveness, but a measure of the efficiency with which it works. And by optimising for high ROIs, a well-meaning marketing team can actively reduce the profit that they produce.
One common question that gets asked following examples like these is, ‘Why wouldn’t you choose the high-ROI option and then increase your investment?’. But this misunderstands the second reason that ROI is not the most meaningful of metrics.
As investment increases, the efficiency with which it delivers returns almost always decreases.
Optimising ROI incentivises a reduction in media spend
Few people are better qualified to comment on the effectiveness, and efficiency, of media buying than Jerry Daykin, who has held senior media positions at Mondelez, Diageo, GSK and Beam Suntory.
In an article for his Digital Sense newsletter, Daykin explains how ROI doesn’t scale linearly. If it did, then a $1k investment that delivers $2k of returns could be scaled to a $1m investment to realise $2m. If you were to depict this relationship on a graph with ‘investment’ on the x-axis and ‘return’ on the y-axis, the relationship would be a straight line. Increase one, and the other would increase proportionally.
But this isn’t what actually happens.
The graph is never a straight line, but a curve that rises steeply, reaches a point of diminishing returns and quickly flattens out. Here’s Daykin again:
“It's almost always true that spending less and buying cheaper forms of media will improve your score [ROI], though it certainly won't lead to a more impactful campaign. Digital channels in particular have benefited from high ROIs in their early days because advertisers invest relatively little in each channel.”
In the lower left of the graph you have your lowest, but most efficient, spend. $100 spent returns $400, for example. As you increase the spend, this efficiency starts to fall. $1k delivers $3k, say. And then as you ramp up spend even further the efficiency falls again. $1m generates $2m, if you like.
Once you understand that ROI does not scale linearly, you immediately understand how dangerous an “optimise ROI” objective is. The easiest way to achieve this goal is to reduce your spend. And as we’ve seen before, whilst this can increase your ROI, it can also result in a reduction in absolute profits.
Tom Roach, VP of Brand Strategy at Jellyfish and ex-Head of Effectiveness at both BBH and adam&eveDDB, puts a fine point on this:
“ROI tends to inversely correlate with profit growth, as due to diminishing returns ROI decreases as you spend more, and increases as you spend less. So the easiest way to increase your ROI is to decrease your media spend.”
Daykin, takes this argument to its wonderful extreme. A high ROI is not something to be proud of. It is a sign that your spend is too low:
“Having a high ROI (…) most likely means you are not spending enough money to maximise an opportunity. Rather than celebrating your efficiency you should really be working to unlock more spend within your business so you can invest and grow. Only when the marginal ROI flat lines and your additional investment starts losing you money should a growth focussed business really put the brakes on.”
Counterintuitively, marketers should focus on maximising their absolute returns, which is generally achieved through higher media spend that delivers a lower ROI.
By why does this happen? Why does ROI decline as spend increases? This brings us to our third, and final, perverse incentive.
Optimising ROI encourages the targeting of existing buyers
In 1961, a young researcher named Andrew Ehrenberg published his first ever article in the Journal of Advertising Research (JAR). In the decades that followed, Ehrenberg published more than 300 papers on subjects including advertising, pricing and new product launches. But perhaps his biggest breakthroughs were the ‘laws of growth’ that he, and his colleagues, uncovered from analysing decades of data on buyer behaviour and purchase propensities.
One such finding, was that brands grow primarily by increasing penetration rather than loyalty. The implication of this, is that growth focussed brands must be constantly recruiting new buyers.
In his book ‘How Brands Grow’ Byron Sharp, Director of the Ehrenberg-Bass Institute of Marketing Science, puts in plainly:
“To grow, a brand needs to recruit more light category buyers.”
This poses a problem for marketers who are looking to optimise ROI. Whilst new buyers are critical for growth, they are also the least likely to convert, and thus reaching them leads to lower ROIs.
Here’s Paul Worthington, again:
“In order to grow a business, advertising ROI will go down. Why this happens is simple. Growth means attracting customers who aren’t already inclined to buy from you, and customers who aren’t already inclined to buy from you are more expensive to attract than those who are. It’s really that simple.”
A campaign that specifically aims to optimise ROI, then, will focus on reaching heavy buyers who are already most likely to convert. This in turn will harm the long-term growth of the brand as it will fail to develop demand beyond that which already exists. It will pick the fruit without watering the tree.
Here’s Tom Roach, referring to Return on Advertising Spend (ROAS):
“Brand growth comes disproportionately from light buyers, but focusing on high ROAS can lead to you targeting more and more heavy buyers, so limiting growth. It can make brands inward-looking and too focused on existing customers, rather than on reaching new customers.”
Celebrating campaigns that have a high ROI, usually means celebrating work that only reaches those who are already most likely to buy. Whilst these metrics are seductive, they ultimately indicate a failure to reach a wider audience, a failure to increase penetration and a failure to grow a brand’s buyer base.
The opposite is also true. Dismissing and deriding campaigns that deliver a low ROI is, quite often, deriding campaigns that are attempting to achieve advertising’s most meaningful objective: generating demand among the many light buyers.
To wrap up, here’s Jerry Daykin once again:
“It's (…) lighter users that ultimately allow a brand to grow and become more successful, penalising campaigns aimed at reaching them starts to look like shooting yourself in the foot.”
Conclusion
So there you have it. Around half of marketers say that return on investment (ROI) is their senior stakeholders’ most important campaign metric.
And yet, in doing so they are falling into the same tired trap as our Dead Sea archaeologists. They are setting a singular metric without understanding the unintended consequences that will follow.
For this reason, the legendary Les Binet and Sarah Carter pull no punches in their book ‘How Not to Plan’:
“Trying to maximise ROI is a good way to destroy your brand.”
Strong stuff.
However, there is a slightly more nuanced take.
In his Media In Focus report, written alongside marketing consultant Peter Field, Binet describes how metrics such as market share correlate with profit growth whilst metrics such as ROI correlate with activation effects.
“Profit growth correlates most closely with broad improvements across the range of long-term business metrics, especially sales and market share growth (all at the 99% confidence level). It also correlates with improvements to brand strength. Correlation with ROMI is much weaker and not significant with 99% confidence. In marked contrast to this, ROMI correlates most closely with very large activation effects.”
In short, whilst ROI is a dangerous metric to use for long-term, brand-building campaigns, it is entirely valid for short-term, tactica, sales activations.
This is a sentiment echoed by Professor Mark Ritson in the documentary Making a Marketer 2. I’ve edited this quote slightly, for brevity:
“There’s two sides to ROI. For performance marketing ROI makes perfect sense. You’re doing short, targeted, product-based, bottom-of-funnel activations. You know what you’re spending, you know when it’s operating, and you have an immediate measure of what return it generated. I have no problem with ROI for performance. The problem is, it’s going to shift our investment in favour of short-term performance and out of longer-term brand-building. We might make more money for a couple of months but across two or three years we’d have been better keeping that balance between long and short. And ultimately the ROI will take us almost to zero. It’s a dangerous metric but also one that can be used happily for short-term stuff. It’s when we get to that long term brand-building investment, that ROI lets us down.”
And so, this is marketing’s misleading metric.
By setting ROI as a metric for long-term brand building campaigns, well-meaning marketers set in motion three insidious implications which work against their ultimate objectives.
They prioritise relative efficiency over absolute effectiveness. They incentivise a harmful reduction in media spend, and they encourage the targeting of existing buyers over the light-buyers who matter most.
This, I believe, is a symptom of marketing’s most major problem. ROI has become the predominant metric in advertising. The number of ‘short-term’ campaigns has almost tripled, and the average effectiveness of those campaigns has fallen through the floor.
It’s time we rallied against this. It’s time we injected some serious ambition back into our brands. It’s time we stopped setting the stage for brand-building attempts to backfire.
Because no big brand is built on the spending of a little, to target a few. They are built on spending seriously. They are built on reaching the masses in the middle. And they are built on prioritising profit above all else.
So focus on maximising your total returns, not on your returns on investment.
Or as our three Bedouin shepherds might say, focus on the scrolls and not on the scraps.
Notes
A big thank you to Toby Ososki for his guidance and support on this essay.