Rory Sutherland, Nate Silver and the measurability bias

In an article for The Spectator, Ogilvy Chairman Rory Sutherland discusses how companies filter prospective employees based on their grades.

“You need to whittle down applications somehow. And to create a spurious veneer of objectivity, recruiters all fall back on the same, lone quantifiable measure (degree class).”

Rory argues that this may not be the best technique.

“I have asked around, and nobody has any evidence to suggest that, for any given university, recruits with first-class degrees turn into better employees than those with thirds (if anything the correlation operates in reverse). There are some specialised fields which may demand spectacular mathematical ability, say, but these are relatively few.”

Reading between the lines a little, Rory believes that to excel in most fields of employment, recruits need an array of skills, most of which cannot be quantified by exam results. Despite this, a majority of value is attributed to a minority of measurable skills.

“In the words of F.A. von Hayek (praise be upon him) ‘Often that is treated as important which happens to be accessible to measurement.’”

To labour the point, we are biased towards the measurable.

You may ask why an ad man is interested in education. The truth is that the measurability-bias frequently affects the creative industries. In his TED talk, ‘Life Lessons From an Ad Man’, Rory tells the story of a brief given by Eurostar.

I’ve edited the transcript slightly for brevity.

“The question was given to a bunch of engineers, about 15 years ago, “How do we make the journey to Paris better?” And they came up with a very good engineering solution, which was to spend six billion pounds building completely new tracks from London to the coast, and knocking about 40 minutes off a three-and-half-hour journey time.

Now, call me Mister Picky, but it strikes me as a slightly unimaginative way of improving a train journey merely to make it shorter. What you should in fact do is employ all of the world’s top male and female supermodels, pay them to walk the length of the train, handing out free Chateau Petrus for the entire duration of the journey. Now, you’ll still have about three billion pounds left in change, and people will ask for the trains to be slowed down.”

While he’s painting a humorous picture, his point is clear. The speed of the journey was measurable, so it was overvalued. The experience of the journey is far less measurable, so it was undervalued.

Nate Silver covers the same point in The Signal and the Noise:

“Statheads can have their biases too. One of the most pernicious ones is to assume that if something cannot easily be quantified, it does not matter. In baseball, for instance, defence has long been much harder to measure than batting or pitching. In the mid-1990s, Beane’s Oakland A’s teams placed little emphasis on defence, and their outfield was manned by slow and bulky players, like Matt Stairs, who came out of the womb as designated hitters. As analysis of defence advanced, it became apparent that the A’s defective defence was costing them as many as eight to ten wins per season, effectively taking them out of contention no matter how good their batting statistics were.”

It’s important to uncover what aspects of a product or service consumers actually value. What demands aren’t being met? What is valued but underserved? Then, when you build a proposal, build on top of that understanding. Don’t build on something purely because it is measurable.

Value and measurability are independent.