Measuring in vain
Imagine a multi-disciplinary team at a ‘brand health’ debrief. Marketers are around the table in force, along with colleagues from innovation, planning and operations, plus a smattering of agency people.
Version A: the news is good. The brand equity pyramid is in improved shape, with a nice tight line from awareness to bonding. Customer satisfaction and net promoter scores are now best in class. Social measures are good, with Twitter followers and Facebook ‘likes’ both up.
Version B: the precise reverse. All those metrics are pointing the wrong way, down from where they were last quarter.
Now imagine the emotional reaction of each individual member of the team. In version A, each attributes the positive news to the good work they and their team have been doing. In version B, conversely, it is pretty clear to each person around the table that it is someone else’s fault.
How can this be possible? Given the unequivocal crispness of the data, how can so much room be left for interpretation? The answer is that these metrics, like so many in our industry, quantify a status, but do not reveal what led there.
They are what the entrepreneur and author Eric Ries has termed ‘vanity metrics’ – because they allow everyone to view them in the most agreeable light. It’s not that they don’t measure something important – they do, and often with great accuracy – it’s just that they leave out causality, which is a pretty big gaping hole.
Ries argues that business needs to focus, instead, on ‘actionable metrics’. The high ground here is the binary test, whereby two variants of the same product are released at the same time.
If Version Red attracts more paying customers than Version Blue, and all else is equal, you have causality – and you have a plan.
The examples that Ries offers to show the power of this approach in practice are impressive – but they are all in the online domain. Therein lies the caveat for most marketers. Without denigrating the task of developing thousands of lines of code, in two slightly different versions, it is simpler than producing simultaneous variants of physical products.
You can’t, for example, build two hotels in exactly the same spot and compare how one kind of reception design brings back more paying guests than the other.
Nonetheless, techniques do exist that can help marketers avoid vanity metrics and encourage the habit of linking measurement to action.
A classic but neglected tool is conjoint analysis, which forces people to imagine iterative trade-offs in product or service features, including price, revealing what customers value most.
Hierarchy of effects models – sometimes called choice chains – measure where customer fall-off is most marked between the extremes of ‘unaware’ and ‘evangelical’; they won’t tell you exactly what to do, but they can tell you where to prioritise action.
Marketers often complain about ‘data overload’ and the ‘paralysis’ it engenders. Is that a result of too many assessments of current status and too few directional tools?
Like it or not, marketers do need to measure. The challenge is to develop more imaginative ways to ensure we do not measure in vain.
What the man behind the Lean Startup movement has to say about ‘vanity metrics’
Ries, a thirtysomething Silicon Valley entrepreneur, is credited with pioneering the ‘Lean Startup’ movement. After building momentum through blog posts, conference talks and advisory stints, the movement reached fruition when Ries’ book was published in 2011. Nowadays, Lean Startup concepts are employed by high-tech companies, such as Dropbox, and taught at Harvard.
The Yale graduate’s early successes and failures with various high-tech enterprises helped him formulate the principles that brought the no-waste ideology of Japanese lean manufacturing to the digital domain. By utilising continual deployment, testing and feedback from the start of development, and hence cutting out unnecessary processes early on, Ries believes his concept can reduce wasted time and resources.
Central to Lean Startup’s philosophy is the elimination of any practices not adding value to the customer; ‘vanity metrics’ is one of Ries’ biggest targets. These include commonly used culprits such as website hits and registered users and give ‘the rosiest picture possible’, but do not uncover business drivers or raise actions.
‘Vanity metrics’, explains Ries, represent energy wasted on ‘making people think that you’re being successful rather than energy put into serving customers’.