8th July, 2018

How dodgy metrics replaced judgment based on experience

There is a sweeping belief that more data is the answer to everything. The power and profitability of tech companies seem to confirm this. So track records, experience, domain knowledge, intuition and judgment are cast aside in favour of a fixation over metrics.

I often complain about an addiction to spreadsheets at board meetings. These rows of numbers overemphasise the importance of quantitative stuff and make even intelligent people believe in guesses about the future. I frequently fall for the illusion that financial forecasts are true, simply because they are backed up by lots of detailed spreadsheets, but in reality, they are just speculation.

A historian called Jerry Muller has torn into the infatuation with quantifying performance in his new book, The Tyranny of Metrics. It is a short and highly readable account of the way such management systems are undermining important institutions, such as universities, schools, policing, charities and even companies.

Many bureaucrats like metrics because they are supposedly objective. The vast spread of IT into every aspect of life means many managers rely too much on computer generated data, substituting it for thought. Fear of litigation or prosecution has increased the quantity of record-keeping, red tape and box-ticking, rather than finding the most efficient way of doing things.

I’ve learnt over the years that key performance indicators (KPI) are frequently gamed by executives to make their organisation appear more successful. So sharp doctors turn down difficult patient cases to improve their scorecards; public company bosses ignore long-term investment to pander to the demands of analysts and impatient investors; police distort crime statistics to make their clear-up rates look good. And everyone manipulates results to meet bonus targets.

Tom Peters might be partly to blame, since he popularised the concept that “what gets measured gets done”. The modern organisation replaces judgment based on experience with numerical indicators and worships at the altars of “accountability” and “transparency”.

Muller itemises the problems with metric fixation. They include goal displacement through diversion of effort to what gets measured; short-termism; costs in employee time; diminishing utility; rewarding luck; discouraging risk taking, innovation and common purpose; and the degradation of work.

When deciding whether to make an investment, it is easy to become snow blind, overwhelmed by the vast quantity of material generated by financial, legal, market and other due diligence reports. Despite all that work, most deals boil down to a few salient issues — price, management, timing and so forth.

Numbers and ratios matter a great deal, but they are not everything. Vital intangibles such as brand goodwill, customer loyalty, culture, research and development and staff talent are impossible to count but are hugely important. Yet there are no standard measures for such factors.

A new book that partly contradicts the philosophy behind The Tyranny of Metrics is John Doerr’s Measure What Matters. He is the billionaire chairman of venture capital firm Kleiner Perkins, which has backed both Google and Amazon. He claims to have helped create more than 425,000 jobs.

His doctrine is that all organisations must measure OKRs — Objective Key Results (which sounds awfully like KPIs). He learnt this from Andy Grove, who ran Intel in its heyday using this model.

Doerr says all great companies have structured goals and are clever at setting priorities to obtain the best from their teams. This involves applying quarterly and annual OKRs and using them to reward managers. At least he focuses on measuring outcomes not inputs — what is produced rather than what is spent.

Doerr is very rich indeed, and hugely well connected. I suspect his OKR tool has been productive when deployed at Google, Intuit, YouTube and Intel. In not-for-profits and the public sector the approach may not be so effective.

Taxpayers want their cash to be spent wisely, but when the not-for-profit sector apes business, it can be destructive. Motivations among doctors, teachers, police and other public servants are different — performance payments and rankings can encourage gaming and distort behaviour.

Over the past 10 years, society has probably got carried away in its mania for metrics inside organisations — there needs to be some rebalancing.