If I were to ask people what their car dashboard gauge labelled “Speed” measures, all but that annoying engineer with the pocket protector would say, duh, the speed of the car. Obviously.
But they would be wrong.
The speedometer measures the rotation speed of the wheels. When multiplied by the circumference of the wheel this yields a very good approximation of the speed of the vehicle. So good that the average driver thinks that the speedometer measures the speed of the car.
Add wheel spin from icy roads, and the speedometer is suddenly not so good at measuring speed, especially if the car is sliding sideways towards a tree. The novice driver, seeing that they are going 92 miles per hour slams on the brakes, thus virtually guaranteeing that the car will simultaneously occupy the same space as said tree and sadly, abruptly ending their ‘good driver’ discount.
Over the past year, we have been examining rather closely the gauges which measure unemployment. In my weekly commentaries, I often whined that the unemployment people clearly were incapable of counting – really, how hard can it be? The three main gauges – weekly new claims, continued claims, and the monthly statistics from the Current Population Survey – had apparently gone all wonky.
Measuring short term trends in unemployment by using “new unemployment claims” is a reasonable choice, but alas, a surrogate. What happens if there is nobody in the unemployment office to receive the claim and the forms get stacked on an empty desk? Or their servers crash because of the unanticipated demand and nobody can even file the claim in the first place? The gauge is still reliably measuring what it always did – the number of completed new unemployment claims – but it is no longer a good surrogate for unemployment.
Dashboards are all the rage these days. Everybody wants one so that companies can nimbly adjust to an ever-changing environment. The terminology may have changed, and the speed at which gauges can be updated increased dramatically, but the reality is that companies have always used dashboards – the weekly sales report, the quarterly P&L, the annual balance sheet. Essential business information, summarized on a regular and predictable basis, are the gauges by which companies successfully navigate. They almost always work very well in monitoring normal conditions, and for the most part, provide useful information that management can use to make slight course adjustments.
The problem occurs when there is a disconnect between our surrogate measurement and the actual phenomenon, or when relatively little effort is put into the measures which are so neatly displayed with splashy graphics. The gauges will most assuredly go all wonky at those very times when a measurement is critical. In our car, the experienced driver can see with their eyes that they aren’t doing 92 mph and appropriately ignore the gauge. But what if the dashboard is our only window to reality?
A locational filtering and selection dashboard that flashes green or red may be reliable most of the time. But it will fail eventually because it uses “population” as a surrogate for demand when the area has a high institutional population. Or miss an opportunity because non-resident demand is not neglected. Or overestimate spending based on median incomes because housing costs are unusually high.
The dashboard user, having long come to believe that the “Speed” gauge actually measures the vehicle speed, will probably not recognize the disconnect until moment of impact.
Some obvious suggestions arise –
- Make sure that your surrogate measures (and they almost always are just that) are as precisely defined as possible. If you are spending more on the programming to make the dashboard pretty than on the underlying data and analytics, you are doing it all wrong.
- Minimize error where possible and practicable. While a free data source which is several years out of date is perfectly adequate “most” of the time, scrimp on presentation before you cut costs on data. An ugly and accurate dashboard is orders of magnitude preferable to a pretty and wonky dashboard.
- Make sure that users know exactly what that fancy graphic shows, and more importantly, what it doesn’t. If you don’t know the limits of the expected normal operation of your gauges, don’t make them in the first place.
- When in doubt, trust that one annoying field professional who is quietly shaking her head while pointing at the flashing green light on the dashboard. If she can’t tell you exactly why she disagrees, send her out into the field and let her find out. There is no substitute for human experience and intuition.