Not everything that matters can be measured

  
Not everything that matters can be measured, and furthermore, not everything that  can be measured, matters.

VW is evidently in big trouble. Much is being made of their disingenuous software that optimises engine performance to the preferences of 11m drivers hitting buttons marked “Economy” and “Sport”, but also with scientists hitting the button on another dashboard, marked “Emission Test”. 

Just like crime in multi-story car parks, this is wrong – on so many levels. And I don’t mean VW’s crime; for me (and New Scientist) it’s the reaction that’s wrong, and here’s why:

We can measure the weight of your sofa, or the width of your doorway, because these don’t change. 

However, today’s temperature cannot be measured. Nor can your height, or the heat from your fridge or anything that constantly changes. We could report the highs and lows and that would be easy. 

But a third type of measurement is even harder; consider the performance of your local post office. Should the unit of comparison be the time to get served? Or the time spent in the queue? Should we measure this with the smallest number of customers and workers? Or the average, or the maximum?  Or the ‘typical’? Or a fixed number to ensure the comparison is, err, “Fair”. “Equal”? “Comparable”?

You might wonder how these kind of things are measured? As it happens, I know a lot about that!

I have been in the testing business, testing computers. What we did was to run a ‘Standard Program’. However, we would then make adjustments to get a good result. We would turn off recovery mechanisms, adjust the system options to provide the resources in demand, and divide the data between large numbers of storage disks (which are not being tested). 

Them we would run the system as fast as we possibly could! To do this we used another computer to throw work at it, and yet another computer to measure its performance. The other computers would need to be about twice as fast as the computer being tested. This makes it particularly hard to measure the world’s fastest computer!

If we succeed in this we have measurement that can tell people how fast their computer could theoretically go in a ridiculously unrealistic situation that would never occur outside the test laboratory. And it is a fantastic success! The measurements are keenly awaited, fastidiously curated, and used ubiquitously o estimate the relative actual performance of systems from their relative theoretical performance. 

Back to VW; I know from the theoretical result that the car could in extreme circumstances achieve the reported performance. I know that I can compare this with other cars measured under similar conditions, even if my conditions will be different. 

After all, how could those conditions be the same? Would the tested car have the same weight of people as my family? Would the humidity and temperature match the average in my corner of London? Would the test car  be driven with the same mixture of Eco-conscious cruising and playful (but legal) acceleration? I don’t think so. 

As a computer programmer I can only admire VW for their considerate provision of a truly optimised ‘test mode’ in the engine control unit (or ECU). 

Perhaps one way in which VW went wrong might be the same as the problem explained by Peter Sellars in “Dr.Strangelove”. It’s great that that you have Mutually Assured Destruction, but it only works if you TELL EVERYONE!

So, as we observed before, not everything that matters can be measured, and furthermore, not everything that can be measured, matters.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s