Why Metrics Fail

How often do you run across metrics used by larger corporations that count and catalog nearly everything, but really communicate nothing?  How often are these systems an overwhelming collection of numbers but not focused on the vital few, and worse yet, don't indicate where you stand relative to benchmarks, or how long it will take you to get there?  Unfortunately we run into this all to frequently, but fortunately, this is easy to remedy.

The most important element is very basic, and easy to correct.

For example, which representation tells you when to expect you will reduce your error rate to 20%?

1.  Most Typical:

Date: November 12, 2009

Error Rate:  42%

2.  Close Second:  

Date: Nov. 12, 2009

Error Rate: 42%

Date: Nov. 19, 2009

Error Rate: 39%

3.  Recommended

You can easily see that a graphical display can tell you much more quickly and accurately where we stand relative to the goal, how much deviation we have against the target, and what the predicted trend is likely to be.   Good metrics have the following properties:

 
Error rate versus time
 

1. They are always shown in a graphical format

2. The target curve is shown, and if appropriate changes over time

3. Good is indicated so there is never confusion

4. Definition of the metric, the data set, and assumption is declared on the chart itself