For those of you Python fans out there, I suspect the title of this post draws a smile of recollection from you. It draws out a big hearty grin from me.

For those of you who don’t know what I am writing about (and for those who do…but would like to relive the moment – go on, you know you want to!), here’s the famous clip from the Monty Python film ‘The Life of Brian’:

What have the Romans… (1 min. 25 secs)

This clip was triggered in my mind the other day when pondering how people collect and use data in reports (I had just seen one that offended my sensibilities!). I get frustrated when I point out a serious fault within a report and the response I get is “yes, but apart from that….”

Here’s my attempt at a Python-like response:

Leader (John Cleese): “Look at what this report is telling us!”

Minion 1: “…but we don’t have enough data to know what’s actually happening.”

John Cleese: “What?”

Minion 1: “We are only using a couple of data points to compare. This tells us virtually nothing and is likely to be highly misleading.”

John Cleese: “Oh. Yeah, yeah. We have only got this month vs. last month. Uh, that’s true. Yeah.”

Minion 2: “…and we’re using averages – we’ve got no idea as to the variation in what is happening.”

Side kick 1 (Eric Idle): “Oh, yeah, averages, John. Remember some of the mad decisions we’ve made in hindsight because of averages?”

John Cleese: “Yeah. All right. I’ll grant you that our lack of data over time and the use of averages makes our report a bit suspect.”

Minion 3: “…and, even if we did have enough data points and could see the variation, we don’t understand the difference between noise and a signal (common and special cause variation)”

John Cleese: “Well, yeah. Obviously we don’t want to be caught tampering. I mean, understanding the difference between common and special cause goes without saying doesn’t it? But apart from a lack of data, (miss)using averages and tampering – ”

Minion 4: “We often compare ‘apples with pears’: Lots of the things we ‘hold people to account for’, they have virtually no ability to influence.”

Minion 5: “Much of the data we use is unrepresentative and/or coerced out of people, which makes any data biased.”

Minions: “Huh? Heh? Huh… “

Minion 6: “And we are focusing on one KPI and not seeing the side effects that this is causing to other parts of the system.”

Minions: “Ohh…”

John Cleese: Yeah, yeah. All right. Fair enough.

Minion 7: “and we are using targets, which are arbitrary measures that have nothing to do with the system and cause dysfunctional ‘survival’ behaviours from our people.”

Minions: “Oh, yes. Yeah… “

Side Kick 2 (Michael Palin): “Yeah. Yeah, our targets cause some pretty mad behaviours, John, and it’s really hard to spot/ find this out because our people don’t like doing ‘bad stuff’ and, as such, don’t like to tell us about it. Huh.”

Minion 8: “Our reports are focused on people (and making judgements about them), rather than on the process that they have to work within.”

Eric Idle: “And our people are ‘in the dark’ about how the horizontal value stream they work within is actually performing, John.”

Michael Palin: “Yeah, they only know about their silo. Let’s face it. If our people knew how the horizontal flow was actually doing, they’d be far more engaged in their work, more collaborative (if we removed some of the management instruments that hinder this) and therefore far more able and willing to continually improve the overall value stream.”

Minions: “Heh, heh. Heh heh heh heh heh heh heh.”

John Cleese: “All right, but apart from a lack of data, (miss)use of averages, tampering, comparing apples with pears, biased data, focusing on one KPI, the use of arbitrary targets, reports focused on judging people, and our value workers being ‘in the dark’….Look at what this report is telling us!”

Minion 9: We’re using activity measures (about outputs), rather than seeing the system and its capability for our customers (about outcomes).

John Cleese: Oh. Seeing the capability of the system from the customers’ point of view? SHUT UP!

• THE END –

In short, many (most?) organisations are terrible when it comes to measurement. They are stuck in a weird ‘conventional reporting’ world. Perhaps this is a blind spot in our human brains?

‘Statistics’ is a word that strikes fear into the hearts and minds of many of us. I’m happy to admit that I’m no expert. But I think we should have a healthy respect for data and how it should and should not be used. I’ve heard many a manager raise their voice to say that they have the data and so can ‘prove it!’…and then go on to make inferences that cannot (and should not) be justified.

(Personal view: I think that it is better to be mindful (and therefore cautious) of our level of competence rather than blissfully ignorant of our incompetence, charging on like a ‘Bull in a china shop.’)

Where to from here?:

I’ve previously written a few posts in respect of measurement. I’ve linked a number of them in the skit above or in the notes below. Perhaps have a (re)read if you’d like to further explore a point I’m attempting to make.

…and here’s a reminder of the brilliant Inspector Guilfoyle blog that is dedicated to measurement. He writes nice ‘stick child’ stories about the mad things we do, why they are mad…and what a better way looks like.

Some closing notes on some of the ‘reporting madness’ points made above:

Binary Comparisons: Here’s a really great explanation of the reasons why we shouldn’t use a couple of data points: Message from the skies

Averages: If you don’t understand the point about averages, then have a think about the following quote: “Beware of drowning in a river of average depth 1 metre.” (Quoted by John Bicheno in ‘The Lean Toolbox’)

Variation: Deming’s red bead experiment is an excellent way to understand and explore the point about variation that is inherent in everything. I’ve written about variation in (what happens to be my most read post to date): The Spice of Life

Tampering: This comes about from people not understanding the difference between common and special cause variation. I wrote a specific post about the effects of tampering on a process: Tampering

Biased data: There are loads of reasons why data collected might be biased. The use of extrinsic motivators (as in contingent monetary incentives) is a BIG one to consider and understand.

Targets: John Seddon  is the place to go if you want a deeper understanding of the huge point being made. His book ‘Freedom from Command and Control’ is superb. Also, see my post The trouble with targets.

Capability measures: I believe that this point can take a bit to understand BUT it is a huge point. I wrote Capability what? In an attempt to assist.