So, for this post, I’m going to use a ‘true story’ as explained by Daniel Kahneman in his mind-bending book ‘Thinking fast and slow’.

(Kahneman is a Nobel prize winning giant in the field of human psychology and I will be adding him to my group of giants soon).

Some years ago, Kahneman was invited to speak at an investment firm whose advisors provide financial advice to wealthy clients. I can almost hear them shouting “buy, sell…buy” across the trading floor.

Pre-meeting preparation.

Kahneman asked the firm’s executives for some data so that he could prepare for the talk he was due to give.

He was provided with a spreadsheet containing the investment outcomes of 25 of the firm’s advisors, for each of 8 consecutive years. No names, just anonymous identifiers.

The firm used the investment outcome success of each advisor as the main determinant of their (potentially large) year-end bonus.

…so what was Kahneman interested in understanding about this data set? And what did he do to interrogate it?

His thinking: That investment outcomes will be a combination of skill (on the part of the advisor) and luck¹

His question: How much of the outcome in this ‘providing expert investment advice’ work was down to skill and how much to luck?

How to determine the answer: Kahneman was interested in understanding whether any apparent differences in skill were persistent i.e. did the same adviser consistently achieve better (or worse) returns year on year?

To work this out he calculated the correlation coefficients² between the advisor rankings in each pair of years: year 1 with year 2, year 1 with year 3….all the way along to year 7 with year 8. This gave him 28 correlation coefficients from which to calculate the average.

• An average score close to 1 would mean that it was a very highly skilled job and the best (and worst) advisors were easy to identify – in this scenario, luck plays virtually no part;

• A score midway between 0 and 1 would mean that skill mattered a bit but that luck also had a huge part to play.

• Anything nearing 0 would mean that it was really just about luck.

So what were his findings and what does this mean?

Drum roll…he was surprised to find that the score was…0.01 or put more simply ‘zero’.

In Kahneman’s words “The consistent correlations that would indicate differences in skill were not found. The result resembled what you would expect from a dice-rolling contest, not a game of skill.”

Clarification: Just in case you are thinking “hey, that’s just one set of data. He got lucky!”…Kahneman knew roughly what he was going to uncover because this ‘person or system’ type analysis has been done many times by many people. He knew the theory and the evidence….he expected it to be low but he didn’t expect it to be soooo close to zero!

So what happened next?

Well, he ended up having dinner with the investment firm’s executives the night before he was due to give his talk.

He explained the question he had asked of the data they had provided to him and asked them to guess the year-to-year correlation in the rankings of their advisers.

The executives (being intelligent and self-protecting people) thought they knew what was coming and calmly accepted that performance certainly fluctuates and, yes, there was an element of luck…however, none of them expected the average correlation to be zero.

Kahneman gave them the clear message that “the firm was rewarding luck as if it were skill”.

This should have been a major shock, but it created no great stir…they calmly went on with dinner as if nothing of note had been said.

Kahneman goes on to write about “The illusion of skill: Facts that challenge such basic assumptions – and thereby threaten people’s livelihood and self-esteem – are simply not absorbed….people consistently ignore statistical studies of performance when it clashes with their personal impressions from experience.”

Why write this post?

There are two key points within the case above:

The first is that Kahneman’s story is an (extreme) example of the system vs. the individual. Yes, some people may be outstanding but a great deal of ‘performance’ can only be ascribed to the system in which they operate. (You might perhaps take note that investment advice is little more than a game of chance.)

But perhaps the second (and main) point is clearly expressed in the phrase “I don’t want to change my world”. The executives may very well accept ‘the maths’ and the conclusion…but that doesn’t mean they are about to change anything.

Consider that executives are probably also on a (larger) bonus structure which will have a similarly dubious rationale. We can expect little change unless and until those ‘at the top’ of an organisation understand, agree and want it.

People (such as me) can bang on about performance reviews and contingent rewards, providing ever increasing evidence and logic…yet (and this is an open question) what will cause a change?

1 This is another way of stating Deming’s x + y(x) = the result equation. i.e. the result is partly down to the person and a large part down to the system in which they operate (which is simply luck from the person’s perspective).

2 A correlation coefficient (usually denoted with the letter R) is a statistical measure of the strength of the relationship between two sets of data.

R = 1 means that the two sets of data are a perfect positive fit.

R = -1 indicates a perfect negative fit

R = 0 indicates that there is no relationship i.e. any relationship is purely random.

A correlation greater than 0.8 is generally described as strong, whereas a correlation less than 0.5 is generally described as weak.