So, for this post, I’m going to use a ‘true story’ as explained by Daniel Kahneman in his mind-bending book ‘Thinking fast and slow’.
(Kahneman is a Nobel prize winning giant in the field of human psychology and I will be adding him to my group of giants soon).
Some years ago, Kahneman was invited to speak at an investment firm whose advisors provide financial advice to wealthy clients. I can almost hear them shouting “buy, sell…buy” across the trading floor.
Kahneman asked the firm’s executives for some data so that he could prepare for the talk he was due to give.
He was provided with a spreadsheet containing the investment outcomes of 25 of the firm’s advisors, for each of 8 consecutive years. No names, just anonymous identifiers.
The firm used the investment outcome success of each advisor as the main determinant of their (potentially large) year-end bonus.
…so what was Kahneman interested in understanding about this data set? And what did he do to interrogate it?
His thinking: That investment outcomes will be a combination of skill (on the part of the advisor) and luck1
His question: How much of the outcome in this ‘providing expert investment advice’ work was down to skill and how much to luck?
How to determine the answer: Kahneman was interested in understanding whether any apparent differences in skill were persistent i.e. did the same adviser consistently achieve better (or worse) returns year on year?
To work this out he calculated the correlation coefficients2 between the advisor rankings in each pair of years: year 1 with year 2, year 1 with year 3….all the way along to year 7 with year 8. This gave him 28 correlation coefficients from which to calculate the average.
- An average score close to 1 would mean that it was a very highly skilled job and the best (and worst) advisors were easy to identify – in this scenario, luck plays virtually no part;
- A score midway between 0 and 1 would mean that skill mattered a bit but that luck also had a huge part to play.
- Anything nearing 0 would mean that it was really just about luck.
So what were his findings and what does this mean?
Drum roll…he was surprised to find that the score was…0.01 or put more simply ‘zero’.
In Kahneman’s words “The consistent correlations that would indicate differences in skill were not found. The result resembled what you would expect from a dice-rolling contest, not a game of skill.”
Clarification: Just in case you are thinking “hey, that’s just one set of data. He got lucky!”…Kahneman knew roughly what he was going to uncover because this ‘person or system’ type analysis has been done many times by many people. He knew the theory and the evidence….he expected it to be low but he didn’t expect it to be soooo close to zero!
So what happened next?
Well, he ended up having dinner with the investment firm’s executives the night before he was due to give his talk.
He explained the question he had asked of the data they had provided to him and asked them to guess the year-to-year correlation in the rankings of their advisers.
The executives (being intelligent and self-protecting people) thought they knew what was coming and calmly accepted that performance certainly fluctuates and, yes, there was an element of luck…however, none of them expected the average correlation to be zero.
Kahneman gave them the clear message that “the firm was rewarding luck as if it were skill”.
This should have been a major shock, but it created no great stir…they calmly went on with dinner as if nothing of note had been said.
Kahneman goes on to write about “The illusion of skill: Facts that challenge such basic assumptions – and thereby threaten people’s livelihood and self-esteem – are simply not absorbed….people consistently ignore statistical studies of performance when it clashes with their personal impressions from experience.”
Why write this post?
There are two key points within the case above:
The first is that Kahneman’s story is an (extreme) example of the system vs. the individual. Yes, some people may be outstanding but a great deal of ‘performance’ can only be ascribed to the system in which they operate. (You might perhaps take note that investment advice is little more than a game of chance.)
But perhaps the second (and main) point is clearly expressed in the phrase “I don’t want to change my world”. The executives may very well accept ‘the maths’ and the conclusion…but that doesn’t mean they are about to change anything.
Consider that executives are probably also on a (larger) bonus structure which will have a similarly dubious rationale. We can expect little change unless and until those ‘at the top’ of an organisation understand, agree and want it.
People (such as me) can bang on about performance reviews and contingent rewards, providing ever increasing evidence and logic…yet (and this is an open question) what will cause a change?
1 This is another way of stating Deming’s x + y(x) = the result equation. i.e. the result is partly down to the person and a large part down to the system in which they operate (which is simply luck from the person’s perspective).
2 A correlation coefficient (usually denoted with the letter R) is a statistical measure of the strength of the relationship between two sets of data.
R = 1 means that the two sets of data are a perfect positive fit.
R = -1 indicates a perfect negative fit
R = 0 indicates that there is no relationship i.e. any relationship is purely random.
A correlation greater than 0.8 is generally described as strong, whereas a correlation less than 0.5 is generally described as weak.
4 thoughts on ““I hear what you say…but I don’t want to change my world””
Funnily enough I got an amazing opportunity yesterday to do something similar. A bunch of data landed on my lap which measures ‘performance’ which people are asked to justify and are made accountable for. A cursory glance suggests this data and subsequent performance is all over the place and it is the work system that defines performance, not ‘performance management in 1 to 1’s’.
I am a really big fan of your blog and I will investigate what a squire to a Squire is.
I just wanted to say thanks for this. It’s really useful to have something tangible that could be used to prove/ disprove luck or skill. I’m going to try and use it to show the impact of the targets my organisation insist on using. I’m sure it’ll get most people clamping their hands over their eyes, but what fun to prove it!
Well in my experiment so far I have found differences between individuals which has surprised me.a little. That said there is clear evidence that the reward one receives bears no relation to performance, the expected output per day is more aspirational than an average and performance varies significantly per day per individual and as a team. Whilst I can see that some people are better at working ‘in the system’ it is the type of demand, flow of work and performance of the IT system and associated work processes that impact on productivity, not ‘1-1’ meetings or the people in general. Anyone surprised?
Hi there. Glad you are finding it of use.