By Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein
I think I grabbed this on a whim because I was a fan of Thinking Fast and Slow and I saw Kahneman’s name on this as I walked through Changing Hands. It was at the top of my stagnant pile of non-fiction book purchases, so I started on it. I ended up switching to audiobook on Libby. With a bunch of cardio to do and a couple long plane flights, I got through it pretty quickly on 2x. It was not a real page turner and felt repetitive. I think I would’ve struggled to get all the way through without the help of audiobook. I’m sure my retention is lower for it, but while I was listening, I wasn’t hit by any unusually impactful points that I was afraid of losing. The conclusion chapter summarized everything pretty well. If I want to recall the insights offered by this book, I think I could get 90% of it by just re-reading the conclusion.
- Bias is a common concern for why judgments are made poorly, but noise is also very prevalent but ignored
- In many cases, noise occurs independent of bias and can be resolved independent of bias. Also, it is worth doing so even if bias remains (although not necessarily if the anti-noise solution makes bias worse).
- Noise can be observed and addressed even when the “correct” outcomes are unknown or unknowable. The book keeps referring back to a metaphor of archery observed from the back of a target. The observer wouldn’t know how close any of the shots were to the bullseye, but it would be clear if the grouping was tight or if it was all over the place. If the shots are all over the place, there’s a noise problem regardless of where the bullseye was located on the front of the target.
- Performance reviews are typically 25% signal and 75% noise
- Unstructured interviews are very noisy
- Simple rules/formulas are less noisy than unbounded human judgement and often (but not always) result in a lower amount of error. Many people don’t like this and take it as an insult to their intelligence and ability to judge.
- Besides variations between people, a single person is inconsistent with himself for the reasons I’ve seen in other books (like Determined, Thinking Fast and Slow, Predictably Irrational). Blood sugar, sleep levels, prompting with positive or negative information prior to the decision, and common fallacies make us unreliable.
- Averaging (truly) independent judgements can reduce noise. See: Wisdom of Crowds. They must be truly independent. Cascades can happen if communication happens before judgements are made.
- Hard rules that are objectively true or false depending on the facts reduce noise in a way that standards (open to personal judgment) do not.
- Ranking is less noisy than rating separately although forced ranking can create errors of its own. A small group doesn’t necessarily represent the full bell curve of performance, so fitting the rankings to the curve could be entirely wrong.
- Well described examples can be used to make rating more like ranking. A thorough rubric reduces noise. Comparing a review subject with levels on a rubric allows for clear comparisons like ranking but without a forced rating outcome.
- In cases where measurable predictions are made repeatedly, data can be collected to measure the efficacy of decision making. In other circumstances, the outcomes may be immeasurable or unknown. In others, a decision may be unique and made only once. The same principles of good decision making, that reduce noise, can be applied to those other cases.