You know those tech interviews we all hate? Science says they're the best we can do. At least the modern kind.
I was reading (audiobook) Kahneman's latest book – Noise – and one of the chapters talks about interviews. The whole book is great but this chapter stood out because everything people say is bad about modern interviews is actually good 🤯
Before we get into interview stuff, a few basics from the book.
Bias vs Noise
Kahneman et al talk about statistical noise and how it impacts decision making in situations where human judgement is involved. The book uses examples like interviews, judicial proceedings, the behavior of juries, business decisions, medical diagnoses, and forensic science.
If you think of shooting at a target, noise describes the grouping of your shots. How similar to each other they are.
Bias describes a consistent error – drift from the target. There are stats formulas you can use to compute bias by measuring "average" distance from target. Not all noise is bias and not all bias is noise.
Crucially: you can study noise without knowing the target. But you can't study bias without relating results to a target.
PS: the shooting example comes from the book, this is not commentary on anything
Types of noise
Kahneman et al identify 3 types of noise.
- Systematic noise caused by consistent biases in judgement. Like an engineer who always under-estimates Jira stories.
- Interpersonal noise happens when different experts review the same data and come to different conclusions. Like when engineers disagree on a story estimate.
- Situation noise describes how the same expert changes their judgement based on random situational factors. Like an engineer who under-estimates when they're excited and over-estimates when tired.
Bias cascades
An important lesson from Noise focuses on bias cascades where different reviewers influence each other.
Kahneman et al share a story from a college admissions committee. They used to have blind evaluations where each member evaluated applications independently. But that caused too much strife and disagreement so they changed their approach.
Now each person writes their estimate on the application for all to see. Peace and harmony ensued.
Can you spot the problem?
Yep. Whomever writes the first eval determines that student's fate. Evaluators see the score and think "Yeah that sounds about right" instead of creating their own score.
Worryingly this also happens in forensic science. If the case-file shares the cops' opinion, the fingerprint expert is more likely to agree that fingerprints match 💩
Insights for interviews
Here's what all that means for interviews.
Old style "chat" interviews where you talk to a candidate and have a conversation feel the most effective. People love them because they get a strong gut feel for the candidate and think that's good.
Those interviews have a 50% correlation with on-the-job performance. Useless. The decision has more to do with the interviewer than the interviewee.
Structured interviews work better. Prepare questions in advance, define an evaluation rubric, and ask everyone the same questions in the same order. Then evaluate their answers against the rubric, not against how you feel. This reduces situation noise.
When you take structured interviews and add a committee without bias cascades – everyone submits their eval against the rubric before discussing the candidate – you get about a 65% correlation with on-the-job performance. They mention Google's process as having data to back this up.
About 4 independent interviews is a good balance between noise, time, and diminishing impact of additional opinions.
Structure interviews around job-like tasks and you get a 70% to 75% correlation with future performance. That's the best we can do.
Why should you care?
Next time you're interviewing and think this bullshit is a random hazing ritual, remember that a good interview process has a 75% chance of predicting how you'll do on the job. And that the more diverse a candidate you are, the more structured interviews are in your favor. ✌️
If you get to design the process in your company, do the above. You'll find great people that way.
Cheers,
~Swizec
PS: according to the book, executives have about a 55% success rate with business decision making. About as good as a chimp throwing darts 😂
Continue reading about Insights for interviews from Kahneman's Noise
Semantically similar articles hand-picked by GPT-4
- What if tech interviews aren't bullshit
- What to expect in senior level interviews
- You're not asking for a job, you're selling a service
- Life is Poker
- The best engineering books get good 5 years into your career
Become a *true* Senior Engineer
Get promoted, earn a bigger salary, work for top companies
Getting that senior title is easy. Just stick around. Being a true senior takes a new way of thinking. Do you have it?
The Senior Minset email crash course
Get a free chapter from the Senior Engineer Mindset book and a sample audiobook chapter, followed by a Senior Mindset 101 email course.
You'll get insights to apply at your work right away.
Have a burning question that you think I can answer? Hit me up on twitter and I'll do my best.
Who am I and who do I help? I'm Swizec Teller and I turn coders into engineers with "Raw and honest from the heart!" writing. No bullshit. Real insights into the career and skills of a modern software engineer.
Want to become a true senior engineer? Take ownership, have autonomy, and be a force multiplier on your team. The Senior Engineer Mindset ebook can help 👉 swizec.com/senior-mindset. These are the shifts in mindset that unlocked my career.
Curious about Serverless and the modern backend? Check out Serverless Handbook, for frontend engineers 👉 ServerlessHandbook.dev
Want to Stop copy pasting D3 examples and create data visualizations of your own? Learn how to build scalable dataviz React components your whole team can understand with React for Data Visualization
Want to get my best emails on JavaScript, React, Serverless, Fullstack Web, or Indie Hacking? Check out swizec.com/collections
Did someone amazing share this letter with you? Wonderful! You can sign up for my weekly letters for software engineers on their path to greatness, here: swizec.com/blog
Want to brush up on your modern JavaScript syntax? Check out my interactive cheatsheet: es6cheatsheet.com
By the way, just in case no one has told you it yet today: I love and appreciate you for who you are ❤️