Why Interviewing Prospective Hires is Futile, and How to make it (only slightly) Less Futile
Idk how this relates to job interviews. But posts need cool pictures and this one is awesome.
Note: Republishing this post I shared on the Medium paid subscription a few years ago. This was originally written in July 2017.
We’re back with a new Edition about Job Interviews. Here’s what you get in this post:
Why Job Interviews are (Nearly) Useless
Manage a Good Job Interview Process
What To Not Do (What not to do? Idk which is right)
What to Look for When Interviewing
A Shitload of Great Interview Questions
Skip around as you please — enjoy and please add any great content or concepts you think are missing.
Why Interviews are [Nearly] Useless
The first interesting fact that I learned researching how to do job interviews well is that they are nearly useless. This is actually proven with academic studies.
I didn’t know this when I chose it as a topic, but here we are. We’re all going to keep doing them anyway so we may as well eke out all the benefit that we can from them.
Here’s why interviews have so many flaws.
The Observer Effect
The act of observing something will influence the thing being observed. This absolutely seems like the right place to start. The Observer Effect turns an interview into a professional lying session. What I hear in a typical interview:
Interviewer: “Welcome! Please lie to me about your relevant work experience?”
Employee: “[Extrapolates, expands, and exaggerates]”
[…]
Employee: “Now, please lie to me about a typical day at your company?”
Interviewer: “[Skips, skims, and smoothes over].”
This seems unlikely to create a strong lasting relationship. It is like getting married after the first few dates, when everyone is only sharing their most generous assessments of themselves.
The *Utter* Uselessness of Job Interviews
According to the NY Times, job interviews aren’t just useless, they are *utterly* useless. Citing studies they had done on whether interviews where additive in a GPA prediction challenge:
In the end, our subjects’ G.P.A. predictions were significantly more accurate for the students they did not meet. The interviews had been counterproductive.
Research that my colleagues and I have conducted shows that the problem with interviews is worse than irrelevance: They can be harmful, undercutting the impact of other, more valuable information about interviewees.
To go even farther to illustrate the point, they also conducted a round of the study where some prospects, unknown to the interviewer, were to give answers determined at random.
Strikingly, not one interviewer reported noticing that he or she was conducting a random interview.
Granted, maybe these people sucked at interviewing, but it does show the weaknesses of the interview structure rather well.
The key psychological insight here is that people have no trouble turning any information into a coherent narrative. People can’t help seeing signals, even in noise.
The Job Interview Process
No matter your process, have a plan. Have it down on paper so your whole team is aware, and you can communicate clearly with your prospect.
Division of Labor
Evaluating prospects is a nuanced skill. It requires practice. Managers who are hiring only a few positions per year are NOT coming from a good place to evaluate talent. They have no sense of base rates.
Chet Cadieux, the CEO and legendary operator of QuikTrip, has a great take on Interviewing. He has dedicated full-time interviewers. Not just an HR screening call, but full-time expert hirers for every position in QuikTrip, down to cashiers in convenience stores.
QuikTrip is a case study in genius operators using simple principles. To read more about QuikTrip, check out Intelligent Fanatics, a wonderful book about some of the world’s best operators.
And thanks to Sean Iddings for contributing Chet’s talk!
Extreme Reference-Checking
My friend Jonathan Howard sent in a post by VC Josh Hannah about Extreme Reference-Checking. It was the only post submitted that talked about it in any detail, and I think it completely nailed it. Everything I’d wish for in a post about Reference Checks.
Here are the gems:
Resumes, CVs and interviews are all sales pitches. The candidates have carefully crafted the attributes they show you in order to seem perfectly suited to the job.
Base 80 percent of your hiring decision on thorough reference checking
But not the way you’re currently doing reference checks… because you’re probably doing them wrong.
In my experience, the leading cause of executive hiring mistakes can be traced to waiting to make (perfunctory) reference checks until the final step in the hiring process, perhaps even after the decision has been made.
You call those references with a confirmatory bias, practically begging them to tell you that the interviewee is a great marketer, so you can get her on board and get back to work.
If you’re not making these calls seeking truth, you shouldn’t even bother. Be in the mindset to dig for negativity, take any answer as an opportunity to drop a prospect.
Expect at least a 50 percent chance of eliminating the candidate based on references. If you are not in this mindset, you will only hear what you want to — that you’ve found the perfect person.
Keep that confirmation bias out of your reference checks — do them early, and do them thoroughly. Read the rest of Josh’s post, there was too much great stuff to excerpt all of it.
Tryouts are better than Interviews
Get as close as you can to a real work environment with a prospect. Contract project, Shadow day, a few sales calls, research sample, etc.
Matt Mullenweg is known for an extreme implementation of this:
The most significant shift we’ve made is requiring every final candidate to work with us for three to eight weeks on a contract basis. Candidates do real tasks alongside the people they would actually be working with if they had the job. They can work at night or on weekends, so they don’t have to leave their current jobs; most spend 10 to 20 hours a week working with Automattic, although that’s flexible.
The goal is not to have them finish a product or do a set amount of work; it is to allow us to quickly and efficiently assess whether this would be a mutually beneficial relationship. They can size up Automattic while we evaluate them.
It’s not always possible to do this scale of “tryout”, but the more you can stretch for, the better.
Evaluating prospects during actual work with the team is the strongest predictor of success in the role, according to a fancy academic study shared with us by Itamar Goldminz.
What Not To Do
As always, we try to learn by inversion. Not just what to do, but we take the steps to find and avoid common pitfalls. Avoiding stupidity is usually easier than achieving brilliance.
No Ghosting
Ben Center shared this great short post by veteran recruiter Beth Scheer called “The Maybe Pile”. The short version? You don’t get a Maybe pile.
I had the confidence to tell hiring managers and executives that putting people in the “maybe” pile was likely to give the candidate a bad experience. “Maybe” really means “no for now.” Better to tell a candidate that you don’t have a role for them right now so they can put their energy into finding a job somewhere else. If something changes, you will be in touch, but for now the park is closed.
Toughen up, buttercup. You want to run a good process for prospects? Don’t leave them hanging. (This has the added benefit of forcing decisions and keeping your hiring funnel clean and clear.)
Don’t Talk Too Much
I once witnessed someone conduct a 90-minute interview in which they talked at least 80% of the time. This interviewer decided that they loved the candidate and hired the prospect on the spot. He was fired in less than two weeks. Not a happy ending for anyone.
Roger Toor has corresponding advice:
Don’t fall into the trap a lot of interviewers do, of making it about you. All too often I see interviewers asking questions to show off how smart they are or that they happened upon something obscure. Or they end up talking a lot about themselves.
We’re all smart and like to show off a bit. An interview provides an imbalance of power uncommon in everyday life, and it can be tempting to use that to bask in our own egos for a while. But that is not going to help you build the right team. Shake it off.
The Other Mistakes
I was going to collect all of the mistakes managers make when hiring… but it turns out I don’t have to because Jason Evanish already did it for me on his Lighthouse Blog.
What to Look For When Interviewing
Ok — Here’s the heart of this Edition. The good stuff on what to look for, and how to find it.
Have a Blueprint
It seems like most companies have a loose idea of what they want in a hire, and a set of questions… but few are truly connected. An interview process should be seeking confirming and disconfirming evidence. Having a rubric-type document to work through the process seems the best way to stay organized, impartial, and intellectually honest.
Tucker Max, founder of Book In A Box, shared with us some of the objective measures they use in their interviews.
Find Proof of Internal Motivation
Is there proof of internal motivation or external motivation?
Does their resume focus on credentials and titles (inputs), or results (outputs)?
Do they talk about high status people or places they worked, or about the actual work they did?
Why do they do what they do?
Find Proof of Ethics and Character
Are they a good person? How do I know?
Do they do the right thing, even when they don’t have to? Why?
Do they speak truth to power?
Do they try to bullshit you at any point in the interview?
Pay attention to the little things — Do they interrupt me a lot? Do they respect me? Do they respect all people, e.g. how did they treat Rick (our driver) when he picked them up from the airport?
Having all of these “proof points” written out, and as a structured guide to the interviewer reduces the dangers of confirmation bias, liking bias, narrative bias… It keeps things more organized, more effective, and more honest.
Keeping this sheet updated throughout the process, by different interviewers can help keep candidates straight, and ensure your process gets deep, rather than 4 different people conducting the same surface-level interview.
Trajectory, Not State
It is much easier to assess a snapshot in time (state) than correctly predict a pattern (trajectory). I can tell you the score at half-time, for sure. I’m less certain who is going to win the game.
Be sure you are evaluating a candidate’s trajectory, not their current state. When you’re going to have a long and happy working relationship with someone, you want to know they are growing with you, not in stasis or declining.
This short post, Trajectory Matters More Than State, covers this dynamic in candidates and companies. Pay attention to trajectory and do your best to understand what this candidate might be like in months or years.
Thanks to Vinish Garg for submitting this!
Look for those “Soft Skills”
If you’ve ever worked with a talented jerk, you have an appreciation for the importance of soft skills. Internal Communication and coordination are incredibly important to the function of a company, and these are doomed if your team lacks soft skills.
This post by Seth Godin talks about how you can make a plan for assessing soft skills, and how to look for them.
Wisdom — Have you learned things that are difficult to glean from a textbook or a manual? Experience is how we become adults.
Perception — Do you have the experience and the practice to see the world clearly? Seeing things before others have to point them out.
Influence — Have you developed the skills needed to persuade others to take action? Charisma is just one form of this skill.
There are many more examples in the post, and you’ll need to develop your own set of desired attributes for each position.
Finding Attributes through Questionnaires
Matt Frost shared this post by Adam Grant: “What’s wrong with job interviews, and how to fix them.” Grant covers all the ways Interviews miss the mark, and has a novel suggestion for solving them.
The most effective questions are called situational judgment questions. Instead of asking candidates to describe how they handled a unique situation in a previous job or organization, it’s more fruitful to describe consistent situations that candidates could face in this job or organization, and ask them what they would do — or how they would reason.
That’s not the novel part. That’s just the setup for this:
Interestingly, these situational questions don’t require interviews at all; they can be completed and scored online. Researchers have validated situational tests to assess applicant characteristics as diverse as integrity, personal initiative, emotional intelligence, and aggression. In one study with Dane Barnes of Optimize Hire, we gave salespeople a situational questionnaire about their tendencies to take initiative, and then tracked their annual revenue. Here’s what we found:
Low initiative: $53,798
Moderate initiative: $118,808
High initiative: $155,663
If this works, it should mean big things for hiring — knowing what to look for, and having scalable, cheap online tests that look for them is a dream come true for companies. Of course, it means honest cooperation from applicants.
I’m super interested to see how processes like these find their ways into hiring over the next few years. It seems like pure hubris that has kept more objective research from entering the hiring process thus far.
Why Google Loves Structured Interviews
Google has some of the highest standards for its employees of any company today. Solving the challenge of how to keep this bar high across every hiring manager in the company has some challenges:
In other words, most interviews are a waste of time because 99.4 percent of the time is spent trying to confirm whatever impression the interviewer formed in the first ten seconds.
This article in Wired (submitted by Kathryn O’Day, thank you Kathryn!) talks about how they’ve solved it.
The goal of our interview process is to predict how candidates will perform once they join the team. We achieve that goal by doing what the science says: combining behavioral and situational structured interviews with assessments of cognitive ability, conscientiousness, and leadership.
To help interviewers, we’ve developed an internal tool called qDroid, where an interviewer picks the job they are screening for, checks the attributes they want to test, and is emailed an interview guide with questions designed to predict performance for that job.
This makes it easy for interviewers to find and ask great interview questions. Interviewers can also share the document with others on the interview panel so everyone can collaborate to assess the candidate from all perspectives.
The article is excellent, but it is just the start — get the book, Work Rules! by Laszlo Bock for the full Google hiring playbook.
Here’s a Shitload of Great Interview Questions
As promised — all of the best questions and collections of questions we could find for you. There’s a lot of them so I kept the context short.
Ask “Why? Why? Why? Why?”
Until they start crying or just walk out. Kidding. This post shows you how to use “Why?” to get past the interview-level bullshitting and see what people are actually made of.
Thanks to Vignesh Srinivasan for contributing.
How to ask about a candidate’s “Grit”
Grit (determination) is a key attribute for success in most roles. But you can’t just ask if someone is a determined person. Here is how to ask questions that will show you the secret of Grit.
One Question to ask to find the top 5%
Shane Mac learned something valuable from Navy Captain David Marquet. One question that would help identify people most likely to be top performers. And here it is:
“Did you feel at your last job that you were appropriately appreciated?”
Read (slightly) more in Shane’s short post.
Vinod Khosla’s Giant List of Questions
This amazing list of questions is the final section in Khosla’s magnum opus on hiring. It has all kinds of great advice and suggestions that are worth setting aside some time to work through.
This post also from Vinish Garg, thanks Vinish!
Another Giant, Comprehensive Hiring Guide
There’s a ton of info in this “How to Hire The Right Person” post from NYT Business, from questions to the setting in which to ask them. Too much great stuff to excerpt, but a valuable resource for those about to hire.
What to Ask and What it Tells you
We couldn’t make it through an Edition of Evergreen without linking to some fantastic post on First Round Review, and here it is. Excellent Questions about surfacing soft skills, and interpreting candidate answers that aren’t always easy to parse.
The Importance of Great Follow-Ups
Lastly, let’s look at Book in A Box again. Tucker’s company doesn’t just have a great rubric to work against, they keep a great bank of questions to have on hand. Importantly, they ALSO plan follow-up questions, which is where the good stuff really comes out.
Here’s a sample:
“Tell me about your all-time biggest professional screw-up.”
Wait for their full answer. THEN follow up after they tell you about it, and ask: “What did you do about it?”
“Tell me about the best idea you’ve ever had that your boss/collaborators/client didn’t go for. The idea that got away.”
Wait for their full answer. THEN follow up after they tell you about it, and ask: “What did you do when they said no?”
“What’s your favorite thing or activity to do, work or non-work related? Why?”
Wait for their full answer. THEN follow up after they tell you about it, and ask: “What have you started or initiated in that area?” If nothing, ask why not.
And to close out, here’s my favorite question off any of these lists:
“If we don’t hire you, why do you think that would be?”
Final Thoughts
There are a ton of opinions about how to do job interviews. I think this huge variety is possible because of how little any kind of interview seems to impact the ability to get the right person in the seat.
At the end of all of this research, I came away thinking: “It seems like everything else AROUND the interview actually matters much more.”
Things that matter more than your Interviewing Skills:
Your company’s “brand” to employees
recruiting strength of the organization
available talent pool, the marketing of the position
ability to “sell” prospective hires
company culture shapes how employees work
Sure, try to do it well. But don’t obsess over perfecting it. Work harder on skills with higher returns.