2917
Views
Points
Comments

Hiring experienced engineers is one of the most difficult and important things that engineering leaders have to pull off. But it’s hard to gauge experience in a series of short interviews. I’ve definitely worked with some amazing engineers who probably wouldn’t have been hired in some of my previous hiring pipelines.

Here are some tips on things I’ve found that work and don’t work.

✅ Things that Work

  1. Case Studies. I love this approach, though it requires a major investment to come up with a good case study and it requires a bit more commitment from interviewees to prepare. Basically, you prepare a 1-2 page story that lays out, a particular technical scenario in deliberately broad brushstrokes, and then asks the candidate to figure out what they’d do. The specific problems are left a bit vague, because half of what you’re trying to figure out is how they go about framing the problem (that’s a key indicator for experience). I’ve found this approach to give really high signal: in other words, candidates tend to do really well or really poorly, and there aren’t a lot of “mehs.”
  2. Three Why’s Technique. Gauging experience is not about producing a list of technologies or problem-solution anecdotes. The details really, really matter. I landed on something I’m calling the “Three Why’s” (like 5 Why’s) – it’s the practice of asking someone to describe something, and then pressing them three more times for more details. It’s amazing what kinds of clarity this brings. I’ve had candidates tell me about what seemed like a really boring, generic project, and after pressing for details, it turned out to be filled with extremely interesting problems, tons of learned heuristics and gotchas that they assumed were just not appropriate to include in a typical time-strapped interview.
  3. Ask them to Break the Rules. This is a more specific instantiation of Peter Thiel’s famous interview question “What important truth do very few people agree with you on?” Anyone can tell you about design patterns or best practices. It’s a mark of hard-won experience to get a detailed, well-reasoned answer about when not to follow the rules and go off the beaten path. There can be wrong answers1I know, there’s “no wrong answer” in any interview, but…we all know there are here – not everyone can turn experience into the right intuition. Also, watch out for answers that represent an overcorrection for a time they’ve been badly burned by some mistake, or a dressed up generic opinion that’s in vogue at the time.

❌ Things that Don’t Work

  1. Trusting job history over your gut instinct. There have been a few times where a candidate looked great on paper, worked at FANG, etc, but during the interview, seemed strangely out of touch with their own profession and what solutions actually work. Don’t ignore your gut! I’ve been burned by ignoring these signs. A sparkling resume is not a guarantee of success. Why is that? Maybe it’s because being at a successful company can blur the lines between things that went well due to your contributions and things that were sheer luck (or due to someone else’s contribution that was out of your line of sight). Think of it this way: you know the classic adage that some people have 10 years of experience repeating the same year? Unfortunately, it’s almost impossible to tell from someone’s resume how much experience they’ve actually acquired.
  2. Asking about mistakes they’ve made. I used to ask this question, but don’t any more. My logic behind asking this was well-meaning: no one gains real experience without having made and learned from some serious mistakes. This one really seems like it should produce high signal, but in practice I found that it always seems to result in more, rather than less confusion about a candidates’ qualities. I think the problem with this question is that explaining mistakes in a compelling, positive way is really, really hard, and so this line of questioning actually produces more signal about whether someone’s good at managing up.
  3. “Tell me about a big project where you were a major contributor.” I used to ask this all the time, and the intent behind it is correct: “I want to make sure they know how to find and complete impactful work that cuts across teams, so why not just directly ask?” It’s not that this question is bad – it’s just not-good: it gives very little signal. Why doesn’t it work? First of all, people are either completely unprepared for this one, or have over-prepared for it, complete with a totally synthetic, self-aggrandizing answer that frankly you have no way of verifying. This means at best, you’re getting signal on their level of preparation, not their actual experience – not a useful signal. Secondly, experience is more about triangulating and synthesizing learnings from multiple experiences in the context of a new problem than it is having done one very big thing. Not always true, but true enough that the signal you’ll get with this question isn’t worth the time.