Welcome to the first part of a 4-part series on screening senior engineers.
Step 1. Before You Do Anything (This piece)
Step 2. Designing Impactful Challenges
Step 2a. The Challenge Checklist
Step 3. Setting Expectations, Warming Your Candidates
Step 4. Calibrating After the Screening
It’s never binary. The nuances of evaluating engineers deepen with their experience. Some seasoned engineers have created a radical new piece of software without ever having balanced a tree in their career. Others have gained valuable leadership experience as their coding skills have begun to rust. A few might be 10X engineers but haven’t figure out how to work in a team. When faced with a flood of candidates for a senior individual contributor position, it’s really difficult to gauge their skills. Seniority itself is hard to define. It’s a multitude of elements, including deep technical skills, leadership qualities and maturity. It’s hard to embody the latter soft skills on a resume. Technical skills, on the other hand, can be assessed using programming interview questions.
Traditionally, companies don’t ask coding interview questions until after they’ve screened their resumes or LinkedIn profiles. And they usually ask them to code manually on whiteboards. By then, you’ve already sunk in countless hours of manpower from recruiters and engineering managers. But there are many famous studies proving that resumes are a poor indicator of success (more on this later). It’s just how people have always screened people. When we want to hire great engineers, we ask candidates to prove their skills first and foremost through online programming interview questions. For the purpose of this piece, let’s refer to these questions as “code challenges.”
By implementing code challenges as an early step in the evaluation process, you can partially qualify candidates and pinpoint the candidates on whom you should spend more time evaluating emotional intelligence and other characteristics. After analyzing thousands of code submissions, and interviewing several directors of engineering and consultants, we created this guide to design and deliver the most optimal code challenges for your potential senior engineering individual contributors.
But before you do anything, realize: “The questions don’t really matter. What matters first is a clear understanding of what you need,” says Soham Mehta, CEO of Interview Kickstart. Optimize your time by spending 80% to figure out what you need and 20% to craft the challenges.
The Questions Don’t Matter Yet
So, how do you know what you need? The two most common values of great engineers are intelligence and technical knowledge. Algorithm challenges are best used to test the former and knowledge or tool-based challenges are great for the latter.
But what most managers don’t realize is it’s better to keep these values mutually exclusive. Testing for both at the same time puts far too many constraints, limiting your pool of talent. So, how do you know what you need? It largely depends on the size of your company:
- Finding Smart Senior Engineers
Large companies, like Google and Facebook, are infamous for their algorithm and data structure challenges. When you have extensive teams, it’s better to hire for intelligence than knowledge. You likely have enough engineers to teach newcomers your tech stack. And smart people will be able to learn specific technologies pretty quickly anyway.
This is supported by an 85-year-long organizational research study that concludes: “cognitive ability (or intelligence) tests are the best predictor of success across fields.” Algorithm challenges, including ones that ask you to balance trees, are the programming equivalent of “cognitive ability tests” because they test for reasoning, problem solving and critical thinking skill.
What’s even more interesting is that the screening credentials commonly found on resumes—like education, age (or experience) and academic achievement — ranked among the worst predictors of success.
Years of hosting code challenges on behalf of high-growth companies reveal similar findings.
“Of the thousands of code challenges that companies like Amazon, VMware and Evernote use, algorithm challenges have produced the most successful candidates,” says Dr. Heraldo Memelli, lead content manager at HackerRank.
Some might argue that fundamental challenges are better geared toward junior engineers since they’re unrelated to experience. I mean, why would a senior engineer need to balance trees at this stage of her career, anyway? But McDowell, Mehta and just about every engineering manager we know stresses that testing senior engineers who value fundamentals is crucial for two reasons.
One, if algorithm challenges are the standard measure for intelligence for junior engineers on your team, your bar has to be consistent for senior engineers as well. Otherwise, you’ll hire a mix of smart junior engineers and, well, not-so-smart senior engineers.
Second, David Taylor, head of Sonos engineering, loves asking senior engineers data structure and algorithm questions because “if candidates shy away from that, you have a large red flag.” It’s a way to filter out folks who feel like they’re too good to roll up their sleeves and revisit the basics. Engineers should be “continually interested in keeping themselves up to speed, in revising the fundamentals and taking on intriguing programming problems,” Mehta says. “Those are the people I want to work with.”
Granted, the burden shouldn’t be entirely on the candidates. The onus is also on companies to properly prepare senior engineering candidates to ace these fundamental challenges. We’ll cover more on this in step 2.
2. Finding Knowledgeable Talent
Smaller companies—ones that can’t afford to wait for senior engineers to learn the tech stack they need—are most likely to focus on knowledge over intelligence. By crafting challenges that reveal technical knowledge in specific tools, you’ll attract engineers who can start hacking a new application on day one. So, it’s better to focus on knowledge-specific challenges if you’re a stealthy startup.
Welcome to the first part of a 4-part series on screening senior engineers.
Step 1. Before You Do Anything (This piece)
Step 2. Designing Impactful Challenges
2a. The Challenge Checklist
Step 3. Setting Expectations, Warming Your Candidates
Step 4. Calibrating After the Screening
Please subscribe to our blog to get a quick note when we occasionally write thoughtful, long form posts.