This is the third episode of HackerRank Radio, our podcast for engineering leaders interested in solving developers’ toughest problems today: Hiring the right developers. Hosted by Vivek Ravisankar (CEO & Cofounder, HackerRank). You can subscribe to us on iTunes and Google Play.
About Gayle Laakman McDowell, Author of Cracking the Code Interview
Anyone who is involved in the technical hiring process, whether as a hiring manager or candidate, has probably heard of Gayle Laakmann McDowell.
She's the author of Cracking the *interview books (Cracking the Coding Interview, Cracking the PM Interview, and Cracking the Tech Career).
Her background is in software development, with a BSE/MSE in Computer Science from the University of Pennsylvania and an MBA from the Wharton School. She previously worked as a software engineer at Google, Microsoft, and Apple.
In addition to running CareerCup and writing books, she consults with tech companies on their engineering hiring process and with startups to help them through acquisition interviews.
Listen to the episode, or scroll below to skim the transcript.
Jump ahead:
- Should interview problems be realistic?
- The rise of interview prep companies
- Advice for candidates who are out of practice
- Why there is no perfect interview process
- Do problem-solving questions predict future performance?
- Good versus bad algorithm problems
- The biggest thing that hurts people without CS degrees
- Making the hiring decision
- Gayle's biggest bug in production
Full transcript:
Vivek: Hello everyone, welcome to the HackerRank Radio. I'm Vivek, CEO, and co-founder of HackerRank. I'm excited to have Gayle Laakmann McDowell, founder, CEO of CareerCup.
There is a lot of debate on the coding interviews right now having algorithm problem-solving skills, and you don't necessarily use those data structures in your real-world coding. That you don't use a tie, you don't use a linked list. So why use that during the interview process? How do you reconcile the two?
Gayle: Yeah, I think this is rooted in an assumption that interviews should be realistic and that the more realistic interview is the better it is. And I challenge that a little bit. I think the point of an interview is to hire. Therefore, a good interview is one that is predictive and good candidate experience too. But the end goal of an interview is to predict who you want to hire, who will make a good employee. That is what you should be able to be focusing on.
If that happens to a realistic interview, fantastic. If you somehow create an interview that is nothing like the real world but is also predictive somehow, that would still be a good interview. It'd be strange for that to happen, but that will still be a good interview.
So with data structures and algorithms specifically, there are definitely times when companies go too far in the wrong direction and start asking candidates to implement like sales algorithm and all this other stuff that is not actually real world, it's also not predictive because all you're really testing is someone's knowledge. When you stick to basic computer science, when you stick to basic design and algorithm, they should be there if the interviewer is doing the right thing, to essentially assess the candidate's problem-solving skills, to simplify their intelligence. And problem-solving skills and intelligence that is actually a very, very important thing for a developer.
Vivek: The statement is hard to debate, problem-solving and intelligence is very important, but would you say that it's across the board in terms of irrespective of whatever seniority? Because there's also this whole thing of "Hey, if I am a new grad, I'm just used to solving all of these problems and going on a lot of these sites. But if I'm working at a company for over 10 years, I should still know the basic data structures but I've not solved those specific problems any time in the recent past." So do you think it is applicable across the board?
Gayle: Pretty much, yes. I think companies need to adjust their expectations. I'd argue for almost any job that requires analysis of any sort, whether it's programming or not, intelligence is pretty important. It's not everything. You don't hire somebody who's brilliant and has never written a line of code to be a programmer, but intelligence is really important and really useful cross development, across lots and lots and lots of jobs.
Where things go wrong where companies look at these questions as "Let me see if the candidate gets them right. Let me see if they know the answer." When companies do these interviews well, and often they do well, they need to ask questions that candidates haven't solved before that are challenging, that are challenging because it's actually a hard problem and not because it tests obscured knowledge and see how the candidate solves a hard problem.
Vivek: I see. So you're more concerned about their thought process?
Gayle: Yes.
Vivek: How they deconstruct the problem and how they solve it block by block?
Gayle: Yes, exactly. When I interview a candidate, I don't sit there with a timer and say, "Okay, my best 25 set of candidates are the ones who solved it in the shortest 25 set time." It's about how you move through that problem. Can you move through it? Can you make progress on it?
And there might be candidates who, you know, there's a bunch of steps of the problem, they get stuck on one piece and I help them through it, and then they do really well on the other pieces. It's a very qualitative process. A very qualitative analysis.
Vivek: Which companies do you think actually do a really good job at this?
Gayle: Facebook does a reasonably good job of this. And I think what Facebook does is they do a good job of the process as a whole of having a structured hiring process that's consistent and also creating...They have interview training. I don't know if it's the best interview training.
Most companies don't even have interviewer training and they also create a lot of transparency with the candidates. And so the candidates and the interviewers and the company are all pretty well in sync about here's the goals of the process, here what's going to happen. That is when the process used work pretty well.
A lot of companies have no interviewer training and you've interviewers who are doing a process that they don't know what to do, they don't know what the goals are. And that goes poorly.
Vivek: I don't know if you remember but we first spoke seven years back when we were called Interview Street, and we were doing mock interviews.
Gayle: I do remember.
Vivek: We were doing mock interviews which was our first business model, which actually didn't work. And I reached out to you because CareerCup has a section or maybe it still has a section of mock interviews. And I was just reaching out "Hey, does this product work for you?"
Now I'm seeing just a flurry of companies that are starting to do some version of mock interviews, and you talked about interview training, is this sort of a new trend? How do you even look at this? On one end it looks pretty good. "Okay, I'm going to prepare candidates for the interview." On the other end, it feels like, "I'm going to tell you all the questions that are being asked." How do you figure out the balance?
Gayle: What I do with the job is really split between the two. I do a lot of stuff that's what I'd call like candidate side, not a silly one-on-one coaching, my books, things like that. That's the kind of candidate side. Bigger talks on preparing for interviews, doing all that. And then there's the interviewer side. And you're right that there is a conflict there.
It works actually very, very well for both sides of the business that they're conflicting and not necessarily in a bad way. And what I mean by that is when I go into a company, one of my major points with them is, "Look, I'm doing all this candidate stuff. If I have a question or type of a question that I can prepare candidates for really, really effectively, the more effectively I can prepare a candidate do well on that, the less effective that actually is as an interview question."
An example of this is dynamic programming is something that candidates who have a very weak problem skill will probably never do well in those questions. Candidates who are just okay, I've seen over and over again, give them 10 of those problems and on the next one, they'll look like a great candidate because they're incredibly formulaic. That makes it not a very good interview question.
Vivek: Yeah. But then you should also get really good as an interviewer to figure that out, right?
Gayle: Yes, or just stop asking that question. Because the problem is you need to know if you're evaluating somebody there's a difference between how this candidate struggle with the problem when they've never done a problem like this before. If you have a candidate who's really prepared for those questions, you need to ask questions much more challenging to really look at their problem-solving skills because the formula is largely, say on these problems.
And so the better thing for an interviewer to do is don't ask those kinds of questions. Don't worry about this "Oh, gosh, can they prepare these questions or not." Just don't ask them. There are so many other problems out there that are going to not be as biased.
A lot of the other stuff that companies are doing to prepare candidates, whether that's companies who actually are focused on that or companies who have their own candidate prep side, a lot of that is stuff that is not about memorizing techniques. There is some pattern matching of kind of applying some techniques to other new problems. But that's true across programming anyway. There's a lot of pattern matching, and your ability to pattern match is actually kind of useful for the programmer.
But a lot of the stuff is stuff that's not about learning specific algorithms to apply to some new problem. It's about learning things like, hey, when you get a new problem, come up with an example. Come up with an example that's pretty large. A tiny one is going to actually not help you very much. It's about boosting candidate’s confidence, things like that.
In an ideal world, companies wouldn't need to do that. Preparation wouldn't help at all in an ideal world, but well, this is a real world and this compromises with any interview process.
Vivek: And what would you say for a candidate who's obviously, you know, you're not interviewing on a day to day basis clearly if you're working at a company, and then now you look for jobs and interview environment is sort of different. Sometimes use whiteboards and of course, it's stressful, it is timed. What do you think is sort of the leap that a candidate needs to make from "okay, I'm coding, I'm a very strong developer" to "Now I'm going to go ahead interview"? What are some of the things that the person needs to know?
Gayle: So, I think preparation is useful. Despite being in the business of candidate preparation, I don't tell candidates like, "Go spend six months or a year preparing." I think that's insane. What I tell candidates is "Spend a couple weeks doing all manner of interview preparation each week. Don't spend months and months doing it. That silly. Know your basic data structure and algorithms really well. And you don't want to feel insecure. You don't want to lack confidence with your understanding of a binary search. Be very, very, very comfortable with those topics."
It's better to be super comfortable with the basics than be pretty comfortable with a lot of things up to the advanced algorithms. Get really, really comfortable with basics - and that actually doesn't take that long - and then push yourself on harder problems. For someone who's weaker, that might be kind of medium difficulty problem. For someone who's already pretty strong, it might be very, very hard problems. But push yourself and get comfortable with the idea that you can solve a challenging problem.
Vivek: And are all these problems available in Cracking the Coding Interview?
Gayle: Not all interview questions. I actually specifically tell interviewers, "Do not ask questions from my book because as much as they would actually help me, I don't believe in being dishonest there." I would say there are sufficient problems. The goal of interviewing as a candidate is not to hope that you get asked a question you know how to solve. That actually tends to hurt candidates when that happens largely because they're not honest about it. The goal is to get better at solving new problems.
Vivek: You made an interesting point. I think maybe it was one of your Quora answers or somebody who was tweeting at you on "This is not a perfect process." Right?
Gayle: Yeah.
Vivek: I think it was an engineer who built HomeBlue, the Blue installation system and he got rejected by Google because he couldn't reverse a linked list or whatever - some hard code data structure algorithm. I think what you mentioned was there is no perfect interview process, and there's always is bias and flaws. If you want to expand more on that.
Gayle: Yeah. I think that's something that people need to be really aware of. There's this thing people do where they're like, "This process is broken for X reason." That's why we should all use this other process. Well, that other one is even worse. There is no perfect process. There's not even a perfect process for one company. There's like good enough, there is maybe as good as it gets, but it all has flaws.
You'll see people who say to take a couple of processes. You have people who say, "We should really be hiring people based on what they've done before. The past is the best prediction of success." Well, you're not going to be hiring somebody on what they've done, first of all, you're going to be hiring people on how they talk about what they've done. And there's quite a difference in this.
Vivek: So your presentation skills will probably supersede your actual coding skills?
Gayle: Yeah. And so, one of the things I do is I'll go into companies and I'll help prepare them for their acquisition interviews, and it gives me this really interesting data on how someone's interview performance matches with their actual job force at that company.
Vivek: Interesting.
Gayle: Very hard data to get otherwise. And there was this guy I interviewed where I was like trying to work with him on behavioral questions. "Tell me the hardest part. Tell me a challenge that's been hard." "Nothing has been challenging. It's not been easy." And it was impossible to get out of him anything challenging. And the reason why is he was actually an incredibly bright guy who'd worked on some reasonably hard things, and they're all pretty easy for him because he's extremely strong.
And you can easily work with that interview feeling like, "He's just then some code monkey. He hasn't really done anything challenging." And that totally would have been the wrong impression to get because that was not the case. Or you get people who do a really good job of making what they say sound more complex than it really is. You want to find people who will be good, who will be able to do cool things, not actually people who have.
A person who has been working doing boring IT at a big company in the Midwest and hadn't really gotten any hard problems, they're not going to have a whole lot of hard problems to talk about. Someone who is a very mediocre employee at Google or Facebook might have a lot harder cooler things to talk about. It doesn't make a better. They've just gotten the benefit of having harder problems. They might very well be worse.
Vivek: Do you think the problem-solving questions give you a good idea of their future performance?
Gayle: I think they give you a good idea of their intelligence. A specific type of intelligence but intelligence. And specifically what it does is if you'd conduct the interview process well, people who do well on those questions tend to actually be pretty bright. It doesn't mean that everyone who does poorly is actually not bright.
But your goal as a company isn't to hire everybody who's smart, everybody who'll be good. Your goal is to hire enough people who are good and as few people as possible off the balance. It's okay in other words to have some of those negatives. You just want to limit the [inaudible 00:13:05] for the most part. So, yeah, I think the process does a decent job of hiring people who are pretty smart.
Now you want to add into stuff like don't hire someone who's bright and kind of a jerk. And you can actually get a pretty good rid of that through these questions. It doesn't look at all at work ethic, really to focus things like that, but it's actually really, really hard to get that data.
Vivek: Now, one key thing is probably over a sustained period of time, right?
Gayle: Yes.
Vivek: It's hard to get in an hour interview.
Gayle: Work ethic also isn't a constant thing. You'll have some who work super hard and gets put in environment with a manager they don't like, and all of a sudden they're trying to check out of there as soon as possible.
Vivek: Yeah. It's a function of environment, colleagues, manager, problems that you are working on. It's probably a lot of other factors. So this one really what you're trying to say, if I can, is this assess the basic minimum intelligence for a developer that you absolutely need to continue the interview process or to even feel good about, "Okay, can I hire this person or not?"
Gayle: Yeah. I'd say more than basic minimum intelligence, but I think it does a reasonable job of looking at intelligence on the whole. Not the person who gets very good versus very, very good. A lot of that's kind of random, but if you do it the right way, it does a reasonably good job of ensuring that the people you hire are smart.
And when you hire smart people, even if they don't know the right technology is for that, they can learn them. They can think about problems deeply, they can juggle lots of different components. You're going to want to add on into this some behavioral stuff, you're going to add into it some depending on the amount of experience things like system design or something like that. It should never be the entirety of a company's process.
Vivek: Yeah, yeah. The interview questions are super important. I've seen companies do a question which is based on a trick or which is based on a math formula that you get and it is super, super important and even you touched upon that. You should really focus on figuring out how this person deconstructs the problem and it's okay to give hints and it's okay if the person actually solves a problem with a hint. Don't ding him or her for that.
How would you phrase or frame, here are the attributes of a really good question and here's what you shouldn't be doing?
Gayle: So for the algorithmic problem solving, the design and algorithm problem, there's a few major things. First thing is the question should be challenging not in a [inaudible 00:15:29] not going to ask about the types of algorithm. That's not challenging. That's if anything came to know a lot about algorithm. That's not actually challenging.
Vivek: But you're relying on the fact that, well, you know a fact or not?
Gayle: Exactly. And if you need candidates that know types of algorithm, if it's that important to you and then you're like, "No, we really need advanced graph knowledge," what I find is that most companies who make that argument, you say, "Okay, so if it's so important to you, you're also training your people on it in the first week or two. Well, no." Well, if it's that important to you that you're going to say types of algorithm is fair game as interview question, then you should be training on that. If you're not willing to do that, I don't think it's that important to you actually.
So the other thing is challenging questions. Questions are challenging because they're actually hard and not because they require obscure knowledge. Having questions that have multiple like hurdles, that can be multiple parts to the problem, so it can be a bunch of fault questions. It can also be just a problem that has multiple optimizations to get the most [inaudible 00:16:27].
As much as possible those hurdles should be things that are logically to disposal no tricks, no cool math, things whatever. A good rule of thumb there is, you're kind of struggling and you're trying to give them a hint and you have a part-time figure out how to give them a hint without giving everything away, that's probably not a good question. You've just made things very, very arbitrary and whether they get one piece.
The knowledge that you do require, make sure that you really do require that knowledge. You really do need that on the job. And Ideally, that should be a list that is actually communicated both to the candidates and to interviewers. So if candidates are going to be asked questions about binary search trees, just give them a heads up about that, and give a list to interviewers about what is fair game.
For most programming jobs, I would say binary search trees how they work, things like that, that's fair game. How to balance them, that's cool algorithm knowledge that you probably don't need and you can probably train people on if you really do need that.
Vivek: I think this is very helpful because you've almost deconstructed into don't just blame algorithm problems, blame the bad algorithm problem-solving. I mean, there's a general misconception in the industry and ...this gets on Hacker News is a big framework that's going on, that process that we're good and all of those things. But I'm also curious to know if problem solving in algorithm and fundamentals, which I believe is super, super important for any developer, a lot of nontraditional developers, non-traditional in the sense you didn't go to college and get a CS degree, but you went to a bootcamp or you were self-taught, well, they don't really focus on this.
In fact, a lot of bootcamps actually tell you, "You know what? I'm going to make you a really strong React developer in 90 days." And you just figure out how to build an app and then you go ahead and do it and then you go on to interview process and it doesn't work really well. What would be your advice to them? In general, what's your view on this pool is just going to get larger and larger because people just get along on their own? How would do you think about approaching this?
Gayle: So, I have prepped a bunch of people who don't have CS degrees. Some of those are people from bootcamps, some of those are people who are self-taught. To take a specific example, there was a friend of mine who got laid off from his job at a...you know, if you were to rank startups, this was not a good startup. The company closed down for arguably kind of shady business. It was not a good company.
When I started prepping him, I was like, "You obviously know hash tables?" "What? Never heard of them." I'm like, "Oh, my gosh. This is like very, very, very basic computer science. You cannot walk into these interviews not knowing this."
His starting point was not knowing anything about computer science. He didn't know computer science, data structures, algorithms, and his interview with Facebook was in three weeks, and he got offer. And he actually a very, very well there and he's still there doing very, very well because fundamentally he's a very smart person. No, he never learned computer science, he was totally self-taught, but he's a very smart person. And that is the biggest piece of what they're assessing. He had to learn some things, but it doesn't take that long to learn hash tables, linked list, binary search trees, basic graph stuff. I mean, it really doesn't take that long.
The biggest thing that hurts people who don't have CS degrees is really just insecurity. This guy he's reasonably confident, so he walked into the interview believing in himself.
What I see with a lot of peo
ple who are from bootcamps or who are self-taught is just insecurity. As soon as you ask them a question about "Okay, given a binary search tree, do such and such," they're like, "Oh, wait. Hold on." I don't use binary search trees ever. I know what they are, but I've never worked with them in the real world." Neither has anybody else. And they just count themselves out. For the most part, they usually have the knowledge that they need to have. And if they don't, that does not take long to acquire at all.
Vivek: And do you think there's a bias from interviewers, like the posture of candidates or do you think it's important?
Gayle: When I do interview or training with companies, I specifically tell them "Do not take into account confidence. You are not looking for candidates who are confident." But if the candidate is unwilling to proceed with the problem, if they give up, if they don't try as hard. If you want to find the fastest runner in a race, it's not all about speed. If you did want to find the fastest runner run a race and somebody wasn't doing their best, they're going to get to finish line slowly and you won't know how good they could have been had they really done their best.
And that's what happens here is that candidates they're either exclusively give up or they just don't try as hard. They don't believe that they can do well and so they don't. It's kind of a self-fulfilling prophecy.
Vivek: And calibration should also be heard, right, whenever you do the interview or training for companies like you mentioned because people are coming from different types of companies, different sizes, each company has their own bar, and how do you figure out, this is our bar and here are the kinds of questions that we're going to ask and here are the expect answers. That is going to be a hard challenge. Or do you think it's all [inaudible 00:21:23] versus Silicon Valley where everybody is just asking the same questions?
Gayle: You don't need to calibrate across questions. When I work with companies often they'll say, "Can you give me a description of what a good answer this problem looks like." And I refuse because you can't make that comparison. You can't say, "All candidates must get to this point and they get this score." Because it is much more qualitative.
If I asked you a question and I'm going to be giving you help and hints and guidance and this and that, then you can't make apples to apple comparisons there. Each interviewer really just need to be self-calibrated. And this is what we did at Google was I would give a score and I kind of have a feel for I give a 3 if it's a candidate who was very good, 2.7 if they were pretty good but I'm not sure if I would hire them, but maybe.
And I have this gut feel my 3.0 and your 3.0 might be different things. But when the hiring committee is looking at my 3.0, they see, "Oh, 3.0, that's a pretty good score from Gayle. Usually, she doesn't give a whole lot of 3.0's and above." So each person does not need to be calibrated across the company. An outside force can look at that and get a feel for what their scores mean.
Vivek: And are seeing more of these hiring committees come about in different companies? I know Google was a big promoter for that?
Gayle: You see a lot of the bigger companies there's different models companies can take. Most of them don't think about this way but the other model that I see actually a ton of companies trying to take is bar raisers, which derived originally from Amazon. That gets a lot of the advantages of a hiring committee without a lot of the weight of it. And that still has some of the value of the outside person looking in. That gives a lot the same value and allows a lot of the same calibration.
Vivek: And what are your thoughts on making a decision? Like consensus-based - hiring committee says yes, bar raisers says yes, and then it's a go. How do you think about all of these different models? Like what's the right frame to think about before making an offer?
Gayle: The way I teach companies is it's not about a consensus. It's not about you have five interviewers and all five need to say yes individually. That's not really the right way to think about. People's yes's can mean different things.
If I interview a candidate and they showed very strong problem-solving skills but I'm not sure, when I probed a little bit at their skills in system design they seemed pretty weak, but I only probed a little bit on it, so I'm giving a no. But my next two interviewers now give very strong really program system design and they give very strong yeses, is my nose still a no? Well, I was giving a no because of one thing that other people just feels not that concerned.
So I teach companies instead of say, "We want somebody who is not a jerk. We're all on board pretty much by and large that the person can work with other people. Okay, are they smart? Do we generally believe that?" Look at it attribute by attribute of what you're looking for rather than a thumbs up thumbs down from everybody.
Vivek: So there's a question that we ask all our guests. What's the first thing that you shipped to production?
Gayle: So I interned at Microsoft for three summers and Apple for my fourth summer before graduating college. And actually my Apple work got shipped before because Apple just moved faster and deadlines whatever. My apple work actually got shipped before my earlier stuff, which was kind of funny.
But I guess the very first code that got shipped was software I wrote my very first week at Apple. I was in the iChat team which was there like a messenger. I think now it's been folded into messenger and lots of other things. But I was in college and had bunch of buddies and buddy list and multiple accounts in this and that, and one of the things I found annoying working on iChat was every time I had to switch accounts, it was like a whole process of signing off and doing all these different things.
And so I planned a little thing just for myself to allow me to switch...like a little menu thing that allowed me to switch accounts basically just for me. And they're like, "It seems like, yeah, let's go do this thing." "lt only adds value for people who have it, who have multiple accounts." "Great." It was my first week, whatever. I didn't know Objective-C at all. And then it wound up on the like packaging messaging material for new features and like OS whatever version was that. It was like switching accounts, like, okay guys.
Vivek: Did they put your name as well there?
Gayle: No, I know. They didn't.
Vivek: That's pretty cool. That's a big one.
Gayle: Yeah. It was cool. I mean, it was a very, very easy thing to write, but it was kind of cool that it got listed on the newest features.
Vivek: Yeah, yeah, that's pretty cool. Thank you so much for the listeners. If you have specific questions or topics, tweet to @hackerrank. Thank you so much.