In our most recent webinar, How HackerRank is Leading AI-Powered Hiring, Principal Product Manager Ankit Arya and Senior Director of Product Marketing Danielle Bechtel gave customers a first look at new and upcoming products that let companies bring AI into their hiring process—on their own terms.

While there’s no substitute for watching the webinar on demand, here’s a taste of what went down:

3 developments in HackerRank AI

1 – AI-Powered Plagiarism Detection is live

HackerRank’s industry-first AI-powered plagiarism detection system is live and available to all HackerRank customers. By analyzing dozens of unique signals, our new plagiarism detection model detects suspicious activity with far greater reliability and fewer false positives than industry standard methods, like MOSS code similarity. 

2 – AI is about to make hiring teams’ lives easier

Several upcoming platform features promise to make hiring teams’ lives a bit easier. For example, AI will soon be able to review candidate code quality across several metrics such as efficiency and modularity, and provide a rationale for its analysis. AI will also be able to help members of the interview team provide more accurate interview summaries faster, using transcripts to build a first draft that can be refined before submission. 

3 – AI is coming to the assessment experience

HackerRank customers are fairly divided on AI’s role in assessments. Some want—or need—to keep AI at arm’s length. Others want to use it, and want to see how their candidates use it. To allow companies to embrace AI on their own terms, we’re building AI assistance into the assessment experience. Furthermore, the AI assistance will be highly customizable, from limited AI that can onboard a candidate to a codebase, to fully open AI that can engage in pair programming and code generation. 

At the end of the discussion, we held a live Q&A to chat through questions from the audience. Here are five of the top questions we heard—and how we’re thinking about them in response. 

Top 5 questions from hiring teams

The following responses are from the perspective of Ankit Arya, our principal product manager. His answers have been edited for length and clarity. 

1 – Is ChatGPT ready for primetime code complexity?

Base ChatGPT, the GPT-3.5 Turbo model, is not as good for programming. But GPT-4, Bard, and Anthropic’s models are getting to a place where they’re real coding helpers as you’re building software. 

Teams still need human creativity and developers who understand code, but AI can help take care of some of the more tedious tasks. For example, if you wrote a piece of a function and you want it to do error handling, you can have ChatGPT manage that for you. Of course you still need to review it, because you’re ultimately responsible for deploying it in production. But it can be a great assistant and enhance productivity.

2 – Can you talk more about plagiarism detection and 93% reliability? How do you check false positives? How do you even get that information? And has any other third party validated these claims?

The system has been in limited availability and we’ve run thousands of tests to make sure the system is performing at the level that we’re claiming. We’re also looking at feedback  from customers who’ve been using this product, and that feedback’s been really amazing. So that’s really where we are coming from when we define that internal benchmark. 

We’ve also been audited by an external third party, because it does come under the purview of the NYC law. We’ve gone through the audit process, so the system is ready for you to use. 

3 – HackerRank’s plagiarism detection system will get better over time because it’s built on AI. Can you talk more about that?

These systems are built with training data. Imagine when you’re a kid. How do you learn things? Someone shows you an image of an apple and tells you it’s an apple. Teachers give you a lot of examples and a label, and you start building associations, so you can recognize an apple.

This is how AI models learn, as well. Only they’re not as good at it as humans. We just need to see a thing one or two times, and we’ve got it. I could show you any apple, and you’ll identify it with very high accuracy. AI systems need a lot more data. So in this case, they would need a lot more images to make an accurate, reliable prediction. 

When we say the system gets better over time, this is what we mean. The more customers use it, the more feedback they provide, the more training data the system can ingest, further increasing its accuracy.

4 – Lots of people are interested in the interview assistant. What does that look like in the long term? Is this something you see integrating into an ATS?

Yes. Over the long term, we want to get to where the interview assistant does most of the work, and where we’re delivering it to you in your ATS. We don’t want AI making decisions, so imagine this more like AI doing 80-90% of the work for you, compiling the summary that you’d have to spend an hour doing. Now you would be spending 10 minutes reviewing it, making any changes, and then submitting it. 

But we absolutely imagine the system becoming way more integrated into the workflow than it is now, depending on what ATS you’re using.

5 – How does AI in advanced plagiarism handle copy/paste? Are there any plans to disable that functionality altogether?

No, there are no plans to disable copy/paste. I don’t think that’s something we’d ever want to do. To bring a little more clarity, you can’t copy questions. So when you talk about copy/paste, it’s really in the editor window. We provide a proctoring feature that’s essentially copy/paste tracking. And just because someone pasted, doesn’t mean they plagiarized. It’s just one of the signals the model considers. 

For example, someone might be solving a program question, but forgot how to insert a key in a Python dictionary. Simple, basic things just become signals into the model. What we’re really looking for are large patterns of cheating behavior. Is the full solution being pasted in? Or large chunks of code? So whether copy/paste triggers a plagiarism flag depends on the context of how much was copy/pasted and what was copy/pasted. 

Get the full story

These questions only scratch the surface. Be sure to watch the full webinar to take in the full Q&A session and get more context around HackerRank’s new and upcoming AI products.

And if you want to be among the first to gain access to our future AI releases, be sure to sign up for the HackerRank AI waitlist at hackerrank.com/ai.