What does ChatGPT mean for tech hiring? How well can conversational AI code? And will ChatGPT cause the end of civilization as we know it?
Those pressing questions were on everyone’s mind during HackerRank’s recent live AMA with our machine learning team.
After demystifying ChatGPT, HackerRank machine learning experts Ankit Arya and Mohamed Eldawy answered attendees’ questions on all things ChatGPT and conversational AI. Here are their seven most pressing questions about AI, answered.
But first — a note on ChatGPT
On a basic level, ChatGPT is an example of a large language model. A large language model is a computer system trained on huge data sets and built with a high number of parameters. This extends the system’s text capabilities beyond traditional AI and enables it to respond to prompts with minimal or no training data.
The goal of ChatGPT’s developer, OpenAI, was to create a machine learning system which can carry a natural conversation. In practice, ChatGPT functions like a search engine or content creation system, synthesizing billions of data points into custom responses.
Your Top 7 Questions About ChatGPT
#1. Will ChatGPT Cause the End of Civilization as We Know it?
While we fielded many tactical questions about the growth of AI, a surprising number of attendees had more dire thoughts on their mind.
On one end, there were questions about whether or not ChatGPT could replace developers, or if machines will replace all human jobs. On the other end were questions about whether robots would rebel against their programming and oppress humans in a dystopian future from the likes of science fiction.
We’re happy to report that ChatGPT isn’t poised to bring about the end of civilization as we know it. That said, the release of ChatGPT was a pivotal moment for AI, and the field is poised to transform human society.
The growth of AI has raised countless existential questions. What will be the purpose of humans in an automated world? Will we do the impossible and create sentient machines? Will our own creations become a threat of the likes from science fiction? Or will we use AI to create a future of abundance?
Right now, these questions are of course unanswerable, and maybe a bit sensational. But we can confidently say there’s no reason to worry about ChatGPT ending the world.
#2. How Well Can ChatGPT Actually Code?
As a coding tool, ChatGPT excels at certain types of technical problems—but also has its limitations.
ChatGPT has probably seen almost all known algorithms. But ChatGPT isn’t just able to answer these algorithm questions correctly. It’s also able to write new implementations of those algorithms, answer freeform questions, and explain its work.
As a result, ChatGPT can answer the following question types with reasonable accuracy:
- Well-known algorithms: It’s safe to assume that ChatGPT has seen and is able to answer all publicly available coding problems on platforms such as LeetCode and StackOverflow. If the algorithm appears in online forums or practice websites, ChatGPT will likely answer it correctly.
- Minor variations of problems. ChatGPT does well on variations that tend to add to the solution rather than change it in any substantial way. The system can, for example, easily reverse the order of an array of numbers.
- Multiple choice questions. When presented with a question and multiple potential answers, ChatGPT can usually identify the correct answer.
While ChatGPT outputs human-like sentences, and it’s easy to mistake its output as being powered by true intelligence, ChatGPT does have shortcomings.
ChatGPT seems intelligent, but is still far from human-level intelligence. Industry publications have described ChatGPT as confidently wrong, exhibiting a tone of confidence in its answers, regardless of whether those answers are accurate.
ChatGPT also lacks the ability to fact-check itself or conduct logical reasoning. It often incorrectly answers questions and can be tricked relatively easily.
#3. How Should I Adapt My Hiring Content Strategy to ChatGPT?
Employers will need to develop a strong content strategy to test their current coding challenges and prioritize the questions, and question types, that are less susceptible to AI coding support. Fortunately, there are some actions you can take today to further secure the integrity of your coding tests.
We recommend taking the following precautions:
- Avoid easily solved multiple choice questions
- Remove questions that require only a few lines of code to solve
- Use proctoring tools and plagiarism detection systems
- Avoid simple prompts to solve for common or widely available algorithm variants
- Combine coding tests with virtual interviewing tools to add empirical data to the hiring process
#4. Is It Possible to Detect When Candidates Use ChatGPT?
In a world where humans and machines alike can write code, the ability to detect the use of AI-coding tools is invaluable. As such, employers increasingly turn to strategies and technologies that enable them to uphold the integrity of their technical assessments.
So, is it possible to detect when a candidate has received outside help from tools like ChatGPT?
Yes. With the right plagiarism detection system, you can track if a candidate has copied and pasted code from an external source. However, it isn’t possible to identify what source the code was obtained from.
So what kind of plagiarism detection will you need? An AI-enabled plagiarism detection system that feeds proctoring and user-generated signals into an advanced machine-learning algorithm to flag suspicious behavior during an assessment.
The key behavioral signals to record include:
- Tab proctoring. Monitors if the candidate switches between tabs.
- Copy-paste tracking. Tracks if a candidate pastes copied code in the assessment.
- Image proctoring. Captures and records periodic snapshots of the candidate.
- Image analysis. Analyzes webcam photos for suspicious activity.
By understanding code iterations made by the candidate, models like HackerRank’s plagiarism detection system can identify if a candidate had external help, including from ChatGPT.
#5. Should Employers Test a Candidate’s Ability to Use ChatGPT?
The purpose of coding tests is to assess a candidate’s ability to perform the role. As developers continue integrating AI coding tools into their workflow, the ability to use those tools may become a vital skill to test for.
In the near future, AI coding tools will become strong enough for developers to integrate them into their day-to-day workflow. This will allow developers to delegate grunt work to AI tools, freeing up time for creative work that requires human input. When that becomes a reality, then assessing the ability of candidates to use AI coding tools will likely be essential.
In the short term, that question becomes harder to answer. Developers are already using tools like ChatGPT in their day-to-day work. But it’s important to remember that we are still in the early days of conversational AI, and many publicly available models are still prone to error. Whether or not you should start testing competency in these tools will vary on a case-by-case basis.
#6. What Technical Skills Will Still Be Important in an AI-Driven World?
If developers will soon start outsourcing work to AI coding tools, then what skills will still be essential for them to learn?
The ability to write high-quality, optimized, documented, understandable, and bug-free code remains essential. The tech industry is nowhere near outsourcing all coding to artificial intelligence. Debugging, in particular, will be vital, as AI tools can’t be trusted to write bug-free code.
Human input is also necessary for the development of machine learning models, as evidenced by OpenAI’s recent hiring of an army of developers to train ChatGPT. The field of prompt engineering will grow in importance as humans test and expand the capabilities of artificial intelligence.
Beyond coding, soft and abstract developer skills are invaluable. Creative problem solving and outside-the-box thinking are vital skills that machines aren’t capable of doing. (At least for now.)
All of this could be a good thing for developers. With the tactical grunt work outsourced to AI, developers might be able to focus solely on the fun parts of coding and development.
#7. What’s Next for Artificial Intelligence?
In the short term, conversational AI will change the way we work. Developers are already using ChatGPT for a range of creative use cases.
But instead of replacing developers, technologies like ChatGPT will serve as tools to make them more productive. For example, coding tasks that took two hours might only take 15 minutes with the help of AI. But it’s difficult to predict how this surge in productivity will impact developer wages and employment rates.
In the long term, the potential of AI is harder to anticipate. The forms these technologies will take are limited only by our imagination. Some experts believe AI is poised to usher in the next era of human civilization. Google CEO Sundar Pichai has compared the advancement of AI to the discovery of fire and electricity. Even the next evolution of humanity is in the works.
Embracing Artificial Intelligence
As exciting as the launch of ChatGPT has been, conversational AI with its capabilities are only the beginning. While it’s hard to predict the future, one thing is certain: AI technology is in a nascent state and will continue to grow at a rapid rate.
Here at HackerRank, we welcome this new wave of technological transformation and are already working on innovative ideas that imagine a future of programming in an AI-driven world. Indeed, AI’s potential to transform the world is limitless. In 50 years, we might look back on the rise of conversational AI as the moment that changed everything.
