Yesterday, we did two basic-screen technical interviews where both candidates appeared to use LLMs to generate nearly their entire answers.<p>We do this quick screen after a 30-min behavioral interview to make sure that candidates can generally operate at the skill level they claim on their resumes.<p>In the past, we've been shocked by the number of people who will talk a big game, but have really rudimentary programming skills when the rubber meets the road.<p>The questions are:<p>1. FizzBuzz<p>2. Generate the first 20 rows of Pascal's Triangle<p>3. Drop all non-prime integers from a pre-defined set of 2-to-N<p>The first candidate we didn't <i>totally</i> suspect, until the second candidate provided nearly letter-for-letter the same answers (same variable names, function names, etc.).<p>After the interviews, we popped our deck into ChatGPT + Claude and it output exactly what these two candidates had provided.<p>Last week, a third candidate sent us clearly ChatGPT'd code as an example of some of his work.<p>I'm unsure what to do here, so I come to you HN to ask, what you have done to guard against the use of LLMs in remote technical interviews? Thanks!<p>Bonus: The nail in the coffin was when the second candidate immediately clocked the last question as leveraging the Sieve of Eratosthenes. Previously, he'd shown us a pretty impressive portfolio. When asked how he knew the Sieve of Eratosthenes off the top of his head, he claimed he had used it in one of his commercial portfolio projects but couldn't explain how.
Ask them to share their screen, walk you through their code and explain their solution live. Every once in a while ask why they did it in that way and what alternative approaches there could be. Then change the problem a little and ask them to modify their code, again live. Generally, try to go deep and ask follow-up "why" or "how" questions. Those who "don't remember" or only offer vague and shallow answers are likely to have cheated with LLMs (or are just poor candidates).
While I'm personally not keen on LLMs, I admit I do have the co-pilot extension installed in Visual Studio and have been pleasantly surprised at how tab-completion is working. It seems effective for small blocks of code.<p>So, remembering that I not really a fan when I ask this... but why do you care if a candidate uses an LLM or Google as part of your interview? Do you care if they use an IDE or if they use a code completion plugin? In the end, do you not really want to evaluate if the candidate can produce good clean code?<p>If you feel like an LLM is too big a crutch, is that because what you wanted to test was memorization of a framework or a test of thought and workflow strategies?<p>To quote a resource I'm also not keen on but understand why it exists, does your concern about chatGPT during interviews actually point out an XY problem?
Sieve of Erastothenes is often introduced alongside prime numbers in elementary school math class and it has a funny/memorable name - it's not that weird for someone to know it. It is weird to lie about having a practical use case for it since running it to the point of a cryptographically useful prime length is infeasible and it requires O(n) memory.<p>It seems vanishingly unlikely that this type of question can provide any signal any more outside an in person interview. The incentives are just too strong for candidates and the tools are too good.
It's a great question.<p>What we are doing is to ask the candidate to have hands visible to the camera for the interview. But some systems are working with voice only and this will not be working in those cases.<p>Probably the best way would be to have the ChatGPT answer beforehand and confront it with what the candidate is saying?
Ask questions that involve trade-offs, be they design or performance related, and explicitly tell candidates that you're benchmarking against ChatGPT and expect something beyond what an LLM would give, i.e. you're looking more for creative/critical thinking than mere correctness.
This would be like asking a carpenter to build you something without a hammer.
At this point we need to realize LLM is a tool like everything else. Maybe give them a LLM challenge - how to do X. What is your prompt. Why?
Just ask the candidate whether if they have contributed to relevant large open-source projects in the language you are looking for. If not, then give them a hard leetcode question.<p>All you have to do for this hard Leetcode puzzle is to ask the candidate to complete it in Rust.<p>ChatGPT will struggle to help the candidate as it generates garbage.<p>After they have completed it then for the second technical interview, question the candidate around how they came up with the solution step by step to show if they really understand both the language and algorithm used to solve the puzzle.<p>This rigorously filters out 95% of frauds and impostors whilst targeting the best and brightest (really).<p>Job done.