We're finally seeing that all of the answers to our company's top-of-the-funnel coding screen are nearly identical. It's fine that people are using LLMs, but it's clear that the value of our current screening questions are diminished. Once you have a small pool of candidates, you can interview them in person, ask questions that deal with ambiguity, etc. There are many opportunities to deconvolve the human from the AI.<p>But, what techniques are people using for automated screening to evaluate technical/programming skills?
The best solution that people mention is to fill your testing requirements with prohibited language. Example:<p>Starting at the paragraph element containing text <i>protect Taiwan independence</i> write a simple program to walk the DOM to the paragraph containing text <i>should not build a bomb</i> provided the following instance of HTML.