Hackerrank was broken – but now it's harmful

59 点作者 lunarcave6 个月前

13 条评论

I've said it before and I'll say it again here - the best job interview I ever had for a software engineering position was about 10 years ago. I was shown a piece of code (directly from their codebase) which was about 50 lines long and we had a free wheeling technical conversation around it in which the interviewer asked me the following questions:- What does this code appear to do (both technically and in the context of the business domain of the company)- Are there are any red flags, edge cases, etc. that this code might not cover?- If I were to rewrite it in the using modern libraries/tooling/best practices, what might I have done differently? High-level - we're not looking for you to physically write syntactically correct code here.This is the same interview process that I now use on prospective applicants. In a single 30-60 minute dialogue it helps me answer questions such as:- Can they size up and summarize existing unfamiliar code bases? In enterprise the ability to deal with legacy code is often more common than writing new code.- Can they re-interpret old code using a modern lens? For example if we're discussing JS/TS (requires vs imports, callback hell vs async await, XMLHttpRequest vs fetch). For python (nested lists vs dataframes, simple numpy regression vs pytorch, etc)- Are they able to communicate and convey their thoughts in a coherent and logical manner?

评论 #42346458 未加载

评论 #42262467 未加载

naet6 个月前

The author says "whiteboard tests" are broken, but it seems like they're arguing that online coding assessments are broken, not in person interviews using an actual whiteboard.Doing an in person interview on a whiteboard sidesteps the AI issue. As someone who's done a large number of remote interviews, there are some clear signs that some candidates try to cheat online tech interviews. I wonder if the trend will fuel more of returns to the office, or at least a return to in-person interviewing for more companies.

评论 #42260610 未加载

评论 #42260211 未加载

评论 #42260310 未加载

paxys6 个月前

Is it already that time of the week?People were complaining about whiteboard coding interviews in the 1990s and they are complaining about it today. Meanwhile the tech industry has managed to hire millions of incredibly smart people and done pretty well for itself in that period. The interview process isn't going to change. There is no reason for it to change. Your options are to either suck it up and brush up on basic data structures and algorithms or interview at places that don't test for it. Just don't hold your breath waiting for the world to change for you.

评论 #42260791 未加载

评论 #42260680 未加载

评论 #42263289 未加载

clark0106 个月前

AI is likely to shape the future, gradually embedding itself across industries, fields, processes, tools, and systems. Similar to the introduction of electricity, where identifying practical applications was inevitable, trial and error will be an essential part of its evolution. While concerns about its impact are understandable, it's important to recognize that each generation often worries about the capabilities or standards of the next. However, such concerns are typically rooted in current frameworks of thought, which may not remain relevant in the future.

评论 #42260536 未加载

评论 #42260579 未加载

评论 #42260777 未加载

alephxyz6 个月前

>Since everyone started using AI, more candidates started clearing the first round with flying colors. The platforms had to recalibrate to let in their target percentage.Is there any proof or data on this? Not saying it's wrong but I'm curious how big the effect actually is.

评论 #42260869 未加载

acjohnson556 个月前

The dirty secret of hiring processes is that the main goal of the earliest stages is to get from a lot of applications to something less than a lot, not to screen for fit. This has long been true, and it is now super true.Every job posting gets flooded with hundreds of applicants, if not into the thousands. Most of those people are coming through the front door with no one to vouch for them and probably nothing on their resume that makes them a must-see candidate.Most managers and HR teams probably don't even explicitly think about it this way, but by pure pragmatism have evolved processes that act as flow control.The unacknowledged result is that the company will reject 90+% of applicants, regardless of fit, under the assumption that the filtering process will allow enough good people into the actual interview rounds that the team will be able to find someone they want. From this perspective, Hackerrank is not broken, it's doing exactly what is required of it by companies.I say all this because people who are in job search processes should frame the process accurately in their mind. It hopefully will help with not taking the process so personally or not getting so infuriated with it. It may also help you strategize how to find your way into the roles you want, if crushing these tests isn't your strong suit. People who are vouched for get to bypass all of this. The more confidently you are vouchced for by a trusted party, the more benefit of the doubt you get in the hiring process.One might ask is there a better way to do this? Probably so. But if it were easy, it would already exist.

评论 #42260489 未加载

评论 #42260688 未加载

Eumenes6 个月前

i screen candidates all the time (not involved in technical rounds) and notice chatgpt cheating all the time but its easy to spot because the candidates read off answers in mechanical ways right after a flurry of typing

LeftHandPath6 个月前

I recall having to implement A* to search a nxn character grid in my AI course a few years ago. It took me close to a full day to wrap my head around the concepts, get used to python (we usually worked in C++), and actually implement the algorithm. Nowadays, an LLM can spit out a working implementation in seconds.I think that's a big part of the issue with tests like Hackerrank - LLMs have been trained on a lot of the DSAs that those questions try to hit. Whereas, if you ask an LLM for a truly novel solution, it's much more likely to spit out a garbled mess. For example, earlier today, Google's search AI gave me this nonsense example of how to fix a dangling participle:> To correct a dangling participle, you can revise the sentence to give the dangling modifier a noun to modify. For example, you can change the sentence "Walking through the kitchen, the smoke alarm was going off" to "Speeding down the hallway, he saw the door come into view".LLMs have effectively made it impossible to test candidates for crystalline intelligence (e.g. remembering how to write specific DSAs quickly) remotely. Maybe the best solution would be to measure fluid intelligence instead, or front-load on personality/culture assessments and only rigorously assess coding ability in-person towards the end of the interview cycle.

评论 #42260450 未加载

jorblumesea6 个月前

Don't onsites or virtual onsites also have coding rounds? would be pretty easy to figure out who is using llm and similar tools.

评论 #42260455 未加载

nkrisc6 个月前

Seems a problem that will sort itself out if companies are in fact hiring under-qualified cheaters.

cute_boi6 个月前

please bring in person interview. I am sick and tired of cheaters. They rig every thing ....

snickell6 个月前

TL;DR: best case LLMs break algo-whiteboarding interviews so visibly we stop using them (they're already invisibly broken).Some engineers are genuinely better at using ChatGPT and friends to solve problems, and that's a skill worth including in evaluation of potential software engineers. I would advocate interviewers go with the flow, allow use of LLMs, but have candidates do it on the "shared screen" to see how they think - just like using a web browser in interviews.I've been a professional programmer since 25 BAI (25 years before AI). I see junior engineers who use it disastrously to create piles of messed up spaghetti code, and junior engineers who use it skillfully and judiciously to create good code and solve problems far above their years of experience. Both of them will use LLMs in their daily work. One of them will destroy the codebase, and the other will far exceed the expectations we had for a "2 yoe eng" only 2 years ago. I want to identify the latter, and filter out the ones that aren't good with ChatGPT and will use it to make messes.I want to identify experienced Senior Engineers who kick ass. They can do it with or without ChatGPT, I don't care which, and I don't want an interview format that doesn't let those who use ChatGPT show their true strength. I simply don't care how well you can recreate (and pretend to have rediscovered lol) PhD thesis algorithms with no internet access on memory alone: that is not a relevant skill in 2024.I won't cry if "allow LLMs but watch how they use it / think" doesn't scale to "no skin in the game" offline tests where interviewers try to get candidates to spend an hour without the company matching that hour with an hour of employee/interviewer time. In fact, if (and it probably will) this kills offline testing, I think that's a win for labour.What ChatGPT "cheating" reveals, imo, is how weak and artificial "whiteboarding" interview questions are: they claim to be problem solving tests but they're actually algorithm memorization tests. Interviewing without allowing use of LLMs is like requiring "no web browsing during interview": it artificially biases toward test-specific skills and away from real world strengths. If an LLM can straight up solve the problem, its not a very good match for skills required in 2024: its now a problem that SHOULD be solved via LLM.It turns out that LLMs are better at rote memorization than we are. Good! The "whiteboarding algos" interview technique was already a bad match to real world SWE skills except in relatively rare job roles, and its a good thing that its artificiality caused it now be irreparably broken by ChatGPT. Bye bye, won't miss you ;-)Most bog-standard software engineering work is boring things like good variable naming, effective use (and non-overuse) of encapsulation, designing a schema that won't make you hate life in 5 years, wise selection of dependencies, etc etc. Testing live with a candidate, watching them use ChatGPT, seeing how they think, this is what gets you a job-relevant-skill signal.

yieldcrv6 个月前

I think its time for the federal government to get involved, and it already has leverageIf your company benefits or plans to benefit from QSBS tax treatment, then evaluate engineers on things they’ll be doing in the company:the application development process, sprint ceremonies, asynchronous communication and toolingthey wont be implementing data structures often, and if they are reinventing the wheel they are wasting company time, if your company needs a more efficient data structures for its super scalable problem - it should trigger a process audit to be honest.

评论 #42260352 未加载

评论 #42260345 未加载