|
Going through the interview process now for the first time in half a decade, and while I already would have said five years ago that I preferred work samples, that opinion is only growing stronger as I go through the process again. Leetcode style interviews feel so stupid and divorced from the reality of the job, especially the “one weird trick” kind where you’re expected to discern the best possible solution to a problem on the spot and in a pressurized situation. The reality of the job is usually that when you are under time pressure, a suboptimal solution that does the job is fine (to be fixed later), while if you’re working on something you know is important (hot loop code, core data structures) you have time to think about it and get it right. A leetcode interview doesn’t select for either of those things: it selects for people who have time to grind leetcode problems. On the other hand, a work sample is realistic: a timeboxed task that you can approach in a familiar environment, without an interviewer breathing down your neck, expecting you to think out loud, railroading you to their preferred solution, etc. As an interviewer, I always pushed for either work samples, which I quite liked, or coding interviews where we very explicitly said that we just want a working solution, which we could then talk through and look for potential improvements. We also explicitly viewed the coding interviews as being low signal, and tried to make the bar for passing low, so we could get candidates to higher signal conversations. I do think the work sample route is a little more difficult in the LLM era, in that you are more likely to get a decent performance from a candidate who doesn’t actually know the domain, but a subsequent discussion asking them to explain their approach seems like it would be enough to ferret that out. |
If you're in an environment that is open to agents to begin with, one simple thing to do is just tell people to use agents, and ask for the prompts. Prompts are a high-signal artifact, and you can construct rubrics to evaluate them objectively.
I'll get around to writing a piece about this within the next couple months.
Ultimately work samples aren't a technology technique; they're a longstanding concept in management science. Long before the first coding take home was ever given, firms were using work samples to qualify salespeople, support professionals, factory workers, whatever. AI agents are a part of what the work of software development involves, and there's obviously no fundamental reason why you can't work sample them like anything else.