[tptacek]

I shouldn't keep leaving this comment because it's not that useful but: AI-era work samples are a fun problem (you definitely can't just use your pre-agent default work samples!), and we've come up with a couple useful solutions.

If you're in an environment that is open to agents to begin with, one simple thing to do is just tell people to use agents, and ask for the prompts. Prompts are a high-signal artifact, and you can construct rubrics to evaluate them objectively.

I'll get around to writing a piece about this within the next couple months.

Ultimately work samples aren't a technology technique; they're a longstanding concept in management science. Long before the first coding take home was ever given, firms were using work samples to qualify salespeople, support professionals, factory workers, whatever. AI agents are a part of what the work of software development involves, and there's obviously no fundamental reason why you can't work sample them like anything else.

[csh0]

In a similar vein, in the pre-LLM era I gave a few interviews where the candidate was asked to screenshare while solving the problem. The candidates were allowed to use any resource they wanted from the broader internet.

I often found that I learned more about the candidate by the way they phrased their Google searches and how they selected and explored sites for information than from the actual solution they produced.