[jgafni]

Are you picking a problem candidates can easily cheat on? Could Claude produce the answer in 30 seconds? If so, what's the test actually measuring?

The setups I've seen produce the strongest signal aren't the ones where the candidate can't see the answer that AI produces easily. They're the ones where there is no single answer. Architecture decisions. How you would investigate an outage. How you would tradeoff one constraint against another. The candidate can spend an hour with Claude on it beforehand and it doesn't matter, because the test isn't "did you get the answer," it's "can you talk through the reasoning" and "can I trust your judgement."

Hearing someone explain their thinking on an open problem is a much harder thing to fake. Even if they used an assistant to structure their initial pass, the live unpacking reveals where the thinking is theirs and where it isn't.