[hansvm]

It's been a year since I've actively given out take-homes for hiring, but I'm not sure I agree that everyone will use AI. I designed half the questions to be impossible for current-gen AI to answer without the candidate actually knowing what's going on [0], and only ~1% of candidates who responded did poorly on that half and not the other half (and, if we're worried about LLMs being better than I think, not all that many candidates passed most questions either).

[0] The most reliable strategy I've found for that is choosing questions where the wrong answer is the right answer for some much more common question. Actually spending a few seconds and solving the problem easily lets a human pass, but an LLM with insufficient weights or training data (all of them) doesn't stand a chance.

[seer]

Thanks for clarifying - I kinda get the idea but would love to see an example for this.

I’ve mostly given up on all of the standard techniques for interviewing sadly, just because “using ai” makes a lot of them trivial, and have resorted to the good old fashioned interview, where I screen for drive, values and root cause seeking, and let people learn tech/frameworks/etc themselves.

But I was wondering, isn’t a take home question still good, if you give a more open ended and ambitious task, and let people vibe code the solution, review the result but ask for the prompt/session as well?

People will be doing that during normal work anyway, so why not test that directly?

[hansvm]

> examples

One such question (obviously tailored to the role I'm hiring for) is asking whether SoA or AoS inputs will yield a faster dot-product implementation and whether the answer changes for small vs large inputs, also asking why that would be the case.

I typically offer a test with a small number of such questions since each one individually is noisy, but overall the take-home has good signal.

> why not test that directly?

The big thing is that you don't have enough time to probe everything about a candidate, especially if you're being respectful of their time and not burning too much of yours. Your goal is to maximize information gain with respect to the things you care about while minimizing any negative feelings the candidate has about your company.

I could be wrong, but vibe coding feels like another skill which is more efficient to probe indirectly. In your example, I would care about the prompt/session, mostly wouldn't care about the resulting code, and still don't think I would have enough information to judge whether they were any good. There are things I would want to test beyond the vibe coding itself.

In particular, one thing I think is important is being able to reason about code and deeply understand the tradeoffs being made. Even if vibe coding is your job and you're usually able to go straight from Claude to prod, it's detrimental (for the roles I'm looking at) to not be able to easily spot memory leaks, counter-productive OO abstractions, a lack of productive OO abstractions, a host of concurrency issues LLMs are kind of just bad at right now, and so on. My opinion is that the understanding needed to use LLMs effectively (for the code I work on) is much more expensive to develop than any prompt engineering, so I'd rather test those other things directly.