[blackbear_]

Ask them to share their screen, walk you through their code and explain their solution live. Every once in a while ask why they did it in that way and what alternative approaches there could be. Then change the problem a little and ask them to modify their code, again live. Generally, try to go deep and ask follow-up "why" or "how" questions. Those who "don't remember" or only offer vague and shallow answers are likely to have cheated with LLMs (or are just poor candidates).

[calabin]

This is generally what we do and what raised our suspicions in the first place - they both could "walk us through" their code, but had trouble explaining why they did certain things, how they could improve things, etc.

We thought that the approach you've outlined would generally be good enough, and has led us to catch instances where people are leaning heavily on LLMs, but our issue now is that everyone appears to be using these things. Admittedly, our sample size here is low (n=3). But it's still frustrating nonetheless.

[blackbear_]

You could try to give a challenge that has a few hidden gotchas, and discard candidates that do not spot them. How to do this depends on the role you are hiring for.

For example, in our data scientist interviews we also candidates to analyze datasets with imbalanced classes, outliers, correlated samples, etc. Correctly dealing with these issues requires particular techniques, and most importantly the candidate has to explicitly check whether these issues are present or not. Those who use LLMs mindlessly will not even realize this is the case.

[calabin]

I like this - a huge part of our engineering work is ETL pipelines so giving them some data to process makes things a lot harder to fake.