[Arainach]

Live coding interviews are the future. The risk of false impersonation was low enough for a while to rely solely on virtual screens, but widespread access to and awareness of large language models has made any virtual exercise unsustainably corrupted.

Sure, you can often tell the folks who know nothing when asking them to explain the code, but it's increasingly difficult to tell a great engineer apart from a mediocre one that's cheating, and once you start hiring cheaters the toxic effect to culture sets in fast.

[eesmith]

One of the author's points is that the live coding questions are often the types of problems which, in the workplace, might best be done with GPT.

"So at that point, do they want to see you muddle through it, or would they rather see that you know to have ChatGPT run through the initial pass and then refactor?"

"If a company is evaluating engineers with questions that can be easily answered by AI in seconds, what are they really evaluating for? Perhaps they’d be better off hiring a chat bot."

[pjsg]

If a candidate said that they would use ChatGPT to answer the problem, then I'd start to talk to them about the legal issues surrounding that approach. What would they do if the hiring organization forbade the use of ChatGPT for anything?

I would hope that all SWE candidates would understand software licensing to some extent and how to behave in a way that would not put the hiring organization in a legally risky position.

[eesmith]

Given your hope, shouldn't you be asking software licensing questions to every candidate? Irrespective of them mentioning ChatGPT? Otherwise, what happens if two months in they start using ChatGPT?

ChatGPT isn't the only legally risky software licensing issue.

A lot of people, including me, will use a search engine to find a solution to a given problem. That solution may be covered by copyright, and subject to a license which is not compatible with the company's business model. Some companies prohibit the use of GPL software. I consulted for one company which required authorization before using any open source software.

So it seems to me you should already be asking people about these sorts of issues.

[emodendroket]

I think that’s not really relevant, in the same way that it doesn’t much matter when they take a football prospect’s running times that someone could go much faster on a bicycle or in a car.

[eesmith]

I'm pretty sure I heard the same point made 20 years ago against using IDEs, with their auto-complete and tool-tips and style feedback.

And 40 years ago in the debate over letting kids use a calculator in school instead of calculating by hand.

A footballer follows a very constrained set of rules. If the footballer were allowed to use a car then 1) it would be easy to score a point, 2) the field would be ruined, and 3) people wouldn't pay to watch or support the team.

If a programmer uses ChatGPT to get a handle on a task with a new API, and saves a day of futzing around, how is that NOT relevant to job performance? (For the sake of argument, let's say that experiment showed that API doesn't scale well enough, resulting in a decision to scrap that approach entirely and use a different API.)

[emodendroket]

We’re not talking about whether you should use them at work; we’re talking about whether it makes sense to have an evaluation where you don’t use them. Closed-book exams are similar. There’s no real-world situation where you wouldn’t be allowed to refer to whatever materials you like, but the evaluation uses somewhat unnatural circumstances to gauge how well you’ve assimilated the material in a limited amount of time and with a consistent process that’s fair for everyone.

[eesmith]

"It makes sense" leads us back to the linked-to essay, which argues that live coding a solution to "arbitrary and unrelated topics like creating a script to handle scores for bowling or something equally irrelevant" do not make sense as a way to

If the evaluation doesn't make sense, then the conditions placed on the evaluation don't really matter.

Similarly, "assimilated the material" only makes sense if the live coding interview really does cover "the material". To use your analogy, measuring a football prospect’s cycling times aren't that good of a test of football playing skills. I mean, yes, there's some overlap, but there are more useful ways.

And one of the example tests was "handle scores for bowling", which is far from most work-related issues.

"consistent process that’s fair for everyone."

The author addressed this idea at several points, including "Any belief that a live coding interview is a consistently reliable way to make an objective assessment represents willful ignorance at best."

Picking a name out of a hat containing potential employee names is also consistent and fair.

Just because it's easy to measure doesn't mean it's an effective predictor.

[emodendroket]

I suppose I fundamentally do not agree that the skills are unrelated. I think they are the same skills, but applied to a much smaller problem that fits in 45 minutes. The author does not make any case that the smorgasbord of alternatives he offers are any better. And none of this has to do with your “well I can just Google a well-known problem like this in real life so it doesn’t matter” argument.

[8note]

And then your football team will lose to players who are all on bicycle?

[emodendroket]

No but it’s not like in a football game the other team is just going to let you run in a straight line without trying to impede you either.