[uberman]

While I'm personally not keen on LLMs, I admit I do have the co-pilot extension installed in Visual Studio and have been pleasantly surprised at how tab-completion is working. It seems effective for small blocks of code.

So, remembering that I not really a fan when I ask this... but why do you care if a candidate uses an LLM or Google as part of your interview? Do you care if they use an IDE or if they use a code completion plugin? In the end, do you not really want to evaluate if the candidate can produce good clean code?

If you feel like an LLM is too big a crutch, is that because what you wanted to test was memorization of a framework or a test of thought and workflow strategies?

To quote a resource I'm also not keen on but understand why it exists, does your concern about chatGPT during interviews actually point out an XY problem?

[calabin]

The screen we are performing here is a basic "can you program" type of evaluation.

We've run into a number of people with seemingly-decent resumes (several positions as engineers at reputable albeit non-FAANG companies like insurers or e-commerce firms) who have struggled to complete basic tasks like the Pascal's Triangle question mentioned above.

The intent here is to toss them a couple of softballs that they should be able to knock out of the park, almost like if they were helping a younger sibling with CS 1XX or 2XX level work.

We're not against the use of Copilot, etc. once onboarded. We just want to make sure that these candidates possess basic skills that their resumes would suggest they mastered years ago.

[TheMongoose]

Your questions are pointless. I have no interest in CS puzzles at all, I don't memorize or practice them, and that's what you're testing for. I write code to build stuff.

Why not have a conversation with them about things they have built instead? Not form questions that say "tell us about a time you encountered a problem you had to solve" but an actual conversation. Like... an interview.

[calabin]

This screen is after the "walk us through your resume, explain what you've built" conversation, where we ask them about specific challenges, decisions they made, etc.

Everyone who has made it to this screen has already spoken to us about their previous work and at least seemed to have some skill in their general area of programming.

What we're testing for doesn't need to be memorized or practiced. Anyone who can program for money should be able to do things like FizzBuzz, repeat the pattern in Pascal's Triangle with a for loop, or come up with a basic strategy to eliminate non-primes.

Even if you blow the part of the interview where you need to identify prime/not-prime, as long as you show us that you have a process you're fine.

Maybe we just suck at or are too charitable when it comes to processing what candidates have claimed to have done - I'm unsure.

[jrgilman]

Would you hire someone who can't solve fizz buzz without a LLM? Are we being for real here? Fizz Buzz tests the absolute basics of understanding control flow.

[uberman]

Here is the thing though. Solving "FizzBuzz" is trivial if you happen to know how to do it. Same with "Pascal's Triangle" or many LeetCode style interview questions.

Solving either, particularly the latter when you don't "know" the test and are under pressure is not trivial. I have been in interviews for Ph.D. candidates who when asked simple questions like "Why do you want a Ph.D?" suddenly can't even remember how to speak English let alone explain why they want the advanced degree.

FizzBuzz is kind of a trap. If you don't know it, it seems like it would be easy to try nested if statements as candidates want to write efficient code and can fall into the trap of not wanting to repeat a test for 3 or they try a test for 3 then 5 then 15 before realizing that the easiest solution is three properly ordered if statements and that there is no "elegant" way to solve this trivial problem.

In fact, i work every day with data scientists who write perfect classification code and generate complex statistical models who would likely struggle to write FizzBuzz on the whiteboard in front of the entire team.

I hate interview questions like these and struggle to connect them to performance. If one must test then give a small project (something that might take me 30 mins to an hour) and ask the candidate to complete it at their own pace and publish their work in progress to a git repo. Can they solve the problem (even if they used google and/or an LLM)? Are their commits in meaningful units of work? Can I read their code in a review? Did they ask for proper clarification when uncertain? Do they have tests?

I find this much more salient than FizzBuzz.

[sn9]

The only prior knowledge FizzBuzz requires is the first few weeks of an introduction to programming.

[TheMongoose]

No, I wouldn't waste either of our time asking that question.