[joshuaturner]

I think a lot of this comes down to AI. In a recent hiring round we experienced multiple candidates using AI tooling to assist them in the technical interviews (remote only company). I expect relationship hires to become more common over the next few years as even more open-discussion focused interview rounds like architecture become lower signal.

So with that in mind I'll see you all at ReInvent

[rsanek]

If you're giving remote interviews, your loop should assume candidates can use AI. it's like giving a take home math test that assumes people won't use calculators at this point

[joshuaturner]

I disagree. We pretty explicitly ask candidates to not use AI.

While it's fine when doing the job the purpose of the interview is to gauge your ability to understand and solve problems, while AI can help you with that you understanding how to do it yourself signals that you'll be able to solve other more complex wider-spanning problems.

Just like with a calculator - it's important for candidates to know _why_ something works and be able to demonstrate that as much as them knowing the solution.

[ryandrake]

Asking candidates: "Don't use AI" is like all those other arbitrary handicaps that interviewers used to (and sometimes still do) weirdly insist on:

"Write this code, but don't read the API definition (like a normal developer would do in the course of their work)"

"Whiteboard this CRUD app, but don't verify you did it right using online sources (like a normal developer would do in the course of their work)"

"Type this function out in a text document so that you don't have the benefit of Intellisense (like a normal developer would have in the course of their work)"

"Design this algorithm, but don't pull up the research paper that describes it (like a normal developer would do in the course of their work)"

You're testing a developer under constraints that nobody actually has to actually work under. It's like asking a prospective carpenter to build you a doghouse without using a tape measure.

[jacobyoder]

Take this online test in 30 minutes with awkwardly or ambiguously worded abstract problem. You don't get to ask anyone for clarification on anything like any normal developer would do in the course of their work.

I've never been in a situation where I could not ask for clarification on something except in interview situations. I asked an interviewer once "is this how people normally work here? they just get a few sentences and plow ahead, without being able to ask for more details, clarifications, or use cases?". "Well, no, but you have to use your best judgement here".

[jjav]

> Just like with a calculator - it's important for candidates to know _why_ something works and be able to demonstrate that as much as them knowing the solution.

And this is why I never give coding-puzzle interviews. I just have a chat about your past projects (based on resume). We'll go deep into the technical details and it is easy in such a conversation to get a feel for whether you actually contributed significantly to the things the resume says you did.

[jkaplowitz]

Only sometimes is your historical deep-dive approach going to give you the right signal.

I’ve been out of work for a couple of years due to complicated immigration reasons, and I was most recently a people manager (although with a few direct technical tasks still). I honestly don’t remember many of the deep technical details of things to which I genuinely contributed significantly as an individual contributor or tech lead, despite those being entirely real and despite me still being a capable hands-on technical person. I’ve had so many jobs recently reject me for reasons like this without giving me a chance to actually demonstrate what I can do.

Memory tests are biased toward people who did the work recently, and biased against people with ADHD (who often have worse long-term memory for such details without being worse hires).

The coding interview shouldn’t be just a blind submit and wait for feedback, nor a live rushed and high-pressure puzzle test (you’re quite right in that regard). Ideally it should be the candidate doing what’s expected to be 1-3 hours of work asynchronously at their convenience within a period of a few days, and then discussing (maybe even presenting/demoing) live in a way that shows deep technical understanding and good communication skills. That avoids conflating memory tests with technical tests. Certain live coding tests can also be okay, but I agree it’s easy to make them unnecessarily uncomfortable with a false signal either way.

[grapesodaaaaa]

This is the way.

[randomdata]

There is an interesting dichotomy in your interview process. You say you want someone who can solve problems, but then go on to say (perhaps unintentionally; communication is hard) that you only want someone who has already rote-memorized how to solve the particular problems you throw at them, not someone who can figure things out as the problems arise.

[Izkata]

> but then go on to say (perhaps unintentionally; communication is hard) that you only want someone who has already rote-memorized how to solve the particular problems you throw at them

They said the opposite of that. Unless you think it's not possible to figure out problems and you can only do them by rote memorization?

[randomdata]

> Unless you think it's not possible to figure out problems and you can only do them by rote memorization?

It is not possible to solve a problem from scratch. You must first invent the universe, as they say. Any solution you come up with for a new problem will build upon solutions others have made for earlier problems.

In the current age, under a real-world scenario, you are going to use AI to help discover those earlier solutions on which to build upon. Before AI you would have consulted a live human instead. But humans, while not what we consider artificial, are what we consider intelligent and therefore presumably fall under the same rule, so that distinction is moot anyway.

Which means that, without access to the necessary tools during the interview, any pre-existing solution you might need to build upon needs to be memorized beforehand. If you fail to remember, or didn't build up memories of the right thing, before going into the interview, then you can't possibly solve the problem, even if you are quite capable of problem solving. Thus, it ends up being a test of memory, not a test of problem solving ability.

And for what? AI fundamentally cannot solve new problems anyway. At best, it can repeat solutions to old problems already solved, but why on earth would you be trying to solve problems already solved in the first place? That is a pointless waste of time, and a severe economic drain for the business. Being able to repeat solutions to problems already solved is not a useful employment skill.

[squeaky-clean]

> you only want someone who has already rote-memorized how to solve the particular problems you throw at them, not someone who can figure things out as the problems arise

This is literally what AI is, and why they don't want it used in the interview.

[randomdata]

Literally someone (or, at least, some thing) that can figure things out as problems arise? That seems quite generous. Unless you're solving a "problem" that has already been solved a million times before, it won't have a clue. These so-called AIs are predictive text generators, not thinking machines. But there is no need to solve a problem that is already solved in the first place, so...

It is really good at being a "college professor" that you can bounce ideas off of, though. It is not going to give you the solution (it fundamentally can't), but it can serve to help guide you. Stuff like "A similar problem was solved with <insert research paper>, perhaps there is an adaptation there for you to consider?"

We're long past a world where one can solve problems in a vacuum. You haven't been able to do that for thousands, if not millions, of years. All new problems are solved by standing on the shoulders of problems that were solved previously. One needs resources to understand those older problems and their solutions to pave the way to solving the present problems. So... If you can't use the tools we have for that during the interview, all you can lean on is what you were able to memorize beforehand.

But that doesn't end up measuring problem solving ability, just your ability to memorize and your foresight in memorizing the right thing.

[Toorkit]

Oh, if you give it the old "pretty please" then of course no one is going to use AI.

[snapcaster]

>We pretty explicitly ask candidates to not use AI.

This doesn't work, because regardless of what the rules say if i think all my competitors are using AI (and you won't be able to reliably detect it) i'll feel pressured to use it as well. This is true of any advantage (spending extra time on 2 hour takehome assignment is the classic version of this)

[_heimdall]

To be fair, you don't have to fall into the Tragedy of the Commons trap. If everyone else chooses to use LLMs during an interview that's their choice. If you are asked not to and are uncomfortable using it anyway, just don't.

[snapcaster]

Of course, just like pro athletes don't have to do steroids but it's obviously a competitive disadvantage

[papichulo2023]

Prob everyone is usually AI at this point, only a few bother to make the solution mor human-made-like. Personally, there is not way I am writing boilerplate again unless you are paying me hourly for the test.

[_heimdall]

Well that could explain why I've personally had a hard time finding a job with around 15 years of experience - I don't use LLMs.

With that said, none of the interviews I've had over the last couple months included questions that could reasonably be done with an LLM. The context is usually wrong, technical challenges were done live on a video call and it would be horribly obvious if a candidate was just prompting an LLM for an answer.

[jp42]

I wonder how cheating problem could be solved?

A cheap multi camera system + software, that can be quickly installed at candidates location to watch interviewing candidate. This can be sent by employer before interviews. its cheap enough that it can be thrown away.

traditional way - A company that provides interviewing centers across major cities for software interview, the location will have cameras that will make sure candidates are not cheating.