[m1el]

I've had a displeasure of interviewing someone who used ChatGPT in a live setting. It was pretty obvious: I ask a short question, and I say that I expect a short answer on which I will expand further. The interviewee sits there in awkward silence for a few seconds, and starts answering in a monotone voice, with sentence structure only seen on Wikipedia. This repeats for each consecutive question.

Of course this will change in the future, with more interactive models, but people who use ChatGPT on the interviews make a disservice to themselves and to the interviewer.

Maybe in the future everybody is going to use LLMs to externalize their thinking. But then why do I interview you? Why would I recommend you as a candidate for a position?

[blharr]

The idea that spotting cheating is obvious is a case of selection bias. You only notice when it's obvious.

Clearly, the person put 0 effort towards cheating (as most cheaters would, to be fair). But slightly adjusting the prompt, or just paraphrasing what ChatGPT is saying, would make the issue much harder to spot.

[al_borland]

Maybe I’m a slow reader, but reading, understanding, and paraphrasing the response seems like it would take enough time to be awkward and obvious as well.

I’m not sure why anyone would want a job they clearly aren’t qualified for.

[londons_explore]

As an interviewer, if a candidate can use chatGPT to give a better answer than other candidates, I'm not gonna mark them down for use of chatGPT.

It's a tool, and if they can master it to make it useful, then credit to them.

Alas, ChatGPT seems to be a jack of all trades, but master of none, which is gonna make it hard to pass my interviews which test very specific technical skills.

[photonthug]

> As an interviewer, if a candidate can use chatGPT to give a better answer than other candidates, I'm not gonna mark them down for use of chatGPT.

Tool usage is what separates us from animals and is generally ok where tools are available/expected, but in this case I think you misunderstand which tool we're talking about. The tool involved isn't actually chatGPT, it's more like strategic deception. Consider the structurally similar remark "as a voter, if a candidate can use lies to represent themselves as better than other candidates, I'm not gonna mark them down for use of dishonesty".

The rest of this comment is not directed at you personally at all, but the number of folks in this thread who are extremely eager to make various excuses for dishonesty surprised me. The best one is "if dishonesty works, blame the interviewer". I get the superficial justification here like "should have asked better questions", but OTOH we all want fairly short interview processes, no homework, job-related questions without weird data-structures and algorithms pop-quizes, etc, so what's with the double standards? Hiring/firing is expensive, time-consuming, and tedious, and interviewing is also tedious. No one likes picking up the slack for fake coworkers. No one likes being lied to.

[WalterBright]

> the number of folks in this thread who are extremely eager to make various excuses for dishonesty surprised me

Not me. I see it all the time, online and offline. I suspect they think it confers status on themselves, but what actually happens is honest people wind up shunning them.

[dr_kretyn]

> Tool usage is what separates us from animals

It does not. Please ask any LLM for examples of animals that use tools. (My examples: chimpanzees, gorillas, elephants, dolphins, otters...)

[photonthug]

I assume you know this is just an expression, and you know that I know that animals indeed use tools. So I'll refer you to community guidelines https://news.ycombinator.com/newsguidelines.html

[verve_rat]

The problem is that a good interview is only vaguely related to getting a good employee. Anyone can ace and interview and then slack off once they have to job.

If someone aces the interview using an LLM and then does good work using that same LLM then what should the employer or other employees care? The work is getting done, so what's the problem?

Compare a shitty worker to a deceptive one using an LLM. They both passed the interview and in both cases the work isn't being done. How are those two cases different?

[wakawaka28]

Your hypotheticals are all extremely unlikely. People who ace interviews are usually good, and people who lean on stuff like ChatGPT aren't. I'd also rather not have someone dumping massive amounts of ChatGPT output into a good codebase.

>what's the problem?

Using a LLM is akin to copy/pasting code from random places. Sure, copy/paste can be done productively, except ChatGPT output comes completely untested and unseen by intelligent eyes. There are also unsolved copyright infringement issues via training data, and a question as to whether the generated code is even copyrightable as it is the output of a machine.

[photonthug]

> does good work using that same LLM then what should the employer or other employees care?

Maybe I'm wrong, but I find it very hard to believe that anyone thinks the "good work" part here is actually a practical possibility today. Boilerplate generation is fine and certainly possible, and I'm not saying the future won't bring more possibilities. But realistically anyone that is leaning on an LLM more than a little bit for real work today is probably going to commit garbage code that someone else has to find and fix. It's good enough to look like legitimate effort/solutions at first glance, but in the best case it has the effect of tying up actual good faith effort in long code reviews, and turns previously productive and creative individual contributors into full-time teachers or proof-readers. Worst case it slips by and crashes production, or the "peers" of juniors-in-disguise get disgusted with all the hand-holding and just let them break stuff. Or the real contributors quit, and now you have more interviews where you're hoping to not let more fakers slide by.

It's not hard to understand that this is all basically just lies (misrepresented expertise) followed by theft. Theft of both time & cash from coworkers and employers.

It's also theft of confidence and goodwill that affects everyone. If we double the number of engineers because expectations of engineer quality is getting pushed way down, the LLM-fakers won't get to keep enjoying the same salary they scammed their way into for very long. And if they actually learn to code better, their improved skills will be drowned out by other fakers! If we as an industry don't want homework, 15 interviews per job, strong insistence on FOSS portfolio, lowered wages, and lowered quality of life at work.. low-effort DDoS both in interviews or in code-reviews should concern everyone.

[recursive]

> Anyone can ace and interview

I'm doubting this quite a bit. If it's so easy to ace an interview, why are there so many bad ones?

[wakawaka28]

>As an interviewer, if a candidate can use chatGPT to give a better answer than other candidates, I'm not gonna mark them down for use of chatGPT.

I think that makes you an incompetent interviewer, unless your questions are too hard for ChatGPT. In any case, solving the question without ChatGPT is more impressive than using it. Just like most other tools, like search engines or IDEs.

[kazinator]

Would you also say that, "as an interviewer, if a candidate can use their buddy to give a better answer than other candidates, I'm not going to mark them down for using their buddy"?

Even if you don't mind that situation, shouldn't you get buddy's contact information and offer him the job?

[smcin]

That's not a great analogy: you can't do the job with your buddy; whereas some interviewers are ok with, and even expect you to, use GenAI on the job daily. Depends on the interviewer and job expectations.

A better analogy is an interview where you can use a calculator (and not be detected). If the interviewer were only to ask you simple arithmetic questions with numeric answers then sure you'd seem to do well. So interviewers adjust to not doing that.

[toast0]

> you can't do the job with your buddy

I mean, why not? Me and my buddy are a team, hire us both or none of us. Split the salary if you must.

[thegrim33]

To use a slightly more extreme example .. if you were hiring someone to maintain a nuclear power plant, and when you asked them a question about what actions to take to avoid a meltdown, and they had to ask ChatGPT to figure it out, would you really be OK with hiring that person to maintain your nuclear plant? When they don't actually have the knowledge they need to succeed, but instead have to rely on external tools to decide things? If they need to ask ChatGPT for the answer, how do they know if the answer is right? You really think that person, who relies on tools, is just as good of a hire as someone that fully internally knows what they need to know?

Yeah, hiring someone to code a website isn't the same as maintaining a nuclear plant, but it's the same concept of someone that knows their craft vs. someone that needs to rely on tools. There's a major difference in my mind.

[hackit2]

I hope your statement is hyperbolic because we're all doomed if you expect a person to know how to operate a nuclear power plant. Normally, your testing if they can follow operational procedure that were created by people who designed the power plant in the first place.

Similar it is unreasonable and bordering on negligence to assume a person has the skill set unique to your situation.

[TeMPOraL]

If the job at your nuclear power plant were so simple you only needed the employee to follow operational procedures, then you'd be better off scripting it instead, or training a monkey.

Consider e.g. being a pilot, or a surgeon - two other occupations known for their extensive use of operational procedures today. People in those jobs are not being hired for their ability to stick to a checklist, but rather for their ability to understand reasons behind it, and function without it. I.e. the procedures are an important operational aid, not the driver.

Contrast with stereotypical bureaucrats who only follow procedures and get confused if asked something not covered by them.

Now, IMHO, the problem here is that, if you're hiring someone who relies on an LLM to function, you're effectively employing that LLM, with its limitations and patterns of behavior. As an employer, you're entitled to at least being made aware of that, as it's you who bears responsibility and liability for fuckups of your hires.

[anileated]

Like a university diploma is a signal of being able to learn or at least comply, use of a chatbot is a signal of not bothering enough to learn or comply.

I can see how an applicant who cheats interview with chatbot would later not bother to internalize operation instructions for the job.

[fragmede]

in terms of tools, I absolutely want the nuclear power plant engineer to use a wrench and pliars and tongs and a forklift and a machine while wearing a lead lined safety suit instead of wandering over to the reactor in a t-shirt to pull out the control rods with their bare hands. You could be Edward Teller and know everything there is to know about nuclear physics but you're not getting anywhere without tools.

to your point though, a person needs both. all of one and none of the other is useless. You don't want someone who doesn't know what they're doing to play around disabling safety systems so you don't get Chernobyl, but for the everyday crud website you can just hire the coding monkey at a reduced cost.

[lucasmullens]

That's like being okay with a candidate Googling the answer during an interview. Not unheard of, but unusual. It seems hard to test someone's knowledge that way.

[krisoft]

> Not unheard of, but unusual.

At my company we tell people that they should feel free to google or consult references at practical coding challenges.

> It seems hard to test someone's knowledge that way.

I don’t really want to test knowledge but skill. Can you do the thing? At work you will have access to these references so why not during the interview?

Now that doesn’t mean that we are not taking note when you go searching and what you go searching for.

If you told us that you spent the last 8 years of your life working with python and you totally blank on the syntax of how to write a class that is suspicious. If you don’t remember the argument order of some obscure method? Who cares. If you worked in so many languages that you don’t remember if the Lock class in this particular one is reentrant or not and have to look it up? You might even get “bonus points” for saying something like that because it demonstrates a broad interest and attention to detail. (Assuming that using a Lock is reasonable in the situation and so on of course :))

[batch12]

> I don’t really want to test knowledge but skill

I do want to understand their knowledge. I'll preface questions with the disclaimer that I am not looking for the book definition of a concept, but to understand if the candidate understands the topic and to what depth. I'll often tell them that if they dont know, just say so. I'll start with a simple question and keep digging deeper until either they bottom out or I do.

[londons_explore]

I'm okay with them googling too. And I tell them that at the start. But if they take ages to lookup the answer when others just know the answer, it's gonna hurt their chances.

[gtirloni]

Sure, they can search it live but you have to assess if they understand what they found. Usually, if they really know their stuff, whatever they find is just gently pushing their working memory to connect the dots and give a decent answer. Otherwise it's pretty easy to ask a follow up question and see a candidate struggle.

It's like in college when you're allowed to take textbooks to an exam. You can bet the professor spent more time crafting questions that you can't answer blindly.

That being said, I think both types of questions have their place in an interview process. You can start with the no searching allowed questions in the beginning to assess real basic knowledge and, once you determine the candidate has some knowledge, you start probing more to see if they can connect the dots, maybe it's architecture decisions and their consequences, maybe it's an unexpected requirement and how they would react, etc.

[munk-a]

The knowledge we're testing is related to how well you can do your job. Work isn't closed book - if you can quickly formulate a good query to grab any missing information off the internet then more power to you. I've worked with extremely stubborn people who were very smart and would spend a week trying to sort out a problem before googling it, there are some limited situations (highly experimental work) where this is valuable but... I no longer work with these people.

[yunwal]

Seems much easier than trying to prevent them from using google

[oceanplexian]

I remember the days when Greybeards would look down on me for using Google in my first IT job, they would harp on about how real Sysadmins use man pages and O’Reilly books to solve problems, and if you tried to Google something you were incompetent. I had college professors that told me you can’t use the Internet for research because the Internet is a not a legitimate source of information, only libraries can have real information.

What happened to all those folks? They retired, and turned into Boomers who are now unable to function in society at a basic level and do things like online banking or operate a smartphone.

[esafak]

On the other hand, they knew how their hardware worked. And if LLMs keep improving, we're going to reach the last generation that knew how software worked.

[zerbinxx]

We’re pretty close. I’m not sure that 51% of the people I work with understand what DNS is, what a call stack is, what the difference between inheritance and polymorphism is, or what a mutex is

[nonrandomstring]

When I'm retired, sitting on the beach with my beer and a good book, please don't come bothing me that your smartphone banking and GPT arse-wiping assistant has gone berserk.

You're on your own matey.

But hey, you'll have Google.

[xboxnolifes]

> I’m not sure why anyone would want a job they clearly aren’t qualified for.

$$$,$$$

[dmoy]

Well, five moneys at least. They might figure out and fire you before you get to six moneys (but maybe they won't, who knows).

[WalterBright]

It will take 3 to 6 months to determine that a new hire is incompetent, especially if you're required to document their incompetence before firing them.

[skeeter2020]

I've never had a job without a probation period where you can let someone go without cause within the first 90 days with nothing more than two weeks pay in lieu of notice. It definitely doesn't take 6 months to identify someone who only got their job because they used AI in the interview.

[Aeolun]

> I’m not sure why anyone would want a job they clearly aren’t qualified for.

Well, I suck at interviewing and/or leetcode questions, but have so far done perfectly fine in any actual position.

I can totally see how you’d resort to ChatGPT to give the interviewers their desired robotic answers after 3 months of failing to pass an interview the conventional way.

[Swizec]

> give the interviewers their desired robotic answers

As someone who has interviewed a lot of people – robotic answers are specifically not what I (we?) look for. The difference between hands-on experience and book knowledge is exactly what we're trying to tease out.

It's very obvious when someone is reciting answers from a book or google or youtube or whatever vs. when they have actually done the thing before.

For the record: ChatGPT is very good and the answers it gives are exactly the kind of answers that people with book knowledge would give. High level, directionally correct, soft on specifics.

I mostly interview seniors, you obviously wouldn't expect experience from an entry-level candidate. Those interviews are different.

[munk-a]

I understand that you have no control over who you're interviewing with but... if you're a good fit and the interviewer leaves thinking you're a terrible fit that's a sign of a bad interviewer. Obviously there are non-proficiency things you can do to skew that perception (bad hygiene, late, obviously disinterested) but a good interviewer (especially one used to working with developers) should be good at getting by all the social awkwardness to evaluate your problem solving.

And yes, most large companies have terrible interviewers.

[aydyn]

Yes agreed that it is a problem with interviewers, but in practice all the responsibility falls on the interviewee.

I've never once seen an interviewer getting better in any company I've worked for. What happens is they just move onto the next interviewee.

[Aeolun]

I refuse to believe that all the interviewers I had over the course of 6 months were all terrible. It must be something about the process that is pathologically broken (especially when getting hired at larger companies)

[verve_rat]

I mean... if the interview process is even a little broken then doesn't that mean that over time worse and worse interviewers will get hired, making for worse and worse interviews meaning that worse and worse interviewers get hired...

[davidthewatson]

That resonates. Me too!

Here's Pew's Janna Anderson in 2015:

    "Algorithms are taking over much of the human work of hiring humans. And, unless they are programmed to seek out currently undervalued and difficult-to-track factors, they may tend to find that the more robot-like a human is the best she or he will be at doing most jobs. So, it could be that the robots are most likely to hire the most robotic humans."

https://medium.com/@jannaq/the-robot-takeover-is-already-her...

I find the whole gamified system to be bizarre and disheartening no matter which side of the table you're on.

To me, looking at modern tech interviewing is like comparing the gold standard OCEAN and the emergent HEXACO in personality surveys. Take the former on a bad day and it may leave the test taker feeling bad about themselves. The latter, much kinder and gentler in messaging around strengths and weaknesses.

That "by design" quality strikes me as missing from the entire tech interview system. If it weren't broken, this would not be a 7-year conversation updated yesterday:

https://github.com/poteto/hiring-without-whiteboards

[TeMPOraL]

> I’m not sure why anyone would want a job they clearly aren’t qualified for.

Money, obviously.

Software jobs in particular are magic in this way - the pay is way above the average, and performance metrics are so poorly defined that one can coast for months doing nothing before anyone starts suspecting anything. Years, even, in a large company, if one's lucky. 80% of the trick is landing the first gig, 15% is lasting long enough to be able to use it as a foundation of your CV, and then 5% is to keep sailing on.

No, really. There's nothing surprising about unqualified people applying for software companies. If one's fine with freeloading, then I can't think of easier money.

(And to be fair, I'd say it's 10% of freeloaders, 10% of hard workers, and in between, there's a whole spectrum of varying skills and time and mental makeups, the lower half of that is kind of unqualified but not really dishonest.)

[bradlys]

Just because I can’t recite rabin-karp off the top of my head or some suffix tree with LCA shit for some leetcode question about palindromes doesn’t mean I’m unqualified to do the work of an engineer.

I’ve gone public, been acquired by Google, and scaled solutions to tens of millions of users. I’m probably overqualified for your CRUD app.

[Atotalnoob]

We are interviewing for a principal at my work, and the directions from management are to find someone really good.

Instead of tearing into their experience, my coworker is asking what you would use the X class for.

Drives me fucking nuts. Who memorized all the parts of a random, mostly unused .NET class.

I asked the coworker afterwards if he ever used said class, dude said no.

How is that a fair question if it isn’t even used here?

[_hcuq]

Exactly. I would never interview for a job, be humiliated by a moron.

Luckily I have the contacts and experience to never have to.

[notnaut]

Consider a situation where you’re applying for a job that you’re 50% qualified for and then using chatgpt to cheat on the interview. Would be much more difficult to catch is my guess.

[throwaway2037]

This is an interesting thought experiment.

If you slide from 50% to 99%, how do people feel about using ChatGPT? What is more honest: Many people here were hired when they were less than 100% qualified, and did very well in their new role. It has happened to me more than once.

[redsoundbanner]

>I’m not sure why anyone would want a job they clearly aren’t qualified for.

Easy. They have nothing to lose because the jobs they are qualified for don't even pay enough to survive. You probably could have figured this out yourself.

[ozim]

If someone is smart enough to go away with it is enough that I know but it doesn’t bother me much- I don’t mind.

Had an interview take home assignment done by GPT and it was easy to spot after seeing dozens of solutions. Downside for the guy was - it didn’t work.

[godelski]

You probably should because this is a person demonstrating that they can and will hack metrics rather than consider what you're proxying.

[ozim]

In my current setting it doesn’t work like that.

We have a small team of developers and you cannot hack metrics. You build and deliver what is in requirements or not.

If you don’t deliver we don’t even have to have a discussion because team reviews code, tests features and gives feedback quickly if someone is slacking.

[godelski]

All metrics can be hacked.

[ozim]

Not going to write too much but I think this is best reply:

https://xkcd.com/810/

[irrational]

We will have to start studying people's eyes to see if they are moving as if reading text.

[Fetiorin]

There is already an app from Nvidia that simulates constant eye contact with the camera

[mtlmtlmtlmtl]

Fantastic, now my social anxiety will cripple me in video chats too.

[throwaway2037]

I heard a rumor that Apple FaceTime does something similar, but I could not find any definitive evidence of it. Does anyone know more about it?

[dghlsakjg]

Apple ha it built in to iPhones now too. Center Stage is the name of the feature, but I think it is only available on FaceTime calls

[klausa]

That's not what Center Stage is.

Center Stage is a feature where a device uses an ultra-wide camera, and then is supposed to track your _face_ as you move and shift around it it's field of view.

I find it most useful for FaceTime calls on Apple TV, where you can leave your phone near the TV, and it will automatically frame you sitting on the couch and will follow you as you shift around, etc.

There is a similar feature to what you're describing for FaceTime, but I don't think it has any cutesy name.

[dghlsakjg]

You’re right, center stage is not the name of the feature. The feature does exist though, I believe it is called Eye Contact.

https://appleinsider.com/articles/20/09/25/how-to-use-faceti...

[wellthisisgreat]

It’s really uncanny - it give you an intense, unblinking stare

[dylan604]

Just shrink the width of the text area being read from. It's really easy to not look like you are sitting in the front row of the theater reading the opening text in Star Wars. If an actor on live TV can do it, you can too

[ThrowawayR2]

I predict that that will be followed shortly by a mysterious sharp increase in applicants claiming to have nystagmus (https://en.wikipedia.org/wiki/Nystagmus), which causes random involuntary eye movements, but without any medical documentation.

[Volundr]

What's interesting is this wouldn't necessarily imply cheating. That doesn't sound like an issue I'd necessarily draw attention to under normal circumstances, but if I knew interviewers were likely to be paying close attention to my eye movements I certainly would.

[ThrowawayR2]

Yes, exactly. I have nystagmus myself because of an underlying medical condition that causes other vision problems and it's depressing that interviewers might think it's reason for suspicion.

[bitzun]

I've been a pain in the ass in quite a few feedback sessions when people brought up a candidate not making "enough" eye contact. Usually I mention that they could be treading into infringing on a protected class and they shut up.

[jurynulifcation]

why wouldn't a cheater just pipe a generative audio model through a small earbud? like that one villain from season 3 of westworld

[johnnyanmac]

I mean, at some point if they go through so much effort to hide their cheating they probably have attained some mastery in the process. Kinda like how some friends in high school would try and sneak in note cards on a test but they probably spent so much time prepping them that they coulda gotten an A or B regardless.

It's also why it's kinda annoying to do live interviewing trivia questions. Can I immediately answer what a partial template specialization is? Probably not, I never used them. Can I google it in 2 minutes and summarize it as as way for (often c++) template classes to bound some of the template arguments to values or pointers? Well, I just did. Should that cost me the interview? That's pretty much what I do on the job.

[throwaway2037]

I am a polyglot: Perl, Python, C, C++, Java, C#, etc. Not experts at all, but I can do fine with an existing code base. What is it about C++-heavy interviews that always regress to trivia? And asking about rarely used features? It is a bother. And rarely does the person asking the trivia have any depth whatsoever in other languages. It is my biggest gripe with "C++ people". For many, they have a hammer and everything looks like a nail. Yes, "Java enterprise people" were the same in 2005-ish.

[twic]

What's your evidence for that claim?

[ptmcc]

Yes of course! I'd be happy to answer your short question with a short answer. I look forward to expanding further on the answer, as you previously stated that you expect me to.

Jokes aside, something about LLM responses is very uncanny valley and obvious.

[chewxy]

The peppy, upbeat, ultra-American tone that the LLMs produce can be somewhat toned down with good prompting but ultimately, it does stink of the refinement test set.

[throwaway2037]

True. We need an Aussie bogan mode for ChatGPT. Or, Guy Ritchie villian.

[foxyv]

To be honest, I think in the future we will interview people on their ability to work with an LLM. This would be a separate skill from the other ones we are looking for. Maybe even have them do some fact checks on a given prompt and response as well as suggest new prompts that would give better results. There might even be an entire AI based section of an interview.

In the end, it's just a new way to "Google" the answer. After all, there isn't much difference between reading off an LLM response and just reading the Wikipedia page after a quick Google search, except for less advertisements.

[SOLAR_FIELDS]

I’ve already been allowed to use it in programming interviews where they’ve said it’s explicitly allowed to use ChatGPT. It’s led to some fun interactions because I use it a lot and as such I’m quite good with it and interviewers are often taken aback by how quickly I’m able to just destroy the question they put out with a good prompt

I will say there are still some programming questions you can give that will stump the hell out of ChatGPT. In particular I took one online coding assessment where I used it and there was a question about plotting on a graph with code and calculating areas based on the points plotted that ChatGPT failed miserably at, but someone pretty good with math and geometry would find pretty tractable.

[fragmede]

There are ChatGPT resistant questions you can ask. ChatGPT recognizes the question but doesn't actually think about it, so if you give it the river crossing problem (farmer, fox, sheep, and grain need to cross a river) but tell it the boat can take all the items, it won't actually read those details and blithely solve the problem the expected way. Give candidates a problem that's trivially solvable if you actually read the question and see if they try and solve it the ChatGPT way.

[hirsin]

Indeed - with 3.5 at least it does fail the easy mode riddle https://chat.openai.com/share/b3761807-551d-4cfc-b291-6d37ee...

[fragmede]

it's a fun problem to explore, and gpt-4 doesn't do any better. swapping in other things doesn't help because it internally recognizes it as the river crossing problem and proceeds to solve it normally. I was able to get it to two shot it with a lot of coaching but yeah, it's a trip.

[ric2b]

The downside is that you're now wasting interview time on the river crossing problem instead of actually relevant questions for the job you're hiring for.

[fragmede]

Don't literally use the river crossing problem, but it's existence implies there is a form of question that ChatGPT will solve for where someone actually reading the prompt can solve trivially but someone using ChatGPT will get stuck on.

You're already asking Leetcode questions that are irrelevant for the job you're hiring for. What's the problem with asking one more to test for cheaters?

[theamk]

We didn't start testing people on Google usage when Googling became useful, so I don't see why LLMs would be different.

Instead, there would be tasks that can be completed using any tools available - Google, LLM, whatever. And candidates are rated on how well the task is done, and maybe asked a few questions to make sure they made decisions knowingly and not just copied the first answer off the internet.

This already exists and is called "take home programming assignment"

[jacques_chester]

I agree that this is the likely long term outcome. But for now folks want to think that everyone needs to have memorized every individual screw, nail, nut and bolt in the edifice of computer science.

[outside415]

Me and several friends have used ChatGPT in live interviews to supplement answers to topics we were only learning in order to bridge the gap on checkboxes the interviewer may have been looking for.

We’ve all got promotions by changing jobs in the last 6 months using this method.

You can be subtle about it if it’s already an area you kind of know.

[al_borland]

I like when a person admits they don’t know something in an interview. It shows they aren’t afraid to admit when they don’t have the answer instead of trying to lie their way through it and hoping they don’t get caught. Extra bonus points if they look the thing up later to show they are curious and want to close knowledge gaps when they become aware of them.

People who are unwilling to say, “I don’t know, let me look into that,” are not fun to work with. After a while it’s hard to know what is fact vs fiction, so everything is assumed to be a fabrication.

[jedberg]

When I was 11 I took a live assessment to get into the gifted program at school. I thought I didn't do very well because about 20% of the questions I answered "I don't know".

At the end the assessor told me that I passed specifically because I said "I don't know". They purposely put questions on the test they didn't expect you to answer to see what you do when faced with an unanswerable question.

I've used that in my own life since -- I much prefer working with (and have a much more positive view of) people who are willing to say "I don't know".

[BeFlatXIII]

Doesn't the SAT work similarly? They penalize wrong answers to discourage guessing. Either be confident in your answer or leave the question blank.

[jedberg]

I assume so, at least it worked that way when I took it 30 years ago.

[JohnFen]

I couldn't agree more. When I am interviewing candidates, one of the things that I'm looking for is that the applicant is willing to say "I don't know" when they don't know. That's a positive sign. If they follow that up with a question about it, that's even better.

If a candidate is trying to tap-dance or be vague around something to avoid admitting ignorance of it, that's a pretty large red flag.

[kjkjadksj]

For every one person hiring with your mentality there are a hundred other managers looking to cut down the stack of a thousand resumes in any trivially easy way they can. That starts with saying sorry we are looking for someone else when you say you don’t know x or lack z on your resume. You are literally incentivized to lie and fake it on the job.

[KTibow]

You could argue that researching it then and there proves that you know how to learn stuff quick. I agree that there should be disclosure though.

[noddingham]

I'm going to be pedantic and challenge your use of the word 'learn' here. I tend to agree with the notion that being able to say 'I don't know, let me find out' and then find out quickly with a correct answer is in general a Good Thing™, but I wouldn't equate that with learning the thing they just looked up.

[smcin]

The difference between 'learn', 'cram', 'regurgitate' etc. depends on the level of understanding required, and the length of the recall.

And whether the interview is just asking definitions or silly certification questions, or things requiring deeper understanding.

[al_borland]

Yeah, the disclosure is very important. It’s the difference between an open book test and notes written on their thigh.

During some interviews I’d give people access to a computer. If they could quickly find answers and solve problems, that is a skill in itself, but I could see what they were looking up. Sometimes that part would make or break the interview. Some people didn’t have a deep base of knowledge in the area we were hiring for, but they were really good at finding answers, following directions, and implementing them successfully. They would be easy to train on the specifics of the job. Other people couldn’t Google their way out of a paper bag, I was shocked at how bad some people were and looking up basic things. Others simply quit without even attempting to look things up.

[outside415]

while I agree with you in the context of a work environment 100%, in an interview there is a series of checkboxes interviewers need to hear and if you say I don't know I will look into that you can really screw yourself.

[jacques_chester]

So, assuming they didn't know and approve, you cheated.

[outside415]

I disclosed my use of chatgpt throughout the process and the hiring manager was excited that I was on the cutting edge. I used it for the project they gave me as well. :)

I don't think my friends disclosed that they were using it.

[jacques_chester]

Thank you for the clarification. I am glad you disclosed and disappointed your friends didn't.

[lcnPylGDnU4H9OF]

Dirty, dirty cheater! Sounds like they would have been able to perform the job duties so I'm not sure why one should care.

[dataflow]

There is literally not enough information to tell if they can perform their job duties or not.

[lcnPylGDnU4H9OF]

It's a fair point that I am making this assumption. At any rate, my comment could instead read:

> [If one assumes that the candidate] would have been able to perform the job duties I'm not sure why [they] should care.

This is what I mean; I can see why an interviewer thinks they've been cheated or that a candidate was dishonest but that doesn't mean that the interviewer even has a successful system for determining if a candidate can perform the job duties. A candidate who cheated -- from the perspective of the interviewer, I guess -- but still manages to adequately perform in their role very plainly did not cheat from a less biased perspective. What is that interviewer even thinking? How could that person have cheated?

[staunton]

> determining if a candidate can perform the job duties. A candidate who cheated -- from the perspective of the interviewer, I guess -- but still manages to adequately perform in their role very plainly did not cheat

That's not what anyone means when they say "cheating". Cheating means to violate the conditions and assumptions of an examination or contest.

For example, if a chess grandmaster uses an AI implant to win a game and gets caught, it doesn't make it OK if they could consistently win against the same opponent even without the AI.

[JohnFen]

People care because such a person isn't terribly trustworthy. There's more to being a valuable employee than just being able to perform the job duties.

[ThrowawayR2]

Those who lie about one thing are likely to lie about many others.

[InvertedRhodium]

Given that literally every person alive lies about some things, I'm not sure how much value that observation brings to the discussion.

[sethammons]

Curious outlook. I, for one, avoid lying. The closest I get is omission. I'm not interested in remembering false realities depending on the person I'm talking with. The last lie I recall was a number of years ago where I said to a store clerk that I had recently been somewhere when in fact it was not very recent. Immediately after I felt bad. I value honesty.

[jurynulifcation]

That job could have gone to someone who like actually knew what they were doing and was honest lol not sure why you want to defend professional and intellectual dishonesty?

[lcnPylGDnU4H9OF]

> intellectual dishonesty

This suggestion that a person who can adequately perform job duties could have even possibly cheated in their job interview is intellectually dishonest. If they had to cheat to get the job we should be looking at the interviewer. Why did the qualified candidate have to cheat? Why is whatever-they-did even considered cheating?

[JohnFen]

> Why did the qualified candidate have to cheat?

If they're qualified, they didn't have to cheat. If they're not, then they did. Either way, they're dishonest and that means they're not a desirable hire.

[jacques_chester]

That someone has the skills for a job is distinct from whether they are able to uphold a simple moral principle like "don't cheat".

[lcnPylGDnU4H9OF]

The interviewer is full of themself if they think someone who can do the job cheated in the interview.

[yieldcrv]

Tech hiring is way too dicey right now to give af, and its what I would do on the job anyway, most likely a local ai when company code is involved

[smcin]

Is this junior/intermediate software engineer, or what? What sort of questions? CS exam-type, definitions, whiteboarding, programming, LeetCode, numerical problems, algorithm, data structures...? Programming-language certifications? Riddles?

[jurynulifcation]

[moshun]

To be fair, you were likely already getting out competed by people with better connections or social skills anyway. Years in corporate leadership has cleansed me of the notion that merit is required to be a major factor in hiring decisions.

[mattbaker]

I don't think your comment is in line with HN guidelines, you might consider making edits.

https://news.ycombinator.com/newsguidelines.html

[m1el]

Oh, and to add an insult to the injury, I was using a collaborative editing tool. So I was able to see the person:

1) Select All (most likely followed by the copy) 2) Type the answer 3) Make an obvious mistake when they type else block, before the if

[willsmith72]

i have a really annoying habit of constantly double-clicking to highlight whatever i'm reading or looking at.

i've actually been called out for it in a systems design interview, under the presumption i was copying my notes into another window, but was glad they called me out so that i could explain myself

[storyinmemo]

... as I'm reading through this doing my normal random highlight of text while I read...

[eredengrin]

Same. I sometimes use Edge when a site is broken on Firefox and I get into trouble there because it has super weird behavior when you highlight text. Very annoying.

[frabjoused]

That was me interviewing someone yesterday. The telltale select all is so cringe.

[pests]

Some people compulsively highlight what they are reading.

[bitzun]

I compulsively left and right click random shit all day. It helps me encounter bugs like steam locking up for a few seconds on Linux if you right click the steam windows or overlay too quickly.

[jcranmer]

I'm a compulsive highlighter too, but it's generally in the vein as xkcd (https://xkcd.com/1271/) and not a select all. Frequently, highlighting ends up starting in the middle of a word!

[anonymouskimmer]

On some modern websites I end up having to select all because the selection mechanism is broken (I'll highlight, remove my finger from the mouse button, highlight another piece of text, and yet the original highlighted text is still totally, or even worse partly, highlighted.). Crtl-A Selecting all and then clicking anywhere is the only way to clear all of the highlighting in these instances.

Thanks for the XKCD. I didn't realize how common this is. Now I'm even more annoyed that so many websites and reader apps force context menus or 'gestures' when you highlight, without a way to disable those context menus or gestures.

[m1el]

I too select the lines that I read. However, I never select the entire page, unless I intend to copy it.

[frabjoused]

Oh, I do too. But then his monotone typing out of the answer, eyes darting back and forth between two screens, it’s kind of obvious. The select all just starts me looking.

[baby_souffle]

Exactly.

Ever go to school with a dyslexic that's using a ruler to expose one line of text at a time? Same thing....

[Kim_Bruning]

I just selected your reply here while reading. Some people use mouse selection as a visual aid to keep track of where they're reading. It's there, and it's handy!

(I also just select randomly sometimes. Not even quite sure why.)

[anonymouskimmer]

> (I also just select randomly sometimes. Not even quite sure why.)

I'm fidgety in general. If it isn't highlighting it's figure 8s with the mouse cursor.

[8organicbits]

> Maybe in the future everybody is going to use LLMs to externalize their thinking. But then why do I interview you?

It will become a skill. In 1900 you'd interview a computer (a person who does math) by asking them to do math on paper. Now you'd let them write some code or use software to do it. If the applicant didn't know how to use a (digital) computer, you'd negatively rate them.

I don't love it, but we may reach the point where your skill at coaxing an LLM to do the right thing becomes a desirable skill and you'd negatively rank LLM-illiterate applicants.

Looking at LLM quality, we're not at that point for most fields.

[appleiigs]

You're not asking the correct questions as an interviewer. You should be asking specific questions about projects they've worked on, or about them personally to get to know them. ChatGPT should not be able to answer. Pretend you're Harrison Ford in Blade Runner.

[makeitdouble]

You ask many kind of questions.

A candidate can do very well on personal and web project experience questions, and suddenly blank when you ask them how an http request is structured. Or what's CORS.

Then you dig further and discover a lot more thing about them that wouldn't have surfaced otherwise because hou assumed they knew all of that.

My best advice would be to never skip "dumb" and easy technical questions. You can do it very quick, and warn ahead that it's dumb questions but you ask them to everyone.

[ListeningPie]

Knowing the structure of an http request and CORS is a check for a common technical vocabulary, but I would strike a blank when asked directly. It feels a bit a like, “I had to learn it” even though it’s just googleable labels for simple topics. I heard of interviewees being dropped for not knowing the difference between 402 and 401.

[makeitdouble]

I think blanking is actually OK. From there you could probably explain what you know about it, how it was set in your project, or any peripheral story that comes to mind at that time.

I see it as a different angle to get more information.

[throwaway2037]

I agree with the "drill down" technique. Example: How does a dynamic array class (Vector, List, etc.) work? The very best interview questions have "fractal complexity" that allow you to drill deep.

[dmazzoni]

You can't only ask those questions, because some people are extremely good at bullshitting.

I always start interviews by asking them to explain their own projects. However, sometimes I'll find someone who's great at explaining projects they supposedly worked on in great detail, but then when given a simple coding problem they can't even write a for loop in their own top language.

[mvdtnz]

Chatgpt can easily be instructed to tell a tale about a project it has worked on. It will expand on fake details when pressed.

[Kranar]

As an experiment I gave ChatGPT my resume and background information and then pretended to interview it, just to see how well it would be able to conduct a mock interview. It did exceptionally well.

I'm not sure what specific questions you have in mind, but ChatGPT is almost certainly trained on a vast array of resumes and a diverse range of profiles, possibly even all of LinkedIn itself as well as other job boards. There is little to no reason why it wouldn't be able to make up an entire persona who is capable of passing most job interviews.

[tasty_freeze]

One red flag for me is when the interviewee gives "cork" answers -- the metaphor is that of a cork bobbing in the water. If you ask superficial questions about work they've done, the answer it convincingly. But the further down you go into the details the more resistance you get and the cork keeps bobbing up to the surface level.

[smcin]

'intentionally superficial and vague'

[fragmede]

You want me to explain my role in the tortise flipping app that had a dating feature for lesbians?

[nyc_data_geek1]

Using LLM's isn't externalizing or outsourcing thinking. LLM's aren't performing that. People doing this are in fact substituting thinking with a process, the output of which masquerades as thoughts after a fashion, but are in fact basically word cloud probability based pattern matching.

Sure, the point that superior tool use is a valid job skill makes some sense, but conceding your agency and higher reasoning to a machine which possesses none of these is to my mind not going to be beneficial to a business in the long run.

[osigurdson]

Perhaps interviews need to assume the person being interviewed is using an LLM and can be evaluated on how effective they are with it. Presumably this is what employers want. The challenge is interviewers are busy, would prefer to be doing other things and want to stick to their old playbook ("tell me how to invert a binary tree").

[kfk]

Another take is we don’t like being lied to. Lots of these ChatGPT job candidates don’t disclose they are using an AI during the interview.

[osigurdson]

My suggestion is ChatGPT should be part of the interview.

[TrackerFF]

Yes, agree.

If it is out in the open, with the chat/prompts available, you can ask other questions. You're not on your toes trying to catch a cheater. You're not assuming that the interviewee is lying or trying to scam you.

[wakawaka28]

No, it's not what they want. If they wanted you to use a LLM then they would tell you that up front. It's also too new of a technology to be required anywhere. Hardly anyone I know is even trying LLMs to begin with. Then, what do you do if the interviewee gets garbage code out of the LLM and misses an error? An error that might be forgiven in a normal interview cannot be excused when you didn't even have to write the code. Technically, if the LLM did the coding for you, you might pass without even being able to read code. This is all like the same reason you can't use a laptop on an algebra exam... The tool might do 100% of the work and leave you having shown nothing of your own ability.

[osigurdson]

>> No, it's not what they want

It may not be what the interviewer wants, because they would like to keep using their old interviewing strategies. However, it is what the business wants (or should want, assuming they want the most effective employees).

[wakawaka28]

It's not about being attached to interview strategies. It's about the fact that some people only copy/paste and aren't effective up to basic standards. I bet you'd consider an answer pasted from Stack Overflow and misrepresented as original to be 100% ok too, but both are unacceptable.

[osigurdson]

It is important to focus on the intersection of human and machine intelligence. If you listen to the AI luminaries, the role of the human will be more like a manager so perhaps understanding the code may eventually become unnecessary. However, my own experience with LLMs so far is they do seem to have trouble getting fine details correct. Presumably it will change over time.

[jliptzin]

You should just openly let them use chatgpt (assuming they can use it on the job too). When I interview people I try to create the same environment as the one they’ll be working in. They can use chatgpt, google, stack overflow, etc. I don’t care how many tools they have to use, as long as the work output is good and done in a reasonable time. I really don’t understand the obsession with coding on whiteboards or other situations that will literally never come up on the job. There will never be a time my employees can’t use google or chatgpt. In any case, you can tell pretty quickly how much someone knows about a topic just based on the questions they’re asking chatgpt.

[Terr_]

Whoah, hold up: Why should we believe that success using an LLM to (possibly blindly) look up the answer to interview-questions will strongly correlate to success using an LLM to craft good code, properly tested, and their ability to debug it and fit it into an existing framework?

Heck, at that point you aren't even measuring whether the candidate understood the question, nor their ability to communicate about it with prospective coworkers.

If there are any questions where "repeat whatever ChatGPT says" seems like a fair and reasonable answer, that probably means it's a bad question that should be removed instead. Just like how "I'd just check the API docs" indicates you shouldn't be asking trivia about the order of parameters in a standard library method or whatever.

[jliptzin]

Nothing I hire for requires someone to do the World’s Most Challenging ™ life or death problems under pressure from memory. I think that’s true for the vast majority of tech companies. If I need someone to wire up a database to a react interface, or write some cron scripts, or refactor an old nodejs codebase, that is all stuff that chatgpt would be a great tool to use. I don’t care whether they’re doing it from memory or not.

[Terr_]

> Nothing I hire for requires someone to do the World’s Most Challenging [...] from memory

That's a bit of a strawman: I didn't say anything about the ease/difficulty of the role being filled, and I implied rote memorization was not meaningful.

To reiterate, interviews should measure good data for choosing between candidates.

That's not happening when the given problem is solve-able by an LLM using a human as a proxy, everybody's just burning man-hours of company/applicant time on interview-theater that isn't useful for making a decision. (Well, not unless the hiring goals include "willingness to jump through hoops".)

[jliptzin]

And what if every problem the position I am hiring for is solvable by an LLM?

[recursive]

If they're just putting the question straight into GPT, then what benefit is the candidate bringing? I can use GPT myself, and for a lot cheaper than the cheapest candidate.

[kmoser]

If the interview is for a position in which the candidate will be tasked with solving problems that ChatGPT is able to help with significantly, then they have just proven they are capable of doing the job. (If you have time to do this work yourself, why are you interviewing anybody at all?)

Assuming the interview is to determine somebody's programming chops, without the benefit of ChatGPT, you'll have to ask questions where ChatGPT is little to no help. This was the conclusion of the article.

[seattle_spring]

But how will I know if they can implement a function to rotate a binary tree from memory?

[kmoser]

Why does it matter to you if they can do it from memory, if they can find the answer easily from ChatGPT? It's like asking "how can I tell if somebody knows the exact definition of a word without having to look it up?" If it's really important that they have that ability (e.g. because you will be asking them to perform other tasks which are not so easily solved by ChatGPT, or you simply don't want them using ChatGPT at their job for whatever reason), then you will have to devise an interview scenario where the candidate is incapable of using ChatGPT clandestinely, e.g. by bringing them into your office.

[seattle_spring]

I was being facetious.

[n_plus_1_acc]

You evaluate whether they ceitically review the LLM answer or just take it at truth.

[WalterBright]

At Caltech, exams were typically open book, open note. The time limit on the test, however, prevented attempts to learn the material in the time allotted. Calculators were also allowed (though were useless on Caltech exams, as course material didn't care about your arithmetic skillz).

I suspect the way to deal with ChatGPT is to allow it. Expect the interviewee to use ChatGPT as a tool. Try out the interview questions beforehand with ChatGPT. Ask questions that ChatGPT won't be good and answering, like how a calculator is useless on a physics exam.

[wakawaka28]

Using ChatGPT as a tool makes as much sense as allowing a human assistant to take the exam with you.

In an open-book test, you have to know what you're looking for and roughly where to find it in the book. That implies some knowledge. With ChatGPT you could type the question verbatim and get a potentially right answer, without even understanding the answer at all. It is therefore unacceptable for use on any exam.

[xarope]

As a former tertiary educator (for a brief moment, before I decided academia wasn't my thing), that's how open book exams are set; the assumption is you have knowledge of the subject, and the books are there for you to verify and quote examples of/from.

NOT to browse through looking for a solution from step 0.

[lmm]

> But then why do I interview you? Why would I recommend you as a candidate for a position?

Presumably you have tasks that you want performed in exchange for money? (Or want to improve your position in the company hierarchy by having more people under you or whatever).

[renewiltord]

That sounds great, doesn't it? You got powerful negative signal.

[lcnPylGDnU4H9OF]

It sounds like the problem is really that this is the most obvious cheater. Someone better at manipulation and deception might do a better job cheating the interviewer such that they're hired but then be entirely inadequate in their new position.

[JohnFen]

> Of course this will change in the future, with more interactive models

I think that what will change is that doing interviews remotely will become rarer, in favor of in-person interviews.

[scarlson]

Why?

Interviewing as a process sucks enough as it is. It should just be a culture fit filter that takes you all of 15 minutes to say yes or no to.

Technical interviews are lame and filter for people that are good at technical interviews, not people that are good at the job.

[plasticchris]

That works in a world where everyone is technically competent, but oddly many people applying to software positions are, optimistically, planning to learn on the job. Work with enough folks like that at once and the motivation for the coding interview becomes clear.

[LgWoodenBadger]

Lame? It’s a bare minimum demonstration of ability.

The number of experienced candidates I’ve interviewed just in the past few months who have trouble writing a for-loop in the language they’re “experienced” in might astound you.

Welders sometimes (always?) have to go to a certification center to demonstrate that they can actually perform the types of welds the job they’re applying for requires.

https://www.aws.org/Certification-and-Education/Professional...

[michael_mroczka]

You're 100% right, but I think your experience is different than recent job seekers. I think this is mostly semantics. You're asking simple problems and are amazed at the number of people that can't do them.

In the current job market, however, lots of places are asking ridiculously hard verbatim leetcode questions in an attempt to filter out "bad candidates." Job seekers feel that too many places ask unfair questions (which is true) and employers feel that there are too many candidates that can't write genuinely simple programs (also true).

[JohnFen]

> Interviewing as a process sucks enough as it is.

It truly does, and it sucks just as much for the employer as for the applicants. That's why I suspect that more interviews will be required to be in person: if it's too easy for someone to cheat, that makes everything suck even more for the employer and the employer is likely to adjust the process to minimize that suckage.

> Technical interviews are lame and filter for people that are good at technical interviews, not people that are good at the job.

Not automatically, but yes, bad technical interviews filter for people who are good at technical interviews. And too many interviews (technical or otherwise) are bad.

[batch12]

I've had this happen too, with almost the same responses. It was even more obvious because I was able to see the reflection of their lcd backlight glowing across their face as they switched back and forth to answer the questions. I just directly asked if they were using an external resource to answer my questions. They said yes as if it was normal. I thanked them for their time as that was my last question.

[outworlder]

> The interviewee sits there in awkward silence for a few seconds, and starts answering in a monotone voice, with sentence structure only seen on Wikipedia. This repeats for each consecutive question.

That's a bit better than proxy interviews and people lip syncing, but not by much.

[Espressosaurus]

This seems readily fixable by doing the interview in-person.

[smcin]

How much can you mitigate this by interviewing them remotely but on video? Then you can see if they're typing and reading the answer (unless they have a friend doing that and feeding them it in an earphone, as I hear happens).

[esafak]

Then you'd filter them by resume first to manage costs. Pick your poison.

[m3kw9]

Would get easier with an API that connects with stuff like whisper and voice cloning and a good prompt

[neilv]

> but people who use ChatGPT on the interviews make a disservice to themselves and

I think most people have been thinking that the interviews are mostly BS with little relationship to the job, which you simply have to get through.

Many, many people will cheat to the extent that they think they can get away with it.

It's a bit like many people cheat in school. (On classes they consider irrelevant, they might justify it that way. On classes relevant, they might justify it, that passing or their GPA is more relevant to their goals, than learning that material at that time.)

I think people generally don't believe a "you're doing a disservice to yourself" argument. They choose the tradeoff or the gamble.

Personally, I don't tolerate cheating, and I have a low tolerance for interview BS. Neither is the dominant strategy for the current field.

[duxup]

I’ve wondered how much of the appeal of LLMs is for humans to BS other humans.

[foxyv]

Considering how much time is spent on manufacturing BS for consumption by bosses, professors, teachers, and advertising? I think this is going to automate at least half of the work office workers and students are doing now...