|
It's a good question. In short, yes, I believe it is reasonable to assume custom questions will generally perform better than verbatim or modified leetcode questions. In general, while ChatGPT could handle modified questions well enough for interviewees to pass their interviews, it still choked through them and required more coaxing to get an answer than verbatim questions. Custom questions, by definition, aren't available online, and with no direct tutorials to pull from, the LLM has to make more inferences about the problem and will find the question more challenging. As for asking ChatGPT novel DS&A questions, I think it is harder than maybe you'd think it is. Any question you'd think to ask likely has a tutorial for it online somewhere, so unless you happen to make up questions like this professionally (I'm paid to do this), or you have a large unique question bank that doesn't exist online (few people have this) then my instinct would be that the questions you're giving it aren't as unique as you think they are. As a practical example, just give it a log file and tell it to pull out specific pieces of information from it. ChatGPT struggles to write code that can dynamically check obvious boundaries for humans. Recently, I had a list of times in a CSV that I asked it to pull for me ("1pm", "3pm", "9am"), and it wrote code to just grab 3 specific indices from the string. It didn't consider the need to check for 4 indices ("10pm"). It didn't think to start the check based on where commas were in the CSV, and it didn't consider looking for "am" or "pm". It just sliced a specific set of indices in the string. That's mostly because it's used to getting questions working for specific examples, but fails when you ask it to incorporate simple broader tasks into the coding interview question. |
if a dumb person is using ChatGPT to cheat on an interview, it'll be easy to tell from the code given if it's a bad question thats being given.