[CSMastermind]

Candidates generally use it in one of two ways: either as an advanced autocomplete or like a search engine.

They'll type in things like, "C# read JSON from file"

As opposed to something like:

> I'm working on a software system where ... are represented as arrays of JSON objects with the following properties:

> ...

> I have a file called ... that contains an array of these objects ...

> I have ... installed locally with a new project open and I want to ... How can I do this?

No current LLMs can solve any of the problems we give them so pasting in the raw prompt won't be helpful. But the set up deliberately encourages them to do some basic scaffolding, reading in a file, creating corresponding classes, etc. that an LLM can bang out in about 30 seconds but I've seen candidates spend 30 minutes+ writing it all out themselves instead of just getting the LLM to do it for them.

[neonsunset]

GitHub Copilot Edit can do the second version of this. It is pretty good at it too. It sometimes gets things wrong but for your average code (and candidates typing in "C# read JSON from file" are way below average unless they never written in C#), if you give all the files for a specific self-contained part of the program, it can extend/modify/test/etc. it impressively well for an LLM.

The difference compared to where we were just 1-2 years ago is staggering.

Edit: the above is with Claude-3.5-Sonnet