[zorr]

I'm not saying it will reach Carmack levels of proficiency any time soon but have you tried pasting a non-trivial method into ChatGPT and ask the AI what the method does and how the code can be improved? I was very impressed about the way it was able to explain the code and suggest improvements.

[cjk]

Not myself, but I've seen a bunch of examples folks have posted on Twitter, and to be fair, they were all surprisingly accurate.

As far as I understand it, though, these explanations are based on similar code contained in the training dataset, rather than any kind of ability to reason about what's actually going on, no?

[zorr]

They are indeed based on similar code in the training dataset, but it also takes into account the context of the code in the conversation thread. While the initial code it generates is already impressive and pretty accurate. To me, the real power comes out when iterating on the code through followup conversation.

For example, I asked it to write an implementation for a piece of Kotlin code I provided describing an interface for a CRUD TaskRepository with typical `addTask`, `removeTask`, `getTask(id: UUID): Task?`, `getTasks(parentId: UUID): List<Task>` methods. It generated a TaskRepositoryImpl class backed by a `Map<UUID, Task>` and all method implementations where correct, including checking a projectId property in the getTasks method. Note that I never gave it the code for the Task class.

Then I iterated on it by asking "Add save and load methods that store state in a JSON file" and it did just that: Introducing the Gson library, serializing the backing map to a JSON string before writing to the file, loading the file again and deserializing into the map.

After that I asked "Add a SharedFlow<TaskEvent> that publishes events when creating, removing and updating tasks" and it was able to infer where to emit events, providing sensible names like `TaskEvent.Created`.

In another test case I asked it to write an Android layout xml for a login screen with username, password, login button and login spinner which it did. I iterated once asking it to use components from the Material library and to make sure the Spinner was overlaid over the form. Then I followed up with "Write the corresponding activity code in Kotlin". It generated a full LoginActivity class that loads the layout xml, findViewById for the fields and buttons, input validation with error handling, show/hide the login spinner when logging in, starting the MainActivity when login succeeds and it even mocked the login method with an artificial delay of 2 seconds. Its explanation of the code described that the delay was there to simulate a login, while in real code you would replace this with an HTTP call or a call to a loginService.

And have a look at these snippets of conversation where I asked it to improve a dueLabelString method I provided. https://imgur.com/a/ZtViC3d

Is this "reasoning about the code"? I don't know and it's hard to say.

[cjk]

Damn. That is impressive.