[moffkalast]

Agreed, for me it's on multiple occasions fixed some of those errors that are syntactically correct but can't be seen by the IDE or the interpreter/compiler, like forgetting to change some variable when copy pasting a section or calling the wrong function of the correct type. Stuff that can only be spotted when thinking about code semantically.

GPT 4 is also a serious CSS master, so that cuts so much time from trial and error there.

Current drawbacks though:

- the 2021 cutoff is very apparent, it's terrible at newer stuff since it can't pull from many examples (browsing mode helps, but it usually fails at finding the info it needs)

- it really can't help with the typical workflow of editing some small thing in a huge codebase because there's no way you can give it enough context for an answer that isn't based on heavy speculation

- when doing too much back and forth it eventually starts to cut tokens and no longer knows what the original question was; sometimes it's not an issue but other times it just goes off topic

[lakySK]

Definitely experienced all 3, but still getting more boost than drag overall.

The large codebase context might be somewhat solvable and I've seen projects that use embeddings to find the relevant bits of code to feed GPT to help it with context. No clue how well any of them work though, haven't tested them yet.

I've definitely noticed times when the conversation gets cut off and it can't "remember" the previous messages. Often, it results in a loop of ChatGPT giving me a solution, me getting an error and sending it back, then ChatGPT being terribly sorry and suggesting a new solution. Repeat 3 times and often we make a full circle to the first solution in this way...

[dinvlad]

I know right? To me it feels like cheating in an exam, where I ask a know-it-all who gives a straight answer without me understanding it, but then it turns out that the know-it-all didn’t know it very well either, but just appeared confident, so we both failed the exam.

[lakySK]

If you treat LLMs as know-it-alls and just copy the output, then your expectations of the current generation of LLMs are too high.

That doesn’t mean they’re not useful though.

[dinvlad]

OK let’s say if I don’t, how can I make sure what I get is correct, even after a few iterations?

[hxugufjfjf]

You ask it to explain its reasoning step by step. This has also proven to yield more accurate results. Look into Chain of Thought

[dinvlad]

It’s not very good at explaining stuff though. One example: ask it to explain a subtle joke, and it will keep failing in funny ways.

This is not surprising though, as these kinds of models (LLMs) were specifically optimized for generation, not explanation.