[barrysteve]

Oh come on. I fed Unreal c++ engine code to ChatGPT4 and it couldn't understand inheritence in Slate classes and therefore kept offering me the same broken solution for a parameter with the wrong type.

The Unreal engine code is documented and publicly avaiable for OpenAI to ingest and it still gets the basics wrong.

I wasted hours trying to get it to explain to me what I didn't know, if it doesn't understand the internals of Unreal, I have no hope for it on bigger and better codebases.

It doesn't parse, it doesn't explain, it does not grok. It guesses at best and the blood sucking robot-horse is not telling the truth.

[ragequitta]

>It doesn't parse, it doesn't explain, it does not grok. It guesses at best and the blood sucking robot-horse is not telling the truth.

In my experience with coding (I've only done javascript and python myself) you have to tell it to explain and grok. It takes on the role you give it. Even just saying something like "you are a professional unreal developer specializing in C++, I am your apprentice writing code to (x). I want you to parse the following code in chunks, and tell me what might be wrong with it" before typing your prompt can help the output immensely. It starts to parse things because it's taken on the role of a teacher.

People love to hate on the idea of "prompt engineering" but it really is important how you prime the thing before asking it a question. The other thing I do is feed it the code slowly, and in logical steps. Feeding it 20 lines of code with a particular purpose / question you'll get a much better answer than feeding 200 lines of code with "what's wrong here?" You still need to know 90% of what's going on, and it becomes very good at helping out with that 10% you're missing. But for all I know it is just really bad at C++, that wouldn't surprise me. The things I'm using it for are definitely more simple.

[ksaj]

I do think this is why I sometimes get amazing results, and other times I have to go over a snippet of code so often I just give up and do it myself. It's a matter of how the question was asked in the first place.

Knowing that, it makes sense that your prompt should be as specific as possible if you want the results to be as specific as possible.

The best results I got was feeding it Lisp code that I wanted translated to C (to compile it). It took very little effort on my part because I described what each of the snippets did separately, and the expectation when combined and used together.

Through this, I learned that C doesn't have anything akin to the Lisp's (ATOM). ChatGPT stated clearly that its version of ATOM should only be expected to work in the code it was writing, but might not work as expected if copied out for another use of Lisp's (ATOM).

I asked it to give examples of where it wouldn't work, and it gave me an example of a code snippet that used (ATOM) that would not have worked correctly with the snippet that did work correctly with my original purpose.

Having said that, I myself learned that working with code function by function with ChatGPT, and being explicit about what you need, gives very good results. Focusing on too many things at one time can derail the whole session. One or two intermingling functions works great though.

[kazinator]

GPT4 works best when you assume that you're the professional dev with decades of experience, whereas GPT4 is a bright and broadly-informed co-op student lacking in experience in getting stuff working. You have to have a solution in mind, and coach it with specifics. And recognize the tipping point where it takes you more keystrokes of English to say what should be done, than keystrokes in Vim to do it yourself.

[barrysteve]

I did prompt engineer, using the 'you are an expert, desribe to student with examples' in many different variations.

In my testing prompts did not unlock an ability in GPT to grok the structure of code.

Empirical testing of LLM's is going to prove and map out it's weaknesses.

It is wise to infer from intution and examples what it can handle, leave the empirical map of it's capabilities to the academics, for the provable conclusions.

[JohnFen]

My observation (which could be wrong) is that ChatGPT as a programmer's aid is only useful for the simple cases. Not so much for complex stuff, and certainly not for something as complex as the Unreal engine.