[lubesGordi]

I know, I don't understand what problems people are having with getting usable code. Maybe the models don't work well with certain languages? Works great with C++. I've gotten thousands of lines of clean compiling on the first try and obviously correct code from ChatGPT, Gemini, and Claude.

I've been assuming the people who are having issues are junior devs, who don't know the vocabulary well enough yet to steer these things in the right direction. I wouldn't say I'm a prompt wizard, but I do understand context and the surface area of the things I'm asking the llm to do.

[Tiktaalik]

From my experience the further you get from the sort of stuff that easily accessible on Stack Overflow the worse it gets. I've had few problems having an AI write out some minor python scripts, but yield severely poorer results with Unreal C++ code and badly hallucinate nonsense if asked in general anything about Unreal architecture and API.

[lubesGordi]

Does the Unreal API change a bit over versions? I've noticed when asking to do a simple telnet server in Rust it was hallucinating like crazy but when I went to the documentation it was clear the api was changing a lot from version to version. I don't think they do well with API churn. That's my hypothesis anyway.

[toshinoriyagi]

I think the big thing with Unreal is the vast majority of games are closed source. It's already only used for games, as opposed to asking questions about general-purpose programming, but there is also less training data.

[mac-mc]

You see this dynamic even with swift which has a corpus of OSS source code out there, but not nearly as much as js or python and so has always been behind those languages.

[Tiktaalik]

There would be significant changes from 4 to 5, but sadly I haven’t had any improvement if clarifying version.

[frankc]

Clarifying can help but ultimately it was trained on older versions. When you are working with a changing api, it's really important that the llm can see examples of the new api and new api docs. Adding context7 as a tool is hugely helpful here. Include in your rules or prompt to consult context7 for docs. https://github.com/upstash/context7