[Tiktaalik]

From my experience the further you get from the sort of stuff that easily accessible on Stack Overflow the worse it gets. I've had few problems having an AI write out some minor python scripts, but yield severely poorer results with Unreal C++ code and badly hallucinate nonsense if asked in general anything about Unreal architecture and API.

[lubesGordi]

Does the Unreal API change a bit over versions? I've noticed when asking to do a simple telnet server in Rust it was hallucinating like crazy but when I went to the documentation it was clear the api was changing a lot from version to version. I don't think they do well with API churn. That's my hypothesis anyway.

[toshinoriyagi]

I think the big thing with Unreal is the vast majority of games are closed source. It's already only used for games, as opposed to asking questions about general-purpose programming, but there is also less training data.

[mac-mc]

You see this dynamic even with swift which has a corpus of OSS source code out there, but not nearly as much as js or python and so has always been behind those languages.

[Tiktaalik]

There would be significant changes from 4 to 5, but sadly I haven’t had any improvement if clarifying version.

[frankc]

Clarifying can help but ultimately it was trained on older versions. When you are working with a changing api, it's really important that the llm can see examples of the new api and new api docs. Adding context7 as a tool is hugely helpful here. Include in your rules or prompt to consult context7 for docs. https://github.com/upstash/context7