[SkyPuncher]

I'm curious were you in an existing code base or a greenfield project?

I've found LLMs tend to struggle getting a codebase from 0 to 1. They tend to swap between major approaches somewhat arbitrarily.

In an existing code base, it's very easy to ground them in examples and pattern matching.

[ivan_gammel]

Greenfield. It’s an interesting question though, if on today‘s project some model will perform better tomorrow because of more reference data. I would expect LLMs to lag behind on latest technology, simply because their reference data has more older examples and may not include latest versions of platforms or frameworks. I have seen LLMs breaking on basic CRUD tasks because of that.