[Tier3r]

The number of cases AI is used for is too vast to cover, but I do have the same experience. I've a few heuristics about assessing how well AI will work.

1) AI's performance decreases exponentially with the sparsity of relevant examples available on the web. The fields in which these are prevalent tend to be specific frameworks, libraries or hardware. AI may sometimes extrapolate from similar examples - for example many JS libraries are similar to others - but when it comes to, say, firmware libraries, it fails horribly. How relevant is relevant is a matter of some dispute. I find that it cannot transfer much beyond direct examples of library code. It's not necessarily a matter of bad documentation, even if the necessary docs are there, it has very poor ability to come up with a solution from first principles.

2) AI's performance decreases linearly with the number of different dependencies that piece of code connects to. Part of it is it doesn't have the context, part of it is that the chance of it seeing that particular configuration decreases exponentially with the number of variable configurations.

3) Bizarrely AI is terrible at recommending books. I was asking it for books on introductions to category theory and it made a whole bunch.

Where AI is useful is writing boilerplate, especially starting code which has minimal dependencies and plenty of examples online.

[pthr]

Thanks, those are very good points!

Intuitively it makes a lot of sense that asking for boilerplate code for Flask (to name something widely used and documented) these AI bots could perform way better than on something for which much less content exists, as with the scenario I described.