[thrtythreeforty]

I fed it some physical modeling papers I've been reading to implement a software synthesizer ([1] if you're curious).

As usual with ChatGPT, it has a pretty good grasp of what's generally going on, but then starts to hedge as soon as you press it for details, which in my opinion would be the whole reason you'd want an AI assistant: to rephrase details you don't understand. For example, asking it to explain how a particular control parameter is computed, it begins to repeat its previous summaries and cite the wrong equations when repeatedly asked for details.

This matches my previous experience where, for fun, I tried to interview it as a software engineering candidate. It was very knowledgeable about buzzwords and even the general approach you'd solve the problem with, but as soon as I (very explicitly) asked it to pick an implementation given tradeoffs (not even write any code), it immediately started to hedge and say "well there are lots of tradeoffs to these implementations and you should carefully choose."

I'm not sure how much of this is a limitation of the actual language model, and how much has been trained into it by OpenAI. (e.g. hedging may just be the lowest-average-loss way to complete a prompt.) The overall effect reminds me a little of https://xkcd.com/451/.

[1]: http://lib.tkk.fi/Diss/2000/isbn9512251965/article3.pdf