[uberman]

I was of the same mindset and did a similar test that also failed spectacularly. I asked for authoritative pros and cons on a subject in my field and asked it to cite sources. The pros and cons were ok, but the sources completely made up.

So, not so great there.

Then recently I had to modernize a python app written by someone who no longer works for the organization and was circa python 3.6. Several of the key libraries no longer supported the interfaces being called and there was no documentation at all.

On a whim I asked an LLM to help modernize the code, file by file and it cut the effort in half.

So, pretty great there.