[Animats]

“We don’t know how our AI systems work, we don’t know what they can do.”

That's not a joke. That's a real problem with the current versions of machine learning. You've got a huge collection of weights and no understanding of what they mean.

We've reached a strange place. Large language models have blown through the Turing Test. Yet this isn't AI. What we have is something that generates plausible-looking but not consistently correct text or images. Large language models have mechanized the Dunning-Kruger effect.

It's striking how plausible the output is. What this may demonstrate is that well-written text is mostly an average of a large body of text. This is similar to the discovery that if you average a large number of faces, you get a very good looking one. It's a discovery about the human perception system.

This may be a transient situation. Researchers are starting to figure out that inside those huge collections of weights, models sometimes emerge. Something that works like understanding or common sense may turn up in there. That may take a while.

Meanwhile, though, the business potential of automated blithering is going to result in this stuff being used for far too much.