[Agraillo]

> Today's "artificial intelligence" analyzes words (tokens) based on an input (prompt) to come up with an output. It's predictable. It's fast. But, imho, it lacks creativity ...

I would have agreed with you at the dawn of LLM emergence, but not anymore. Not because the models have improved, but because I have a better understanding and more experience now. Token prediction is what everyone cites, and it still holds true. This mechanism is usually illustrated with an observable pattern, like the question, "Are antibiotics bad for your gut?" which is the predictability you mentioned. But LLM creativity begins to emerge when we apply what I’d call "constraining creativity." You still use token prediction, but the preceding tokens introduce an unusual or unexpected context - such as subjects that don't usually appear together or a new paradoxical observation (It's interesting that for fact-based queries, rare constraints lead to hallucinations, but here they're welcome)

I often use the latter for fun by asking an LLM to create a stand-up sketch based on an interesting observation I noticed. The results aren’t perfect, but they combine the unpredictability of token generation under constraints (funny details, in the case of the sketch) with the cultural constraints learned during training. For example, a sketch imagining doves and balconies as if they were people and real estate. The quote below from that sketch show that there are intersecting patterns between the world of human real estate and the world of birds, but mixed in a humorous way.

    "You want to buy this balcony? That’ll be 500 sunflower seeds down, and 5 seeds a day interest. Late payments? We send the hawk after you."

[aflag]

It's hard to pin point what creativity is. But in your example, the more creative thing was really coming up with the scenario of pigeons selling balconies as real state. What followed was just applying usual tropes for that sort of joke on the subject matter. I feel like LLMs are not very good at coming up with something novel. I'm not even sure they are capable of that. It's not as if coming up with something novel is easy for humans either.

[bsenftner]

Plus, a lot of people are generating hallucination and believing that is invoking creativity. I contend the outputs/generations are junk, but human creativity and human comprehension step in and create meaning to the hallucination.