[pfisherman]

LLMs and NLP are to verbal reasoning what the calculator is to quantitative reasoning.

Language and by extension verbal reasoning is full of ambiguity and semantic slipperiness. For example, what degree of semantic similarity distinguishes synonymous from synonym-ish concepts? When do we partition concepts into homonyms?

I think part of the problem with how people evaluate LLMs is that the expectations that people have. Natural language != ontology. The expectation should be more Chomsky and less Boole. Asking it to solve math problems written in paragraph form is a waste of time. Use a calculator for that! Solving riddles? Code it up in prolog!

Instead you should be thinking of what operations you can do on concepts, meaning, and abstract ideas! That is what these things do.

[missingrib]

Is this really verbal reasoning? It's just a logic problem.

[pfisherman]

How can one / should one combine the concepts of a dinosaur and monetary policy of the Ottoman Empire? What differentiates verbal reasoning from logic?

I don’t know that either of those can be solved well with formal languages or logic.

[pfisherman]

Follow up in this one… I asked an LLM to give me the funniest way to combine the concepts of a dinosaur and monetary policy of the Ottoman Empire. This was the answer.

Imagine a “Dinoflationosaurus”: a giant dinosaur who has the job of overseeing the monetary policy of the Ottoman Empire. However, this dinosaur is hopelessly behind the times, using outdated gold coins that are buried in random locations, like a prehistoric central bank.

Instead of regulating currency or adjusting interest rates, the Dinoflationosaurus spends its days stomping around, either hoarding or releasing massive piles of treasure based on whether it sees its shadow, causing huge economic fluctuations. Merchants and citizens scramble to predict where the dinosaur will dig next, turning the entire economy into a game of dinosaur-sized hide-and-seek with inflation spikes tied to the beast’s mood swings.

The Ottoman economists, dressed in traditional robes, nervously try to explain to the sultan that no one knows when the giant lizard will “stimulate the economy” by smashing a treasury vault open.

Try to code something like that up in prolog!

[samrus]

Yeah but the Cheryl's birthday problem doesn't have any ambiguity like that. It's all in very simple language, the only complexity is keeping track of states of mind, which is easy to abstract away from the language

[pfisherman]

That is exactly the point I was making in my comment above. This type of unambiguous problem is best solved using formal languages - something more like quantitative reasoning. But stuff like prolog or classical automated reasoning approaches are quite brittle. They break down quickly when you start to introduce ambiguity and noise. Statistical approaches like hidden markov models that people used in these instances were the precursor to the LLMs we have today.

But I was going down a rabbit hole there. My main point is that trying to use LLMs to solve logic puzzles - that can easily be solved in prolog - is a waste of time and a failure of the imagination. The applications that should be explored and would be most fruitful are those where there is ambiguity and contradiction.