[theonlybutlet]

You're underestimating the power of LLM's.

I'll address two of your points as the other two stem from this.

They can't backtrack that's purely just design and can be easily trained there's no need to simulate at random until it gets the answer, if allowed to review it's prior answers and consider this, if often can reason a better answer. Further more breaking down problems. This is easily demonstrated when looking at how accuracy improves when you ask it to explain it's reasoning as it calculates (break it down into smaller problems). The same for humans, large mathematical problems are solved using learned methods to breakdown and simplify calculations into those easier for us to calculate and build up.

If the model was able to self adjust weightings based on it's finding this would further improve it (another design limitation we'll eventually get to improve, reinforcement learning). Much like 2+2=4 is your instantaneous answer, the neural connection has been made so strong in our brains by constant emphasis we no longer need to think of an abacus each time we get to the answer 4.

You're also ignoring the emergent properties of these LLMs, theyre obviously not yet at human level but they do understand the underlying values and can reason using this value. Semantic search/embeddings is evidence of this.