It is possible it is just time. Modern humans are considered to have existed for 300k~ years. Civilizations are about 6k years old.
So who knows. Maybe if you gave them an extra 10k years, they would have achived "civilization". It is not much for the scale of human existance. But it is longer than any of our civilizations has existed for.
And makes sense with agriculture and civilization coming together. Agriculture improved the carrying capacity of an acre of land dramatically from what it was from foraging and hunting.
I think that (Incas) was more numerical record keeping rather than written language.
What's special about written language (which some existing hunter-gatherer tribes still don't have), as opposed to spoken language, is that it allows cultural knowledge to be spread, stored, accumulated, and built upon.