For what it is worth, numbers are not necessarily a 'universal concept'.
One of the most talked about examples is Piraha[0], a language which doesn't really represent numbers. This is interesting because a study done by Everett and Frank[1] shows that this can actually have effects on cognition, providing some evidence for the Sapir-Wharf hypothesis[3]
The Piraha language is probably inadequate for developing interstellar travel. As was English hundreds of years ago. You need numbers, and names for concepts like energy, mass, momentum, and time.
Numbers and math are required for theoretical space travel, but not necessarily for practical space travel. Catching a ball requires solving differential equations, but children and animals are capable of performing this action with no problem.
Given that there is so much of physics that we have yet to understand (assuming what we think to be true is actually true), and given the infinite possibilities of what forms alien life could take, you can't make any assumptions about what sort of language they'd have. This was a point they briefly tried to make in the movie.
Another way to do 'practical space travel' without 'theoretical space travel' is with tools. Just because you can't develop it with your language doesn't mean you can't use a tool someone else developed, or even an animal that evolved with the capability.
I don't know if that is true. I agree with you, but the obvious counterpoint is that a lioness has cubs and raises them to be apex predators just like herself, all without a language. You don't need a language to do things, there are many stories of people living just fine lives without language. To get to the stars, I'll bet on your hypothesis, but I think given enough time, some alien somewhere may have done the same without a thought in it's 'head' at all.
Larry Niven's "stage trees" - basically, naturally evolved rockets for seeds - spring to mind, where the space travel is an instinctive, unconscious behaviour.
But I'd bet on the OPs hypothesis as well. At a guess, most space travelling species will have had to climb a similar developmental curve to our own - including agriculture (or whatever passes for it in their biosphere, unless they photosynthesize themselves), tool making, metal smelting, nuclear power, etc. etc.
One of the reasons Everett is talked about so much is because his claims are controversial. Many linguists have criticized his methodologies. I have a background in linguistics but have never looked into Everett's work too thoroughly, but from what I have seen I am skeptical of his claims that Piraha lacks recursion. Still, an interesting line of inquiry and it would be nice if there were enough linguists to actually generate a wide body of research on the many endangered and minority languages. As it stands now, most languages are dying out almost completely undocumented.
AIUI, the controversy was generated based on definitions within generative linguistics that were ambiguous; the Chomskyans clarified what they meant by "recursion" for their purposes, and the controversy should have evaporated. That didn't stop the Chomskyans from getting Everett barred from conducting research among his friends the Pirahã or documenting their language. http://www.chronicle.com/blogs/linguafranca/2012/03/28/poiso...
If I had to boil it down to a sentence, Everett's claim ultimately was not that the Pirahã can't think recursively, it's that their language reflects a cultural aversion to referring to things that aren't concretely present, and this impacted their grammar for subordinating clauses.
The short of it is that Chomskyan hypothesis has a giant edifice built upon a single central idea. A fraca in popular science (i.e., outside academic journals) arises between Chomsky and Pinker over these ideas. Some unknown researcher announces that his obscure language brings the whole edifice crashing down. Add in a touch of Sapir-Whorf to make accusations of racism credible, and Chomsky's well-known fondness for dismissing critics rather than engaging them in scholarly debate.
Basically, there were two debates (did Pirahã have recursion in the sense that Everett meant? and what constitutes proof or disproof of universal grammar?) that went on simultaneously, while someone apparently took the opportunity to smear Everett with the Pirahã study ban.
As someone leaning towards Intuitionism myself, I believe in processes with operations that can be repeatedly applied any number of times without breaking themselves apart; such as applying the "successor" operator to "0", to "successor(0)"... to build any particular natural number, given enough time and memory.
I don't believe in the existence of a set containing all natural numbers at the same time, except as an idea to talk about.