This is far from being the only or even main explanation to their extinction.
The Neanderthal populations were extremely inbred, so I'd guess that was a bigger factor to their decline.