There is a strong evidence that they might have been driven away by the Islamic invasions, because their timelines coincide.