I don't know if smart is the right word and I very much doubt IQ is the correct measure, but you do need to evaluate dozens of factors quickly and continuously while driving, always paying attention and updating your risk assessment. It's a mentally taxing exercise and many people can't do a remotely good enough job.
How is it a fallacy? There's literally a test you have to pass to get licensed and that's up to the discretion of a mandatory observer. There's more than one observer in most cases via the written proctor, the driving proctor, and the eye exam proctor.