When we talk about driverless cars replacing taxis we really mean programmers replacing taxi drivers in aggregate. The cars are a constant, if you're responsible for a fleet of them you are just a better taxi driver.
Such programmers will rely heavily on automated tools and will have to follow a clearly defined process because driverless cars will always involve human life. The cost of a simple misconfiguration is extremely high.
Besides, I fail to see how this can be considered manual labor or even the possibility of allowing average programmers to fill such positions.
We are the elite right now, our skillset is hard and the work we do is hard and we don't want to believe everyone can do it. Definitely not "average programmers" as you mention!
And maybe we have a little while longer to be the elite, but already children can build websites and applications so our days are numbered, soon everybody can know what we know and do what we do and as you mention, tools will be better as well and that empowers them to be closer to us too.
It takes a bunch of programmers to keep a fleet of servers running, vehicles will be maintained by programmers too, but it takes less programmers and less good programmers with every advance we make.
I think one of the nice things about programming, and one of the things that will ensure that programmers will always be in high demand, is that as tools improve, our desires and needs and aims adjust accordingly.
Just consider web development. There was a time you could make decent money with 'simple' HTML/CSS work. Clients were happy with that. Then you had to move to CMS-based sites, because people demanded the ability to add and edit content easily. Then you had to start offering plugins and more complex tools because everyone needed more than just a contact form to interact with their site visitors. And now everyone wants web apps, which are significantly more complex than simple websites.
I suspect that people who love programming and are good at it will keep moving up along the ladder of abstraction, and if anything there will be more and more need for experts at every rung of this ladder as companies get stuck with 'old' technology (for example, I could make a killing as a Drupal consultant).