Nobody knows if we are going to "just" be riding bikes for a long time.
To give time for society to adapt I hope it's the case, but we really have no idea.
Right, but if your real assertion is “we have no idea”, it seems you should point your skepticism significantly more towards the people betting $100 billion dollars that self-driving cars are coming next year than the ones who aren’t.
Evidence towards aside, the risk calculation depends on how much upside is from the 100B if progress starts the same rate. I think it's hard to bound that accurately
It looks pretty clear LLMs don't get us there by themselves, no amount of duct tape, WD-40, etc. gets us past, say, the mathematical certainty of hallucinations.
I mean, we don't know it any more than we don't know someone won't come out with cold fusion tomorrow, but it's a fundamental breakthrough away from where we're at. This isn't some routine engineering project with a guarantee of completion if you're just willing to keep pouring the billions. That's playing the lotto, you can pour away and get flat nothing.
The only difference is they're pouring billions and praying a rabbit comes out of the hat, but it's actually not much reason to expect they're going to pull the cold-fusion level rabbit out of their hat they'd need to get us past bikes.