[ShredKazoo]

Part of what's going on here is that AI researchers are paid to tinker with systems and make them work, not think about these kind of big picture questions. A demonstration of this is that if you ask a big picture question a bit differently, you can get a very different answer. Example:

>...if you ask [AI researchers] for probabilities of things occurring in a fixed number of years, you get later estimates than if you ask for the number of years until a fixed probability will obtain. This looked very robust in 2016, and shows up again in the 2022 [human-level machine intelligence] data. Looking at just the people we asked for years, the aggregate forecast is 29 years, whereas it is 46 years for those asked for probabilities. (We haven’t checked in other data or for the bigger framing effect yet.)

https://www.lesswrong.com/posts/H6hMugfY3tDQGfqYL/what-do-ml...