| I suppose the hinge point is if algorithmic improvement + compute capacity is or exponential improvement. I flip/flop between the options as I listen to the Moonshot Podcast bros (lols because of obvious over-hype) and Lex Friedman's state of 2026 AI with Nathan Lambert and Sebastian Raschka. If exponential, then yes I see more variance humanity's short/medium term outcome. More chance of doom, more chance of boom. If logarithmic, well then improvements continue, but they will be incremental to what we have now and we will continue into some form of widespread agent adoption. Similar to wireless communication technology is prolific, but now taken 100% for granted. Personally, even with Sol, the results I get from "stress test" one-shot prompts are abysmal. Hallucination of solutions to non-existent bugs as popped up multiple times. So AGI take over, meh, low chance IMO. I think one real risk is solo or sleeper cell type nefarious actors having a semi-competent engineering consultant which can enable small-scale weapons deployment. Overall, I think the most probable short term risk is AI companions, hyper individualization and deterioration of social institutions. Much less flashy, but much more likely. I mean look at the effects of social media on mental health. And that was only created 1 generation ago. |
20 years ago computers basically couldn't do voice or image recognition at all. 10 years ago it worked, but was still very flakey. Today it's super human.
I agree that if progress flattens out very rapidly and very soon then we'll at least have more time, but right now everything would suggest progress is still exponential and that flattening will now need to come quick and hard. This is largely why I've significantly increased my doom probability over the last year. I don't think there's any reasonable progress curve you can draw at this point that would suggest ASI isn't coming very soon. I suspect our only hope is nuclear war or a long tail good post-ASI outcome.
In regards to Sol, I think it's worth remembering that people were saying just a couple of years ago that AI can't even count the number of letters in a word or do basic maths. The fact the bar has risen so much we're saying they're still meh because they're not one-shotting problems is at least worth noting.
> I think one real risk is solo or sleeper cell type nefarious actors having a semi-competent engineering consultant which can enable small-scale weapons deployment.
Yep, the problem with AI isn't that there's any one risk that anyone can reasonably predict with high probability, but that there are many potential risks with signal digit probabilities. When you consider the overall risk landscape, that's when things get worrying.
> So AGI take over, meh, low chance IMO.
Right now I'd put this at like 10-20% probability. It seems unlikely to me, but quite possible if AIs of the future develop their own goals and are sufficiently super human.
The biggest risk is collapse I think. I don't think the world is stable in a post-ASI world. When the magic wish box will grant me whatever wish I like, I might wish something bad. In my mind this is the biggest risk by far.