[etienne618]

I disagree with your argument, especially for point 1: these systems are massively constrained. The hardware they run on is fragile requiring massive amounts of power and tightly controlled environments. They don't have any means of replicating themselves (it can't run on arbitrary systems). The datacenters also have massive bandwidth between nodes - even if you could run 'it' on all the personal computers and phones in the world, it will likely struggle. Sure we can compress recent llm's down to being able to run on consumer hardware - but these things cant introspect, reason or adapt. They are completely static models and very far removed from anything agi. A lot of the progress in compute power in the last few years also come from changing representation: moving from foat32 to float 16 and more recently to float8. The silicon itself can only get so much better. It's not super obvious to me that we will have chatgpt4 like models on consumer hardware soon let alone solve true agi. Why don't we have true level 5 self driving cars yet?

We cant even figure out how to simulate a flatworm - and the connectome is solved.

[gonehome]

It's a narrower argument that I'm making - it's not about LLMs or implementation (can put those details aside), it's about possibility that a superintelligent AGI could be created, there's nothing magical about biological intelligence that would prevent it.

How or how difficult, or when are all questions that follow from that first premise (that it is possible). I don't really make strong claims about any of the implementation details beyond it being possible. Though again, what we're seeing doesn't look like failure to me.

If you think it is possible though, then there's a strong argument that trying to work on alignment now is probably a good idea because people are notoriously bad at predicting when advances will happen (and the downside risk of unaligned superintelligent AGI is likely very bad).

https://intelligence.org/2017/10/13/fire-alarm/

> "Two: History shows that for the general public, and even for scientists not in a key inner circle, and even for scientists in that key circle, it is very often the case that key technological developments still seem decades away, five years before they show up.

"In 1901, two years before helping build the first heavier-than-air flyer, Wilbur Wright told his brother that powered flight was fifty years away.

"In 1939, three years before he personally oversaw the first critical chain reaction in a pile of uranium bricks, Enrico Fermi voiced 90% confidence that it was impossible to use uranium to sustain a fission chain reaction. I believe Fermi also said a year after that, aka two years before the denouement, that if net power from fission was even possible (as he then granted some greater plausibility) then it would be fifty years off; but for this I neglected to keep the citation.

"And of course if you’re not the Wright Brothers or Enrico Fermi, you will be even more surprised. Most of the world learned that atomic weapons were now a thing when they woke up to the headlines about Hiroshima. There were esteemed intellectuals saying four years after the Wright Flyer that heavier-than-air flight was impossible, because knowledge propagated more slowly back then."

[etienne618]

I hear you - its just that I dont think that it neccesarily follows that an AGI in whatever form will necesarily be unconstrained or free to improve itself. I also happen to think that the biological systems are in-fact rather special and hard to replicate - especially in terms of efficiency and resillience. Just because some implementation of a universal turing machine can simulate intelligence doesn't mean it can do it well enough to survive the real world.

Finally, to me, nuclear reactions are kind of the opposite of AGI: I think it's vastly easier to blow something up (increase entropy) than to create an intelligence capable of understanding and improving itself (decreasing entropy - possibly at an accelerating rate).

[gonehome]

Yeah - it doesn't necessarily follow, but the behavior of people working on the technology doesn't inspire a lot of confidence.

Even if people were trying to constrain its access seriously I think that's unlikely to work (hard to contain a superintelligence that wants to not be contained - it's possible to trick a chimp to go into a room and the delta in intelligence between a human and a superintelligence is way bigger than us and chimps).

Instead I mostly observe people not really understanding the e-risk argument, focused mostly on small stuff that doesn't matter as much (AI language, bias). The people developing the tech connecting it to the internet and expanding capabilities, giving it access to code/training ability to write code, preparing massive datacenters for it, etc.

All of this without really understanding how to align it or what it's actual internal goals really are.

> "Just because some implementation of a universal turing machine can simulate intelligence doesn't mean it can do it well enough to survive the real world."

This could be true, but I would bet against it - and the downside risk of being wrong (potentially complete extinction) means it seems worth being way more cautious about it than we (humanity broadly) are observed being.

[righttoolforjob]

Nobody says that it necessarily follows, but it's a quite trivial step once you have the AGI, hence pointless that you bring up it as an argument.

[gonehome]

While I appreciate you find my argument persuasive - imo this style of comment will do more to alienate those replying to me that disagree than to persuade them.

[righttoolforjob]

They're likely irrelevant anyway. We're all shouting into the void.