[sometimelurker]

we should never make ASI, what I am saying is strongly sane. ASI = not good. AGI, ~human brainpower, can be made safe enough.

> Are there examples of where we have collective decided not to pursue knowledge? Successfully?

human GMO, some bioweopns, I'm sure theres a long list of awful stuff no one wants to exist.

[voidmain]

If AGI means that AI+robotics can robustly substitute for human labor, and robots are cheaper and faster to build than humans, then (a) anyone ruthless enough can zerg rush and defeat any nations that don't discard humans, (b) no one without a massive robot army will be needed in any way by their rulers. If this isn't a recipe for a horrific outcome, what is?

[sometimelurker]

> defeat any nations that don't discard humans

AGI doesn't do away with nuclear MAD, it just messes with economics and makes many people temporarily jobless. Temporarily because in a literal sense RLVR needs verification to train off of, and a lot of jobs cant be easily checked if theyre done. this includes AI safety people, preschool teachers, psychologists, and probably a lot more, including most of their bosses

[voidmain]

Nuclear weapons are mostly a threat to cities, and a zerg rush build would just not build any.

I think that if all we get is skill cloning for formally verifiable tasks, that is not "AGI" and does not doom us. It probably won't even doom software engineers, because their whole job is also not formally verifiable. I don't trust that the next ten trillion dollars of AI research won't give us data efficient continual learning and agency on par with humans, which is what I would call AGI.

[throwaway27448]

It's not even clear that ASI is a coherent concept.

But, I don't trust capital with either.

[sometimelurker]

LLM + scale = more intelligent, this can be proven more than empirically, https://arxiv.org/abs/2409.15318 shows that neural nets can fit a number of independent AND-gate operations in their weights,

- if you have a system that is large enough to store, lets say 10^12 AND gates (all frontier llms can do this) - and this system can produce outputs based on previous things it has outputed

its turning complete, and RLVR on it is optimization over the space of algorithms. If an algorithm exists to do a task, and the task can be verifiably done, this finds the algorithm to solve the task most often.

it is obvious that this scales, from much-worse-than-human to slightly-worse-than-human, therefore it 100% can exceed humans.

[ajyoon]

What about it is not coherent?

[throwaway27448]

It's not clear that "super intelligence" is a meaningful concept. This presumes that our concept of intelligence can continue to grow beyond human capacity as opposed to asymptotically approaching what humans recognize as "very intelligent". Perhaps, for instance, how we evaluate intelligence is bounded not by some quantitative capacity but rather our inability to agree on basic concepts/values. And do we even have tasks that a supposed superintelligence can tackle but humans cannot?

I predict that what we consider "super intelligence" is just sheer computational power, but any potential of a very capable agent is bounded by the needs/wants of the person wielding it. That is: even if we were to hand, say, Elon musk this "super intelligence", most humans would consider it relatively stupid because the person wielding it is still a person with stupid goals and values.

Or, to put it another way, I suspect we already do have a superintelligence and have longer than any of us have been alive, and it's just "the market", and it is still incapable of overcoming the limitations of a few morons wielding immense power.... power they will never yield to some intelligence with values and goals "more intelligent" than their own (if such a concept is even meaningful), and intelligence wasted on the values and goals they do have.

[brookman64k]

The Orthogonality Thesis (by Nick Bostrom) says that intelligence and ultimate goals are independent. Those non-instrumental goals can’t be stupid nor right or wrong. Increasing intelligence will not change the goals only the capability to reach them.

[aesthesia]

Do note that the orthogonality thesis is a hypothesis, not something we have demonstrated. Weak versions of it (e.g. it is possible to have an intelligent agent with arbitrary goals) are more likely to be true than stronger versions (e.g. intelligent agents we build will have goals uniformly selected from the space of all possible goals).

[throwaway27448]

Ok so what is the point of neurotically obsessing about intelligence? Would it not be more fruitful to emphasize computation?