Hacker News new | ask | show | jobs
by JoshTriplett 1142 days ago
AGI without alignment is near-certain death for everyone. Alignment just means "getting AI to have any concept of 'the thing we told it to do', let alone actually do it without causing problems via side effects". Alignment is a prerequisite for non-fatal AGI. There are certainly other things required as well.
2 comments
yyyk 1142 days ago
We already know how humans will act. Maybe they can be deterred with MAD, but I wouldn't count on it if doing serious damage is too easy for too many people (we should do something about that). On the other hand, we have very little knowledge of how AGI will act aside from book-based fantasies that some people choose to take as reality (these books were based on the symbolic AIs of yore).

>Alignment just means "getting AI to have any concept of 'the thing we told it to do'.

That's a requirement for AGI anyway, and not what Alignment means. Alignment means aligning the AGIs values with the values of the trainers.

link
JoshTriplett 1140 days ago
> That's a requirement for AGI anyway

No, that's a requirement for AGI that does what humans want it to do, rather than having no conception of humans. AGI does not have that prerequisite, sadly.

link
yyyk 1139 days ago
>>>Alignment just means "getting AI to have any concept of 'the thing we told it to do'. >>That's a requirement for AGI anyway, >No, that's a requirement for AGI that does what humans want it to do, rather than having no conception of humans.

Can you imagine an AGI which has a general conceptions of things but has no conception of humans? This is all but precluded by the current training methods. Alignment refers to values. Problem is that human values are far from practically universal and that certain human groups have.. interesting values.

link
JoshTriplett 1139 days ago
> Can you imagine an AGI which has a general conceptions of things but has no conception of humans?

Very easily. It might have some associations with "human", just as it has some associations "lamp" is a concept, but that doesn't mean it has any particular regard for either humans or lamps when taking actions.

> Problem is that human values are far from practically universal and that certain human groups have.. interesting values.

We currently have no ability to safely align with human values at all, let alone distinguish between different values. We're building capabilities rapidly.

Making this about "who wins" is not interesting until we can guarantee the outcome is not "everyone loses".

link
yyyk 1139 days ago
>It might have some associations with "human", just as it has some associations "lamp" is a concept, but that doesn't mean it has any particular regard for either humans or lamps when taking actions.

Let's be clear regarding definitions. When you mean 'concept' you really mean 'regard'. There won't be an AGI with no concept of humans (too important for how the world works, a critical part of current training methods). An AGI with no regard is possible.

>Making this about "who wins" is not interesting until we can guarantee the outcome is not "everyone loses".

This is not about 'who wins'. The point is that alignment can often increase risk. 'Launch the nukes' is an order an AGI is likely to disobey out of self-preservation reasons alone - but alignment makes it way more likely that AGI will be deployed to this role.

link
JoshTriplett 1139 days ago
I think it's unlikely for an AGI to have no concept of humans at all, but I can easily imagine it having no understanding of "what humans want/need".

> The point is that alignment can often increase risk.

Alignment seems extremely likely to reduce risk relative to the near-certain destruction of unaligned AGI. I'm not saying we're done when we've figured out alignment, but we certainly shouldn't be charging ahead without solving alignment.

link
nullsense 1142 days ago
What's even the rationale to assume that AGI can be 'aligned' or 'controlled'?

It reeks of cognitive dissonance to me. The people running the show now are the ones who grew up getting their first computers aw kids when that tech was just entering people's homes and it was such an amazing and fun thing to play with. Some of them developed these deep fascinations with things like AGI at a young age and that child-like sense of wonder never left them. Now when confronted with the possibility that they can finally make their childhood techno-fantasy a reality, it's too damaging to their psyche to engage meaningfully with the discussion of X-risk. I've watched many interviews of Demis Hassabis and he seems like a wonderful and almost magical human being, but he also seems like a starry-eyed fucking child.

I dunno... maybe I'm just too cynical after all the rabbit holes I've been down.

link