[choudharism]

I like John and his body of work, but this and the general slew of posts with very little actual knowledge or expertise making grandiose / prophetic assertions are the soothsayings of our age - people convinced that they've cracked "it", but without any factual reasoning as to why.

They raise philosophical assertions disguised as logical / scientific assertions.

> “Better.”

> That is a concept that cannot be made via cold calculation. A universe boiled down to particles and numbers cannot care whether atoms arranged as a suffering plague victim are “better” than those arranged as a healthy child, or a plant, or a cloud of gas.

It can, John. The engineers teach machines what's good and bad. That's the first step in creating silicon intelligence.

> It is impossible to understand pain without having felt it yourself; it does not convey as an abstract value. I think the machine needs to tangibly interact with human suffering in the same way the Roomba’s wheels need to physically touch the floor.

Pain is nature's deterrence mechanism, bred into us through evolution. Deterrence mechanisms can take many shapes.

> This leads us to the ultimate cosmic joke: This task requires answering for this machine questions that we ourselves have never been able to answer. Let’s say I’m right and this system won’t exist until we can teach it emotion, and the capacity to suffer. How long until it asks the simple question, “What is our goal, long-term?” Like, as a civilization, what are we trying to do? What are hoping the superintelligence will assist us in achieving?

I hate being disingenuous and make a flippant argument here, but reading this from start to finish makes me ask a bad faith question - your assertion is that we won't have self driving cars because we can't tell the robots what to do with the whole civilisation right away? There's a whole spectrum of amazing (or terrible) possibilities between the extremes, my guy.

[jmye]

> It can, John. The engineers teach machines what's good and bad. That's the first step in creating silicon intelligence.

I don’t think a simple “actually, no” is a useful counter argument to his point, especially when he used multiple examples and you just shook your head.

> I hate being disingenuous and make a flippant argument here, but reading this from start to finish makes me ask a bad faith question - your assertion is that we won't have self driving cars because we can't tell the robots what to do with the whole civilisation right away?

That’s not at all his specific argument about self-driving cars (there were entire sections about the decision making issues he saw), but is the crux of his argument about AI as a whole.

> There's a whole spectrum of amazing (or terrible) possibilities between the extremes, my guy.

And there’s a whole article discussing some of those, if you read the parts in the middle. I don’t think the dude’s necessarily right, but I’m also pretty sure you missed a lot of the text in an apparent race to summarily dismiss it.

[choudharism]

> I don’t think a simple “actually, no” is a useful counter argument to his point, especially when he used multiple examples and you just shook your head.

I shook my head because they were similar and equally bad.

> That’s not at all his specific argument about self-driving cars (there were entire sections about the decision making issues he saw), but is the crux of his argument about AI as a whole.

I was talking about the crux of his argument, which starts by saying that we won't get self-driving cars because <start of bad arguments>.

> And there’s a whole article discussing some of those, if you read the parts in the middle. I don’t think the dude’s necessarily right, but I’m also pretty sure you missed a lot of the text in an apparent race to summarily dismiss it.

In my complete reading, the article was geared towards discussing the impossibilities (rather than possibilities) from the POV of someone who hasn't taken any time to understand the cutting edge of research that he is critiquing. That was, in fact, my original point - critique is great, but don't mask personal intuition (potentially uneducated) as science.

[ChatGTP]

But what is really good and bad? We’re imprinting this into the machines but we don’t always have a grasp on this ourselves.

Was the use nuclear weapons at the end of WW2 good or bad ? For example …

[taylodl]

You don't need to define good and bad, instead you focus on better and worse and the metrics used to measure them. Now the goal is to maximize "better" and minimize "worse." You may recognize this as the essentials utilitarianism. The advantage utilitarianism has is it can be applied algorithmically without passion or emotion - in other words, by AI.

Utilitarianism leads to controversial outcomes, but every decision is defensible.

[ben_w]

For each thing T, that T is defensible under at least one ethical framework.

Teaching an optimiser AI any of those frameworks, or even any preference ordering or combination function within Utiliarianism (because value({T, T}) doesn't have to equal 2 * value({T})), will lead to it optimising what you said, without necessarily limiting that to situations anything close to the training distribution.

To put it another way: if you run an AB test on a social media site and it observes that people are more likely to engage with content that makes them angry, then tell it to boost engagement "because socialising is always good, obviously" then it will get your users as angry as possible and suddenly you get Buddhists going off and committing surprise genocide before anyone tells you something has gone wrong.

[taylodl]

I would argue this has been known for decades and is in fact the origin of one of the earliest memes in computer science: To err is human, but to really mess things up requires a computer!

[defrost]

Opportunistic, and morally equivalant to the destruction of the other 72 Japanese cities, and the bombing of European cities by both the Allies and the Axis.

Good v. Bad seems a tad simplistic.

[ChatGTP]

I know, but if we're going to be building T1000's, and I'm sure that's on the cards if it was possible, we better have a good answer to this?

[choudharism]

Does any intelligent life exist on earth which doesn't hold a formative black-and-white answer to that question?

[ChatGTP]

I don't get what you mean?

Are you saying that everyone feels strong one way or another?

[choudharism]

I’m saying that it is possible to learn how to optimise for “better” or “worse” decisions without being trained on an opinionated dataset of every single event that has happened. An “intelligence” could exist without necessarily answering your nukes question.

You’re bringing alignment to a capability discussion.

[ChatGTP]

How do you suppose we do this? It makes decisions based on numbers alone ?

Personally, I don't think there is a right decision, it was just a decision which had an outcome, for some people it was ab absolutely fucking devastating decision.

This is where I think we have blind spots when developing these systems, I don't think there is a "right" or best answer here. Just some answer.