[ComplexSystems]

"When you start reading about AI safety, it’s striking how there are two separate communities—the one mostly worried about machine learning perpetuating racial and gender biases, and the one mostly worried about superhuman AI turning the planet into goo" - great quote.

[rexreed]

What worries me more are bad people doing bad things with AI, malicious use of AI, or just AI negligence. Deep fakes. Algorithmic decision making taking the human out of the loop (such as bad content moderation and automated account shutdown). Lack of disclosure. Lack of consent. Autonomous systems with poor failure modes.

It's not that I'm not concerned with bias and AI systems going haywire, but the above scenarios seem to get less attention from researchers, probably because their employers might be perpetuating many of these above issues of AI safety.

[BeFlatXIII]

IMO, deepfakes are a public good because they reduce the sting of blackmail. Does someone have an incriminating video of you? Let them release it and then point out that the shadows look all wrong.

[joe_the_user]

I'm not sure why bad content moderation would be a problem but bias wouldn't be. Both involve people treated unfairly by a system and both happen because the system uses "markers" for undesirable things, "markers" that don't by themselves prove you're doing the undesirable thing. There was post here a month or two ago about a guy suddenly putting a bunch of high end electronics for sale on some big site and being perma-banned just for a pattern that's common for fraudsters. The software that decides that some people don't deserve bail is operating by a similar method - markers which don't prove anything about the person by themselves, that often involve things that signify race, and are taken as sufficient to deny a person bail.

[gjvnq]

There's also the crowed that worries about self-driving cars doing stupid shit like misreading a 20 km/h sign as an 120 km/h.

[joe_the_user]

Maybe it's the same "crowd" that read the report of the Tesla that mistook a turning semi for empty space and killed the driver.

[genem9]

Pretty sure the self driving vehicles get the speed data from a dataset and not from signs.

[tgb]

There are already non-self-driving cars that get speed limits from signs. I’ve seen that feature in a Honda for example. I imagine you’d have multiple sources like a max speed for that type of road as a fallback. And you need to read speed limit signage due to temporary limits. There’s also variable speed limit roads near me now and so you have to read those electronic signs unless the database is updated very often (though no humans seem to obey those limits).

[jandrewrogers]

No, many ordinary cars contain image classifiers that read speed limit signs dynamically. This is pretty standard. And they sometimes get it wrong e.g. my car reads 80 MPH as 60 MPH about a third of the time, much to my dismay.

The dynamic classification is required because the world isn't static. An increasing number of locales have digital speed limit signs that vary the speed limit dynamically, some times independently per lane. Automation requires cars to respond to the world as it is, not how the world was when it recorded a month ago.

[pas]

then it's not really (self)driving.

[afthonos]

It’s (not) self-driving in the same way planes do (not) fly. Just because something doesn’t do things the way we (or our or a bird) does doesn’t mean it doesn’t achieve its goal.

[pas]

if a street changes into a one-way one day (signaled by a sign) relying on a map will lead to a big unhappy problem.

sure, if you consider everything selfdriving that works on a NASCAR track, then yes, a map is sufficient, but if we are talking about driving on public roads then recognizing and "obeying" signs visually seems like a hard dependency.

[amelius]

Then how about an AI misclassifying a firetruck for a road?

[homarp]

who maintains that worldwide dataset?

[layer8]

Maybe an AI processing Google Street View recordings. :)

[jstummbillig]

Why would they?

[cavisne]

It's a tricky field, and not a coincidence that at least 3 (maybe more) high profile disgruntled employees from Google have been from this area.

I think of it as kind of like security, in that you are sometimes seen as against the push of the overall project/area. However unlike security there are 0 software tools or principles that anyone agrees on.

[blueboo]

These are the themes that get headlines in mainstream press; they’re not reflective of the distribution of actual research

[ductsurprise]

Agreed. Maybe... Short sighted vs. far sighted vs. ignorance is bliss?

[ibrarmalik]

Or practical vs. philosophical.

[Mizza]

Deciding which camp is practical and which is religious is left as an exercise to the reader.

[croes]

Reality vs fiction

[astrange]

It's a secular/religious divide. (As it says in the post.)

Though it's possible the people who think a theoretical future AI will turn the planet into paperclips have merely forgotten that perpetual motion machines aren't possible.

[marvin]

There is nothing religious with thinking about whether failure modes of advanced artificial intelligence can permanently destroy large parts of the reality that humans care about. Just like there was nothing religious in thinking about whether the first atomic bombs could start a chain reaction that would destroy all life on Earth.

Part of such precautionary planning involves asking whether such an accident could happen easily or not. There certainly isn't consensus at the moment, but the philosophy very clearly favors a cautious approach.

Most people are used to thinking about established science that follows expected rules, or incremental advances that have no serious practical consequences. But this isn't that. There is good reason to think that we're approaching a step-change in capabilities to shape the world, and even a strong suspicion of this warrants taking serious defensive measures. Crucially for this particular instance of the discussion, OP is favoring that.

There will necessarily be a broad spectrum of opinions regarding how to handle this, both in the central judgement and how palatably the opinion itself is presented. Using a dismissive moniker like 'religious' for a whole segment of it doesn't give justice to the arguments.

Present a counterargument if you feel strongly about it, and see whether that will stand on its own merit.

[astrange]

The post, which calls Eliezer a "prophet" who says people should drop everything in their lives to work on AI safety, agrees with me.

> Present a counterargument if you feel strongly about it, and see whether that will stand on its own merit.

This is a bad way to talk to rationalists because it's what they think solves everything and is the reason they're convinced an AI is going to enslave them. As long as you're actually right, saying "no that's dumb and not worth worrying about" is superior to logical arguments about things you can't have logical arguments about (because there are unenumerable "unknown unknowns" in the future). This is called "metarationality".

e.g. Someone could decide to kill you because they don't like one of your posts (1). Is there any finite amount of work you could do to stop this? No (2). Should you worry about this? No (3).

You can't logically prove the 2->3 step, nor can you calculate the probability of it being a problem, but it still doesn't seem to be a problem.

[eru]

Could you please explain your reasoning for the comparison to perpetual motion machines?

(Keep in mind that biological machines, ie life, have managed to turn the surface of the planet into 'green goo'.)

[astrange]

Self-reproducing machines are capable of covering the surface of the planet, yes. There's one right now (covid). But there's lots of energy and oxygen up here and they rarely displace other such machines (species) or even displace much of any earth and water. And because they're self-contained and self-reproducing, all of their instructions can be lost over time to entropy including the ones we're afraid of.

None of em replace the entire planet though. That's a lot of rock to digest without any more energy to help you do it.

And a paperclip factory isn't self-reproducing (that would be a paperclip factory factory). It's just a regular machine that can break down. The people afraid of that one are imagining a perfect non-breaking-down non-energy-requiring machine because they've accidentally joined a religion.

[eru]

I'm not talking about covid; covid is not covering the planet. I'm talking about life in general.

All that oxygen comes from all the plants.

Yes, life has so far only covered the top of the planet. You are right that a paper clip maximizer would need quite a bit of time to go deeper than life has gone (if it would get there at all).

> And a paperclip factory isn't self-reproducing [...]

Why wouldn't it? If your hypothetical superhuman AGI determined that becoming self-reproducing would be the right thing to do, presumably it would do that.

No perfection required for that. Biological machines aren't perfect either. Just good enough.

You are right that thermodynamics puts a limit on how fast anything can transform the planet into paperclips or grey goo.

Though the limit is probably mostly about waste heat, not necessarily about available energy:

There's enough hydrogen around that an AGI that figured out nuclear fusion would have all the energy it needs. But on a planet wide basis, there's no way to dissipate waste heat faster than via radiation into space.

(Assuming currently known physics, but allowing for advances in technology and engineering.)

---

Of course, when we worry about paperclip maximisers, it's bad enough when they turn the whole biosphere into paperclips. Noticing that they'll have a hard time turning the rest of the earth into paperclips would be scant consolation for humanity.

(But the thermodynamic limits on waste heat still apply even when just turning the biosphere into paperclips.)

[austinjp]

> a paperclip factory isn't self-reproducing (that would be a paperclip factory factory)

This seems an odd refutation for several reasons.

First, the paperclip AI might determine that self-reproducing factories would be an optimisation, and aim to achieve that by any means necessary.

Second, a single paperclip factory that doesn't reproduce might still develop the means of bringing raw materials to it.

Either way, an all-consuming paperclip AI emerges.

In general, I find the equating of the paperclip problem with a religious cult to be naive.

[falcor84]

Agreed, and being a bit of a religiophobe, the thought of living through some sort of butlerian jihad scares me enough, regardless of whether the machines can actually kill us all.

[DalasNoin]

I think you are conflating different concepts. It is clearly imaginable that a very intelligent agent could end humanity if its objective would require so. How exactly is "perpetual motion machines can't exist" related to this? How is it going to prevent an agent from engineering 1000 pandemic viruses at the same time?

[astrange]

> It is clearly imaginable that a very intelligent agent could end humanity if its objective would require so.

This is quite possible. Indeed, I don't believe this is exclusive to superintelligence or requires it at all. Compare to the closest thing we have to "inventing AGI" - having babies. People do that all the time and there isn't a mathematical guarantee that baby won't end humanity, but we don't do much to stop it, and that's not considered a problem. Mainly, why would it want to?

https://twitter.com/thejadedguy/status/844352570470645760?la...

I don't think superintelligence even gives them much advantage if they wanted to. Being able to imagine a virus real good doesn't actually have much to do with the ability to create one, since plans tend to fail for surprising reasons in the real world once you start trying to follow them. Unless you define superintelligence as "it's right about everything all the time", but that seems like a magical power, not something we can invent.

> How exactly is "perpetual motion machines can't exist" related to this?

It wouldn't be able to do the particular kind of ending humanity where you turn them all into paperclips, though it could do other things. There's plenty of ways to do it that reduce entropy rather than increase it - nuclear winter is one.

[sva_]

> Self-reproducing machines are capable of covering the surface of the planet, yes. There's one right now (covid). But there's lots of energy and oxygen up here and they rarely displace other such machines (species) or even displace much of any earth and water. And because they're self-contained and self-reproducing, [...]

Your analogy is weak and also false: viruses can't self-reproduce, but need to bind to a host's protein synthesis pathways.

[astrange]

That makes my point stronger if you're claiming self-reproducing machines don't exist. Of course I thought of using a bacteria or an algae bloom there, and even though I didn't, pretending I did is a better use of your time than commenting surely? The future robot torture AI isn't going to like that.

[c1ccccc1]

This is your friendly physics reminder that perpetual motion machines have nothing to do with this. It's hard to turn the whole planet into paperclips because paperclips are mostly made of iron, while the planet contains many other elements. Of course, with a high enough level of technology, it might be possible to fuse together the non-iron elements, so that you would end up with just a bunch of iron nuclei. This would even be energetically favourable, since iron is so stable. Then you just have to solve the issue that the paperclips in the center of the planet would be under huge pressure and would be crushed.

[astrange]

It's less than perpetual, but a planet is a lot of raw material to work through for a machine without it breaking down, if the machine's also eaten all the people who can repair it.

[c1ccccc1]

Yes, self replicating machines would probably be required. But self replicators are definitely physically possible.