[lahwran]

Ugh, there's a lot of argument about that in the cfar alumni community. Some folks take it for granted (why??? Take things for granted about such an uncertain subject???) that we're just doomed unless you Give Miri Money(tm). Those folks tend to be pretty good about actually carrying out what is reasonable behaviour in most other ways if only they were right about that one thing - if it really were such a big deal, you'd want to not ignore it.

Meanwhile, another part of the alumni community actually understands the theory behind ai and machine learning, and those folks end up in arguments frequently with the first category about the topic.

The reason you hear about it is the first category is a pretty panicked and hopeless group - for the people who actually believe yudkowsky's "recursive algorithmic improvement" to be able to give large improvements, they generally think that humanity "loses by default" if they do nothing. So they tend to be very, very into recruiting. Thankfully they're not so nuts about it that they'll never change their minds, the problem is it takes a lot of explaining to get the theoretical basis for why the recursive self improvement thing isn't actually as scary as they think it is. No, it's not going to take an hour as soon as an ai is built, learning is hard, and humans are freakishly good at it.

Computers will beat us at data efficiency eventually but it's gonna take a while, and current machine learning is better at being data inefficient but getting good results from the large amounts of data. And the best you can do isn't good enough to make miri's monster - unbiased, maximally data-efficient Bayesian inference doesn't actually fit in the universe in either a time or memory sense if you try to build a full ai out of just that one thing. And approximating it is, you guessed it, less data efficient.

[someguydave]

Okay so: 1.) preaches about new destroyer god 2.) saviors who possess the secret knowledge of salvation 3.) give them money 4.) think correctly, not incorrectly

Yep, it's a cult.

[lahwran]

Eh... Sort of? The two organisations together would form a cult, but the community split I mention makes it a bit confusing. Overall, I agree that miri's level-of-cult is too damn high.

[someguydave]

Interestingly, it's an ancient form of cult called Gnosticism. Gnostics teach that the material world is evil ("gives rise to the AI"), and only through hidden knowledge ("correct thinking") can one find the true path to spiritual salvation ("become an immortal human")

https://en.wikipedia.org/wiki/Gnosticism

[philh]

> Ugh, there's a lot of argument about that in the cfar alumni community.

I take it you're involved with parts of the community outside the CFAR mailing list? I haven't seen it on there. There's certainly argument about it within the broader LW community, but the CFAR alumni community is distinct (but overlapping).

[lahwran]

primarily in person, yes. presumably primarily from people who frequent less wrong. I don't, so I don't really know.

[edanm]

You seem to assume that the belief that humanity "loses by default" is wrong. I'm curious to hear why, since you seem to be know about this topic. (And because I disagree with your view, so hearing might educate me).

"Some folks take it for granted (why??? Take things for granted about such an uncertain subject???) that we're just doomed unless you Give Miri Money(tm)."

Are there really people who would put it that way? I know plenty of people (including myself) who think it's a good idea to donate to MIRI, but I certainly wouldn't put it in the terms you did.

[lahwran]

So as I was writing a reply, I found that my opinion doesn't actually differ in "do we need to do a bunch of stuff"; it's more that I don't think miri is taking a useful approach. After talking to a friend who is also in ml about this really, I think the key point I should make is that it's more a matter of engineering control tools that ensure we can spy on its thoughts. If we can build an agi, we can also build safeguards that it doesn't realize are there until too late.

Anyway, here is the original comment I was going to write. You can read it and extract your thoughts; I'm fairly confident I got the theory right, but I've reduced my confidence in the point/counterpoint. Regardless, I definitely think miri's position is unreasonably extreme, and this is a pretty ok explanation of why.

https://gist.github.com/lahwran/99d84c3f8461ece9153f

[edanm]

Thanks for the detailed reply!

So firstly, I'm glad we agree on the core idea - some effort/funding should be put into this problem. You don't think MIRI is the best vehicle for such funding - that's understandable, although I'm semi-biased in their favor as they probably brought the most attention to this issue, and have been working on it for a long time. I assume that they have good reasons for the approaches they're taking right now. But again, not believing in MIRI or in a foom event is totally compatible with still believing we should be doing something about this problem, so you and I are pretty closely in sync.

Having said that, I read through your notes, and I do have a major object. (You seem to understand more of the technical details than me, so take with a grain of salt).

I think your argument boils down to one main thing - that humans are pretty much "efficient" with respect to data usage already, because of evolution. If we take away that belief, then your reasons for thinking we can't build an AI that's superintelligent goes away - we would be able to squeeze more efficiency out of it.

So here's the problem - I don't understand why you believe that humans are data efficient. I understand the idea behind it, but it seemed to me like the only explanation for data efficiency is that evolution would've taken care of it. You even mention that the default argument agains this point is that eveolution might get stuck in local optimums, but it's very hard to do better.

But there are 2 counterarguments to this:

1. I don't think it's so much difficult to do better than evolution, since we do it regularly. We defeat diseases. We build machines that are faster/stronger/etc than anything evolution has "come up with". Etc.

2. I don't think evolution was "optimizing" for intelligence anyway. Our intelligence, however limited, is enough to become the most powerful species on the planet. Biology faces very hard trade-offs in squeezing more intelligence into us, but this isn't true of digital equipment, which doesn't have to be based on hardware that, e.g. has to be born, therefore has to squeeze through the birth canal, etc. These are limits that evolution was "working under", but that we don't have.

Note: All anthropomorphizing of evolution done figuratively.

[lahwran]

No, there are more dimensions than just data efficiency; what I would say is that humans are just about as data efficient as you could possibly hope for _at that wattage_. It's easy to do better than evolution at something it wasn't trying to do, I agree - and this is the point where, as I thought about it, I realized I do actually think there's some concern. But I don't think we'll be able to do it on one gpu, because:

- human evolution has spent quite a while in an adverserial environment - the smarter you are, the more you win - a recent finding of the neural network research is that local minima are kind of not a problem in very highly dimensional spaces, as long as you have a problem that is smooth and has optima. If it has any optima, then in very high dimensions, there's probably always something you can change that will keep you moving towards the optimum. while evolution may have gotten stuck in a general class of architectures - neural ones - it seems very much like there are many dimensions along which it can change the brain, and that changing the genes for it slightly will change its performance slightly (fsvo slight). - evolution, in species that have learning systems in the first place, optimizes for intelligence per watt. energy is very costly in the wild, and so finding algorithms that work well with low power is very important. It so happens that algorithms that minimize power usage are theoretically tied to algorithms that compress well, but the key thing is that evolution has had a crapload of optimization time for tuning the brains of mammals in general, and then humans got in this runaway optimization process - which seems to have made us smarter primarily by making our brains use more power for the relevant parts.

I definitely think you could do better, I just don't think you're going to do it with a paradigm that looks vaguely like the brain, because if it looks vaguely like the brain, evolution probably passed it up on the way to the general architecture that mammals use, and the specific one humans use. Possible exception for gradient descent and weight sharing, because those would be difficult to implement in the brain, but that doesn't give you results hundreds of times better, and it's not even clear the brain doesn't do that - hinton has made the argument that it could.

the key thing here: if we make an agi with neural networks (which at this point is almost a for-sure thing), then going beyond human level on one gpu will be a very difficult research task, and take it a lot of learning to figure out how to do. Which means we'll get a chance to control it using less formal mechanisms than miri demands out of their work.

(I don't think miri's stuff will be done in time to be useful to anyone.)

[edanm]

Thanks for the reply!

I'm also curious in your answer to Eliezer about why you assume 1 GPU.

But a few other questions:

1. You say: "human evolution has spent quite a while in an adverserial environment - the smarter you are, the more you win". Again, maybe I'm missing something in modern evolutionary thinking, but why that assumption? I always thought the consensus was that we were as minimally smart as required.

2. You seem to be under the assumption that today's most popular algo's (namely neural networks) are definitely for sure the thing that's going to become an AGI, but why that assumption? The more broad idea of some algorithm/method bringing us an AGI is more probable than specifically neural networks.

You also write: "I definitely think you could do better, I just don't think you're going to do it with a paradigm that looks vaguely like the brain". Again, why the assumption that whatever will be built will have to even resemble the human brain?

[Eliezer]

Why on Earth would you assume that MIRI assumes one GPU rather than 10,000 GPUs?