[edanm]

You seem to assume that the belief that humanity "loses by default" is wrong. I'm curious to hear why, since you seem to be know about this topic. (And because I disagree with your view, so hearing might educate me).

"Some folks take it for granted (why??? Take things for granted about such an uncertain subject???) that we're just doomed unless you Give Miri Money(tm)."

Are there really people who would put it that way? I know plenty of people (including myself) who think it's a good idea to donate to MIRI, but I certainly wouldn't put it in the terms you did.

[lahwran]

So as I was writing a reply, I found that my opinion doesn't actually differ in "do we need to do a bunch of stuff"; it's more that I don't think miri is taking a useful approach. After talking to a friend who is also in ml about this really, I think the key point I should make is that it's more a matter of engineering control tools that ensure we can spy on its thoughts. If we can build an agi, we can also build safeguards that it doesn't realize are there until too late.

Anyway, here is the original comment I was going to write. You can read it and extract your thoughts; I'm fairly confident I got the theory right, but I've reduced my confidence in the point/counterpoint. Regardless, I definitely think miri's position is unreasonably extreme, and this is a pretty ok explanation of why.

https://gist.github.com/lahwran/99d84c3f8461ece9153f

[edanm]

Thanks for the detailed reply!

So firstly, I'm glad we agree on the core idea - some effort/funding should be put into this problem. You don't think MIRI is the best vehicle for such funding - that's understandable, although I'm semi-biased in their favor as they probably brought the most attention to this issue, and have been working on it for a long time. I assume that they have good reasons for the approaches they're taking right now. But again, not believing in MIRI or in a foom event is totally compatible with still believing we should be doing something about this problem, so you and I are pretty closely in sync.

Having said that, I read through your notes, and I do have a major object. (You seem to understand more of the technical details than me, so take with a grain of salt).

I think your argument boils down to one main thing - that humans are pretty much "efficient" with respect to data usage already, because of evolution. If we take away that belief, then your reasons for thinking we can't build an AI that's superintelligent goes away - we would be able to squeeze more efficiency out of it.

So here's the problem - I don't understand why you believe that humans are data efficient. I understand the idea behind it, but it seemed to me like the only explanation for data efficiency is that evolution would've taken care of it. You even mention that the default argument agains this point is that eveolution might get stuck in local optimums, but it's very hard to do better.

But there are 2 counterarguments to this:

1. I don't think it's so much difficult to do better than evolution, since we do it regularly. We defeat diseases. We build machines that are faster/stronger/etc than anything evolution has "come up with". Etc.

2. I don't think evolution was "optimizing" for intelligence anyway. Our intelligence, however limited, is enough to become the most powerful species on the planet. Biology faces very hard trade-offs in squeezing more intelligence into us, but this isn't true of digital equipment, which doesn't have to be based on hardware that, e.g. has to be born, therefore has to squeeze through the birth canal, etc. These are limits that evolution was "working under", but that we don't have.

Note: All anthropomorphizing of evolution done figuratively.

[lahwran]

No, there are more dimensions than just data efficiency; what I would say is that humans are just about as data efficient as you could possibly hope for _at that wattage_. It's easy to do better than evolution at something it wasn't trying to do, I agree - and this is the point where, as I thought about it, I realized I do actually think there's some concern. But I don't think we'll be able to do it on one gpu, because:

- human evolution has spent quite a while in an adverserial environment - the smarter you are, the more you win - a recent finding of the neural network research is that local minima are kind of not a problem in very highly dimensional spaces, as long as you have a problem that is smooth and has optima. If it has any optima, then in very high dimensions, there's probably always something you can change that will keep you moving towards the optimum. while evolution may have gotten stuck in a general class of architectures - neural ones - it seems very much like there are many dimensions along which it can change the brain, and that changing the genes for it slightly will change its performance slightly (fsvo slight). - evolution, in species that have learning systems in the first place, optimizes for intelligence per watt. energy is very costly in the wild, and so finding algorithms that work well with low power is very important. It so happens that algorithms that minimize power usage are theoretically tied to algorithms that compress well, but the key thing is that evolution has had a crapload of optimization time for tuning the brains of mammals in general, and then humans got in this runaway optimization process - which seems to have made us smarter primarily by making our brains use more power for the relevant parts.

I definitely think you could do better, I just don't think you're going to do it with a paradigm that looks vaguely like the brain, because if it looks vaguely like the brain, evolution probably passed it up on the way to the general architecture that mammals use, and the specific one humans use. Possible exception for gradient descent and weight sharing, because those would be difficult to implement in the brain, but that doesn't give you results hundreds of times better, and it's not even clear the brain doesn't do that - hinton has made the argument that it could.

the key thing here: if we make an agi with neural networks (which at this point is almost a for-sure thing), then going beyond human level on one gpu will be a very difficult research task, and take it a lot of learning to figure out how to do. Which means we'll get a chance to control it using less formal mechanisms than miri demands out of their work.

(I don't think miri's stuff will be done in time to be useful to anyone.)

[edanm]

Thanks for the reply!

I'm also curious in your answer to Eliezer about why you assume 1 GPU.

But a few other questions:

1. You say: "human evolution has spent quite a while in an adverserial environment - the smarter you are, the more you win". Again, maybe I'm missing something in modern evolutionary thinking, but why that assumption? I always thought the consensus was that we were as minimally smart as required.

2. You seem to be under the assumption that today's most popular algo's (namely neural networks) are definitely for sure the thing that's going to become an AGI, but why that assumption? The more broad idea of some algorithm/method bringing us an AGI is more probable than specifically neural networks.

You also write: "I definitely think you could do better, I just don't think you're going to do it with a paradigm that looks vaguely like the brain". Again, why the assumption that whatever will be built will have to even resemble the human brain?

[Eliezer]

Why on Earth would you assume that MIRI assumes one GPU rather than 10,000 GPUs?