[jorleif]

I feel this solves a very small part of the problem. This is only about the morality of the individual scientist, while the problem is at the level of the collective process of science. I mean, it is of course good if the individual scientist who discovers X way to kill off humanity keeps it secret, but eventually someone else will also discover X, and then someone else, and so on. Do we require of scientists that they all keep these things secret?

Of course, we may not want to have terrorists building atomic bombs, so some secrecy is prudent, but secrecy does not generally hinder: 1) Some bad people from accessing said information, now having monopoly, which may make matters even worse 2) Discovering the same thing independently - Yes, we may keep it secret, but that does not change the laws of physics, the exploitation potential is still there.

[harperlee]

It is a complicated matter... as long a science advances and is accesible to all, we all gain more power. That enables everybody to be dangerous. I can go anywhere in the world, access a huge amount of information, purchase and have sent home lots of things very cheaply, etc.

We crave power, both in the sense of ability and energy, and it seems to grow monotonically (Freeman Dyson proposed searching for efficient star-energy-capturing devices as a way to find ET life: http://en.wikipedia.org/wiki/Dyson_sphere).

But I agree, it seems that it is not very easy to prevent new science to be discovered. We are a curious species, and even if we go back to the dark ages, even if we move up or down the knowledge scale randomly, this brownian movement will end up jumping too high, and giving too much power to someone.

So at least we should incorporate an element of respect to such power in our education, our ethics, and try to prevent it from being lost. In my opinion, this kind of thing is most efficiently managed through religion, but current religions do not advance very quickly nor are amenable to "improvements", whereas the idea of scientific religion, to us, now, seems abhorrent and oxymoronic. I believe this might change in the future, though. If science "just works" (and it is indistinguishable from magic), I could picture someone stripping out the "scientific inquiry and skepticism" part, and building a religion on top of the huge building of true scientific facts that we faithfully take for granted.

So in my opinion we have to focus on education. Sorry for the rambling :)