[jacques_chester]

I think his point is that they may be irresolvable.

To think about the world, you must first have a model of the world. Then you reason about the model, finally you take action.

Somewhere in there you have a motivation for following this observe-decide-act loop. Motivation provides a reference point towards which you want the observed system to evolve.

But there's a problem. The easiest way to satisfy the motivation component is to lie to it. Tell it things are just hunky-dory.

Any singularity-style intelligence will necessarily need to be built with some kind of anchoring motivations to try and stop it from getting out of control. But what's to stop it simply lying to itself and ignoring the outside world?

Trying to add a meta-motivation be as realistic as possible won't work, such a system will seize up with analysis paralysis.

One of the things that Hayekian economists argue is that knowledge about reality cannot be centralised; it is unevenly, lumpily distributed across the whole of humankind. No one actor does, and no one actor could, perceive the entire system. But it works, because no one actor has to.

[marcusrobbins]

I do not believe that it is irresolvable, I'm looking at biology for my counter example. In biology there exists a rigid framework inside which a "general" search of language/model is taking place, after a few billion years it seems to be doing ok. It is my hunch that there is a big something out there which will pull a lot of these issues together:

How do you build efficient markets? How do you build an effective AI? How do you design effective distributed systems? How do you build effective languages/models with which to compress the world?

Under what circumstances do such 'meta' searches fail and succeed? It's all beyond me but this is what my nose is saying...