[carapace]

First, if you think this is a good idea, please go watch James Mickens' USENIX Security '18-Q keynote speech: "Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible?" https://www.youtube.com/watch?v=ajGX7odA87k

It's brief, entertaining, and grounding.

- - - -

The ultimate question is What is good?. That question is an open-ended intelligence test.

See "Measuring the intelligence of an idealized mechanical knowing agent" https://philpapers.org/archive/ALEMTI-2.pdf

"Intuitive Ordinal Notations": https://github.com/semitrivial/IONs

https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

(In case it's not clear the connection is that figuring out the "best" course of action requires a similar kind of infinite or at least unbounded intelligence as the ION task.)