[blamestross]

Stack depth is finite. If consciousness is cognitive recursion, we only can get so far down before the results are garbage. My working theory is that "max cognitive stack depth" is a measure of consciousness.

The scarier concern is that consciousness is a useful intelligent behavior bootstrapping tool, but once we encode everything it has helped generate into cultural DNA or literal DNA it becomes an evolutionary redundant appendix-like organ that will atrophy in the coming generations (See Blindsight by peter watts)