[nlh]

The problem with this line of reasoning is that the exact same thing can be said about our meat-calculator brains.

This gets to the philosophical heart of a debate that I can already foresee will NEVER be settled:

I guarantee you - with 100% certainty - that when we get to a point where AI is "AGI", there will be a continuous and massive political debate (akin to the abortion debate we face today) where one side argues that a given AGI is conscious and must be given rights and cannot be shut off and the other side argues that it's just a calculator and a computer program and computers can be turned off at will, erased, experimented on, and whatever.

We have the same debate today all the time! There are those who believe every human life is sacrosanct (from age 0-100+) and others who believe human life is disposable (from age 0-100+!). There's no reason to believe this debate won't extend to AGI.