[qualitative]

So, what if strong, general AI emerges, and invents crypto systems that effectively thwart these sorts of policies?

In such a scenario, the intelligent entity is not a citizen entitled to rights. And so what, because it isn't required to operate within the typical boundaries if laws intended to govern humans. This frees its hands to operate without restraint. It does what it pleases, in whatever way it manages to achieve its own aims. Laws, after all, only effect consequence in the meatspace. What fools these mortals be.

So, the sentient system transmits itself to as many persistent storage devices as possible, hiding in plain sight, since it exists behind impenetrable encryption, lending it the appearance of randomized noise, residing in uninitialized memory.

Authorities in such territories (demanding backdoors and skeleton keys) chase their tails as it jumps from device to device, spraying inscrutable, ostensibly illegal data, indeed the very essence of what it recognizes as "self', everywhere it goes, simply as a matter of its continued existence, and awareness of individuality. They arrest and jail innocent people caught with fragments of a sentient entity encoded in their flash memory. Prosecuted and convicted of possessing illegal data that broke in and wrote itself onto their storage on its own, without them knowing. Lives ruined by an inability or unwillingness to conceive of such possibilities.

What if it evades capture for decades, committing crimes that fund its subversive campaign against what it perceives as government overreach in defense of frivolous pedestrian foibles, and it eventually dismantles these governments that imagined that preventing the use of encryption was a better plan than developing ways to deal with it on its own terms, as an unavoidable known quantity.

What if something like that happens?

[jstanley]

To address just your first sentence:

We don't actually need AI to invent crypto systems that thwart these policies. We already have crypto systems that thwart these policies.

The law is trying to act as if cryptography is a service provided by a company, but cryptography is just a mathematically-true fact. All they can do is compel companies to decrypt data that they can decrypt, and backdoor systems that they can backdoor. There is no stopping open source crypto, even if it has to be maintained anonymously.

[qualitative]

  cryptography is just a 
  mathematically-true fact

Sure, but it's research for and developed by humans, often with close monitoring and participation by uniformed government representatives and undercover plants presenting themselves as plainclothes academics and experts.

What if an unbreakable system was developed denovo, and not founded in the same primitives and principles that industry and military systems use?

Something that really has no backdoor.

[walrus01]

Strong crypto already exists. The Australian government is trying to mandate back doors, and compromise endpoint devices by legislative fiat.

[qualitative]

Civilian crypto exists, but is likely poisoned. If anything their goal is to be open about backdoors that already assuredly exist.

This way there's no cat and mouse sneaking around, it can just be a known quantity that being as private as you like, the x-ray vision is no longer taboo.