[PrunJuice]

# SPOILERS

Great writing. Then ending was a real let down.

How could an "AI" as they describe simultaneously be so naive and ALSO protect itself in any meaningful way? Especially in its early stages. It wouldn't even know to hide. And why would Big Corp give up trying to fix this sort of problem.

Overall not a credulous conclusion. Hand waving in the final paragraphs after the author crafted an accurate and believable narrative left me disappointed. (grammar)

[teekert]

Biological virusses are also very naive yet many have been incorporated into our genome and many continue to bug us even today in this scientific age.

[TheOtherHobbes]

Biology has a four billion year head start and the benefit of adaptive feedback.

Where would this hypothetical machine code micro-AI find adaptive feedback selection pressure?

[darkmighty]

The evolutionary pressure is just people or programs detecting certain strains (it would generate lots of strains randomly).

I think the only real issue to plausibility here is I'm not sure brute force is enough to make enough plausible behavioral branches, at least with current computing power/internet bandwidth. A reasonably efficient self-modification mechanism (in terms of viable strain per transmission) is probably extremely large, I'd say at least 1GB. Not unlike deep learning systems, this would consist of a large functional composition of heuristics, codifying how to write code that can embedded itself in other programs and write modifications to itself that are likely to work.

Note that we haven't yet gotten a good neural-generated code modifications, even using large networks, GPU training and large computing time. Best examples I could find:

http://karpathy.github.io/2015/05/21/rnn-effectiveness/#linu...

https://arxiv.org/abs/1611.01989

So we're not yet at a point this could be plausible (as it couldn't hide itself in small programs), but eventually it will be -- once there is enough headroom on most GPUs and certain types of software are large enough it could hide it's network inside, and generally enough internet bandwidth to spread it's >GB-scale code. I'd imagine something like a game, which usually has networking -- it would be using GPUs partially to generate and spread new strains of it trying to infect other games and such.

Note there are biological viruses with tiny genomes however -- the smallest are on the order of ~1kbyte. But as you cite they had billions of years, producing maybe quadrillions of viruses every year, giving this tiny efficient and specialized weapon. Interestingly, they rely on other cells machinery to even replicate their genome -- analogous to using the compiler here.

http://www.lehigh.edu/~jas0/viralgenomes.html

If everyone could send >10^18 different small self-replicating viruses over your network, it seems likely some would exploit bugs in certain kinds of hardware/software, evolving through this selective pressure.

[PrunJuice]

Consider that even with billions of years worth of evolution natural viruses have not developed any sort of Hivemind (tm).

Also, now that science has illuminated the human genome we are quickly (on an evolutionary time scale) advancing toward gene therapy treatments that could combat viruses.

Now consider executables. We have the ability quickly inspect AND edit executables, source code..., etc. Not to mention a much better conceptual framework for interpreting assembly instructions (compared to codons). After all they were created by humans for machines that humans built.

Then consider the resources viruses have had at their disposal to evolve. Every single cell [1] of every single living creature that has ever been infected over billions of years [2]. Assuming a conservative average number of 1e31 cells over the course of life's history that means (1e31 * 3.5e9 years) = (3.5e40 cell * years) of computations time. Then consider RNA transcription rates ~6.3e12 nt/year/cell [3]. So all together something like (3.5e40 cellyears) (6.3e12 nt/year/cell) = 2.2e53 nt. Approximation of course but probably with a few orders of magnitude.

Now compare to the number of instructions since the epoch. (1.5e9 s) * (2e18 instructions/s [4]) = ~3e27 instructions since the epoch. Again approximation.

(2.2e53 nt) / (3e27 instructions) = 6.6e26 nt/instruction.

That means we would be seeing the equivalent of viruses that evolved (3.5e9 years since first life) / (6.6e26 nt/instructions) = ~17 nanoseconds after life emerged.

Wow that was quite a tangent.

[1] https://forum.cosmoquest.org/showthread.php?141294-How-many-... ~5e31 cells currently on earth

[2] http://www.pbs.org/wgbh/evolution/library/faq/cat06.html life has existed for 3.5 billion years

[3] http://book.bionumbers.org/what-is-faster-transcription-or-t... In E. coli: (~5000 RNA polymerase molecules) * (~40 nt/s) * (60 s/min) * (60 min/hr) * (24 hr/day) * (364 day/year) = 6.3e12 nt/year/cell

[4] http://www.worldometers.info/computers/ 1 billion computers in use in 2008. with say 1 instruction per cycle and one core per computer at 2GHz that's (1e9 computers) * (2e9 inst/computer/s) = 2e18 inst/s