[akkartik]

"The main thing we know about the QHL’s software is that it is large and complex enough to defeat a years-long “software archeology” effort directed at determining whether the nodes can be trusted (in the end, they cannot)."

"cheap, effective sensor nodes lead to a government with the capacity for ubiquitous surveillance, which leads to a police state, which leads to societal collapse or worse."

It feels helpful to reframe "can it be trusted?" as "who knows more about its emergent runtime behavior?" This makes it more obvious that there are answers that can't be found by staring at the source code. Vinge was likely aware of the unanticipated behavior of the Morris worm (https://en.wikipedia.org/wiki/Morris_worm) when he wrote Deepness. In the years since this book, we've seen more examples (stuxnet[1], ethereum[2]) of software with behavior unanticipated by -- and even weaponized against -- its creators.

(I'm very interested in this subject, and research ways to prevent unanticipated runtime behavior. My approach can be summarized as, "keep software small and simple to stay out of dark forests[3].")

[1] "accidentally spreading beyond its intended target" https://en.wikipedia.org/wiki/Stuxnet#History

[2] https://en.wikipedia.org/wiki/Ethereum#The_DAO_event

[3] https://www.paradigm.xyz/2020/08/ethereum-is-a-dark-forest