[codezero]

I didn't catch this when it was first posted, I'm glad it showed up here.

Possibly you didn't get negative responses because you were sincere and introspective. Most people can understand the idea that when in need of money, you slowly make concessions and compromises that you wouldn't otherwise make.

I remember when I was at Red Hat, one of the engineers had found a way to clear a worm off Red Hat servers using our auto-update tool, but we weren't allowed to push it because of the possible unintended consequences.

I'd be really curious to hear about whether you had to face any of those kinds of things, were there any catastrophes (technically) when doing so much low level wrangling? It seems like one false move could drop half your nodes, were there any fail-safes?

Also, unrelated (and apologies for the Quora link), here's a really cool answer from someone who used to spam people: http://qr.ae/Ga85e It's very different from your experience in that they were non-technical and in a poor region, but still interesting to see global perspectives on similar work.

[mattknox]

I think you're right. If people talk honestly about why they did things and weren't obviously trying to do ill from the start, it's pretty easy to empathize.

I read that the participants in the Milgram experiments were _way_ disproportionally likely to be conscientious objectors later in life, and that they attributed this to having been in the experiments. I hope that I've been immunized like they were.

We were crazy careful about screwing up people's machines, because anything that we did that made it seem to malfunction would likely result in them reinstalling the OS over us, and while we had some ideas about how to persist across a reinstall, it was a few bridges too far.

I certainly can't say that we avoided all catastrophe, but I'm not aware of us ever causing one. We had pretty good abstractions: the stack-shuffling code was fully encapsulated so that it wasn't just littered about our normal code, etc..

We also tried pretty hard to avoid the really dangerous stuff. It sounds crazy to put arbitrary code in some random process, but if you know that it doesn't leak, and it never interacts with other threads in the process, it's not really _that_ risky, really.

One thing that probably also helped is that we had so much feedback from the individual ad clients. So we would know pretty fast if something started happening.

[codezero]

Really interesting. I've watched controlled environments of less than 50 nodes go completely haywire from bad code, so it's quite a feat to push code into 4 million questionable environments while at the same time dodging the virus soup from competitors.

[mattknox]

I think it's harder to safely do changes to development machines, because the coupling between components is greater. If you want to change, eg., libxml or something like that, lots of processes that you don't know about might be effected, and all hell can break loose.

By contrast, I was generally nuking random userland processes, which no process (or user) would mourn or miss. I think that is a lot safer. There were cases where we would touch something important, like the CreateRemoteThread stuff, but that was a relatively small amount of our code, which rarely changed, and again, it had very little interction with anyone else's code.

It's also possible that we _did_ create a lot of havoc, but I didn't know. I think that's less likely because I think we would have noticed the loss of revenue, but it's possible.