[londons_explore]

I guarantee you that there are 100+ people who could take Facebook down for 24+ hours if they went rogue.

For example the people responsible for the bootup scripts of Facebook infra could sneak in a "0 0 1 * * /bin/rm -rf ${TEMPDIR}/*" into crontab... They'd set the commit message as "clear out temp monthly" and it would get deployed across the entire fleet till in the first of next month every disk at Facebook gets erased because TEMPDIR isn't defined...

I guess they have enough pending stock to deter them...

[viraptor]

This wouldn't "delete" Facebook or many much smaller companies. It would result in maybe a small outage and get restored immediately in most cases. It's also an infra change you'd need across many systems - this isn't possible as a single change "across entire fleet".

This is not how non-trivial services work.

[austinheap]

That’s...now how any of this works. You can’t just change integrity-bearing things without FIM systems kicking in. And you’d need collusion to get something mainlined that would bypass that.

[londons_explore]

You don't need collusion - just a code reviewer not paying proper attention.

[tracyhenry]

How though? Every past author of that script would be notified of such a change. It'd be insane if all of them would pretend they didn't see it and accept that change.

[londons_explore]

It would be hidden amongst a big refactor, and it would have a bunch of unit and integration tests that all work correctly (because they do set the environment variables correctly).

Writing code that appears to do one thing and actually does another is very doable - you only need to hide one malicious line amongst thousands in a code review, while the reviewer needs to inspect every line.

Besides, the vast majority of code reviews are 10 minutes or less.

[tracyhenry]

I think you oversimplified the code merge process at Facebook. Their internal code review system (Phabricator) would automatically add tens of reviewers to your PR because you changed an important script authored by those people. Also, there are linters that would call out the use of potentially hazardous commands. Actually deploying the thing is another process that requires jumping through a series of other review hoops.

p.s. I had a short internship there.

[ta20210405]

Facebook servers do not have cron installed.

[londons_explore]

Example using widely understood tooling... But there are plenty of other one line changes with similar impact