I have a tiny $5 Onion Omega2 on an independent cellular connection that checks file integrity on the production web servers every 15 minutes.
If the content of any of the files change, I get an e-mail.
If the alerts start coming in when I know I've just pushed a new version to production, the mail has a link that I can click that will re-scan all of the files and build new checksums.
If the alerts start coming in in the middle of the night, then I know something is up.
Obviously, this only works in small environments like mine where I'm the only one capable of updating the production servers. But it managed to catch a backdoor left in by the previous developer, who for some reason stored and updated his resume on the production server.
It would be interesting to deploy a few of them in different places and check that they all see the same as well maybe.
That's a great idea. And since they're only $5 each (I think I spent $15 with the power shield), it's not a big deal.
Also did you do this as a belts and braces thing or is the system you are auditing particularly high security/risk in some way?
When I got here, most of the web sites were riddled with worms and trojans and spambots other bad stuff. One by one I just nuked them and started over. This was a deliberate isolation to keep an eye on things in case the sites or the dev machines ever got compromised.
What that won't do is save you from malicious code inserted into 3rd party content (script libraries, etc.) that you load from a CDN. If you're worried about that, you should make a copy of a verified version and serve it yourself.
You can just set up Tripwire to do this sort of thing
I wanted something that was completely independent of the machine. Separate box, separate network, separate architecture, etc...
What that won't do is save you from malicious code inserted into 3rd party content (script libraries, etc.) that you load from a CDN. If you're worried about that, you should make a copy of a verified version and serve it yourself.
I don't CDN on work projects. It's not worth the risk. If something goes wrong, I'd rather it be my fault and something I can understand and fix, whenever possible. Farming stuff out just leads to layers of things that can break, be compromised, or simply go wrong.
Again, it works at my scale (about 15 sites). It won't work for everyone.
What if a developer's machine was somehow compromised and the bad version of the file was put into the actual repo or deploy? I have no idea how likely this is or if it is a possibility in NewEgg's environment but that would be a case where only scanning for changes on the servers wouldn't catch.
What if a developer's machine was somehow compromised and the bad version of the file was put into the actual repo or deploy?
In a complex environment, that's a complex problem. In mine, it's not a big problem. Keeping the security routine on an external device with no other function I think helps. And since the device is on a completely different network, and a cellular connection with changing IP addresses, if someone was targeting the company they'd never find it.
Whenever someone complains about another company's product, code, features, security....I always wish it was mandatory to include a link to the kind of software the poster is putting into production.
I have. I deploy in ASP.NET and get a hash of the uploaded DLL. I check it twice a day. Never had any incidents to this day but as the saying goes, it's better to be safe than sorry.
In ASP.NET pretty much all output, including html and js is included in the DLL. Only external js files are left out. Sure they could hack them which means my solution isn't bulletproof but I could also produce a hash for them too.
I have a tiny $5 Onion Omega2 on an independent cellular connection that checks file integrity on the production web servers every 15 minutes.
If the content of any of the files change, I get an e-mail.
If the alerts start coming in when I know I've just pushed a new version to production, the mail has a link that I can click that will re-scan all of the files and build new checksums.
If the alerts start coming in in the middle of the night, then I know something is up.
Obviously, this only works in small environments like mine where I'm the only one capable of updating the production servers. But it managed to catch a backdoor left in by the previous developer, who for some reason stored and updated his resume on the production server.