Hacker News new | ask | show | jobs
by reaperducer 2830 days ago
I do.

I have a tiny $5 Onion Omega2 on an independent cellular connection that checks file integrity on the production web servers every 15 minutes.

If the content of any of the files change, I get an e-mail.

If the alerts start coming in when I know I've just pushed a new version to production, the mail has a link that I can click that will re-scan all of the files and build new checksums.

If the alerts start coming in in the middle of the night, then I know something is up.

Obviously, this only works in small environments like mine where I'm the only one capable of updating the production servers. But it managed to catch a backdoor left in by the previous developer, who for some reason stored and updated his resume on the production server.
2 comments
noir_lord 2830 days ago
Ingenious, Have you considered a blog post on this approach?

It would be interesting to deploy a few of them in different places and check that they all see the same as well maybe.

Also did you do this as a belts and braces thing or is the system you are auditing particularly high security/risk in some way?

link
reaperducer 2830 days ago
It would be interesting to deploy a few of them in different places and check that they all see the same as well maybe.

That's a great idea. And since they're only $5 each (I think I spent $15 with the power shield), it's not a big deal.

Also did you do this as a belts and braces thing or is the system you are auditing particularly high security/risk in some way?

When I got here, most of the web sites were riddled with worms and trojans and spambots other bad stuff. One by one I just nuked them and started over. This was a deliberate isolation to keep an eye on things in case the sites or the dev machines ever got compromised.

link
ams6110 2830 days ago
You can just set up Tripwire to do this sort of thing. It's in most distro package managers.

https://github.com/Tripwire/tripwire-open-source/#open-sourc...

What that won't do is save you from malicious code inserted into 3rd party content (script libraries, etc.) that you load from a CDN. If you're worried about that, you should make a copy of a verified version and serve it yourself.

link
reaperducer 2830 days ago
You can just set up Tripwire to do this sort of thing

I wanted something that was completely independent of the machine. Separate box, separate network, separate architecture, etc...

What that won't do is save you from malicious code inserted into 3rd party content (script libraries, etc.) that you load from a CDN. If you're worried about that, you should make a copy of a verified version and serve it yourself.

I don't CDN on work projects. It's not worth the risk. If something goes wrong, I'd rather it be my fault and something I can understand and fix, whenever possible. Farming stuff out just leads to layers of things that can break, be compromised, or simply go wrong.

Again, it works at my scale (about 15 sites). It won't work for everyone.

link
foobarrio 2830 days ago
What if a developer's machine was somehow compromised and the bad version of the file was put into the actual repo or deploy? I have no idea how likely this is or if it is a possibility in NewEgg's environment but that would be a case where only scanning for changes on the servers wouldn't catch.
link
reaperducer 2830 days ago
What if a developer's machine was somehow compromised and the bad version of the file was put into the actual repo or deploy?

In a complex environment, that's a complex problem. In mine, it's not a big problem. Keeping the security routine on an external device with no other function I think helps. And since the device is on a completely different network, and a cellular connection with changing IP addresses, if someone was targeting the company they'd never find it.

That's the theory, anyway. So far, so good!

link