[ultimoo]

Wow, he actually patched 'netstat' and 'ps'. It must have been to hide certain processes and port numbers from showing up. I wonder how may one go about understanding if they are using a 'hacked' version of something as non-trivial and comprehensive as 'ps' and 'netstat'. I mean, if I'm not suspicious, I wouldn't give it a second though and trust the output of these commands.

[saurik]

While this isn't guaranteed (all tools, including the compiler, may be patched), you can use checks and balances: verify /proc doesn't contain phantom processes, compile your own copy of ps, try more-obscure tools like top. If by "understand" you just mean "notice"... well, you don't, until one day you accidentally stumble across one of the above and start digging. (Maybe, for example, you install some kind of server monitoring tool, and when you log in to the web portal it provides you see a process that you find very suspicious; when you use ps, it doesn't show.) In my case, I've noticed this kind of thing twice: once, when the tool was binary pacthed to death (and just crashed), and once when the "patch" was "replace binary entirely", and the replacement was older and did not support a command line argument I knew that it should.

[ultimoo]

Nice. So both times that you noticed this, was it malice on someone's part due to which the tools were patched? I have just never heard or encountered such a situation and am frankly paranoid about something like this happening to one of the tools I use.

[saurik]

I've been pwned with 0-days in various email servers: sendmail over a decade ago, and exim4 more recently (still many years ago, though). The patched copy of ssh on one of my boxes was then distributing passwords to someone, and which then was used to gain access to another machine.

What I'm always paranoid about is that I work in a community of security researchers that sit on and occasionally drop 0-days: I have very little trust that much software is actually remotely "secure". Meanwhile, the only reason I had noticed those other attacks is just how sloppy they were... a more targeted-to-me run by a more careful attacker would have maybe never been noticed.

It has drastically changed the way I think about security, FWIW; as one example: I don't every store logs on a box being logged anymore. Instead, logs are immediately transported to another machine whose only purpose is to accept and store logs (and so is listening for incoming log packets, OpenSSH, and nothing else. The first thing anyone does is attempt to patch themselves out of logs (one attack I noticed because wtmp was mysteriously damaged).

[saurik]

(When I said "more obscure tools like top" I meant to say "pstree" but edited the statement one too many times before posting.)

[drzaiusapelord]

Ideally, you should be running some kind of tripwire scan. If the hash for common utilities changes and you didn't update those binaries, then something bad has happened.

Hacked versions of common utilities is a common payload for rootkits.

[tptacek]

If you lose superuser, you've almost certainly lost your kernel, at which point your only hope for Tripwire-type scans actually working is an attacker that doesn't know where to download a good rootkit from.

You should still run the scans, just be aware of the limitations.

[shuzchen]

That would be helpful for rootkits coming from outsiders, but would only serve to slow down (not stop) an insider. An insider knows what protections are in place (probably implemented them too) and can defeat the hash check if she knew how the hash was calculated, or can ship the binary alongside a regular update.

[bigiain]

It's been some time now, but I've been there...

Once you've lost root to a sufficiently competent attacker you can't trust _anything_ on that box any more.

One thing that'll help against (some) script kiddies with rootkits is to have available statically linked copies of every tool you might want to use to see what's happening on your box - for a long while, every colo-ed box I managed had a cdrom drive with read-only versions of /bin, /sbin, and useful bits of /usr - all with the binaries statically linked. They can in handy a few times (mostly to confirm that "yep, we're screwed. Get this box off the network and powered down immediately and implement the bring-up-a-new-server-from-scratch plan right now".

At some stage though, you can't trust the kernel or the hardware - an attacker who's got into your booted kernel or your bios or your network card firmware, if they're good enough, they probably can't be detected by examining anything you could see logged into the box itself. The only way to identify that level of attack is by monitoring the traffic from the box from some trusted piece of network gear upstream of your rooted server (and against a sufficiently talented attacker, even identifying unexpected outbound traffic might be impossible. If your list of likely attackers includes three letter agencies or nation states, I hope your getting your secutiry advice from somethere other than HN comments…)

[trotsky]

Read only media is good advice and low effort compared to most viable defense. Certainly valuable at times worth doing. Starting from the assumption you'll get beaten typically pays off over perfecting the hack-proof nirvana.

Nobody should lose any sleep about BIOS embedding and similar - that level of attacker and sponsor imply a level of threat that no typical organization has a chance against.

In my opinion, after years of pondering dozens of intrusions with many types of ways in and regular failure of all kinds of defenses I don't think there is much advice to give aside from the flaw is in your custom software, stupid.

I have become a really big advocate of CM and push button provisioning with identical replacement hosts that build from scratch and commonly get refreshed - relying on code and configurations managed centrally.

The best way to remove an attacker is a complete rebuild. If you're already using Chef etc why not just dump them proactively? Some roles don't lend themselves to this, but I assure you it is like a massive weight has been lifted.

[w1ntermute]

Reminds me of Ken Thompson's Reflections on Trusting Trust: http://cm.bell-labs.com/who/ken/trust.html

[__david__]

> I wonder how may one go about understanding if they are using a 'hacked' version of something as non-trivial and comprehensive as 'ps' and 'netstat'.

So this happened to me once, but it wasn't ps or netstat but sshd that had been replaced. We only noticed it because the replacement sshd was bad and didn't set up the PATH correctly so "darcs push" stopped working (it was presumably set up for a different linux distro than we were running).

[MertsA]

If it's a Redhat based server just run rpm -Va and that will verify the integrity of every rpm package on the system. If you're in a situation where you can't trust the current computer you can run it in a chroot from a known good copy.

[badgar]

He had root, so he could have instead installed a rootkit, which can hide the existence of processes from all of userland. In a graduate OS class I took, we had an assignment to do hide a process live on OpenSolaris 8 using the kernel debugger (kdb). I wrote some assembly and overwrote some bytes in the syscall functions for process listing. We were on developer builds so you could just use the function symbols by name in kdb. I forgot to cover /proc/ though.

[ttrreeww]

Amateur hour, quite a few rootkits out there are almost completely undetectable...

[lawnchair_larry]

Really? Can you name one?

[stevenrace]

Anything that lives outside of userspace.

If one has root and patience - flash BIOS and wait for a coldboot. One can even get an IP stack to pull down new firmware between boots. The user sees a normal post screen and your hypervisor sees normal hardware adapters.

See Jonathan Brossard's 'prior work' slide from his Defcon talk on his work [1] for more details on the state of X86 backdooring.

'Trusted computing' and all that.

[1] http://www.youtube.com/watch?v=yRxDvkKBMTc

https://media.defcon.org/dc-20/presentations/Brossard/DEFCON...

[lawnchair_larry]

I know how they work, I'm asking GP to name one that he could have used. There isn't actually much "out there" that is undetectable. You'd likely have to write your own, which is highly non-trivial.

[trotsky]

Obviously undetectable isn't true, but i'd guess that kbeast is likely to go undetected in all but the most prepared operations. Forensics shouldn't count since you already know you're fucked by then.

[ttrreeww]

You get what you paid for.