[saurik]

While this isn't guaranteed (all tools, including the compiler, may be patched), you can use checks and balances: verify /proc doesn't contain phantom processes, compile your own copy of ps, try more-obscure tools like top. If by "understand" you just mean "notice"... well, you don't, until one day you accidentally stumble across one of the above and start digging. (Maybe, for example, you install some kind of server monitoring tool, and when you log in to the web portal it provides you see a process that you find very suspicious; when you use ps, it doesn't show.) In my case, I've noticed this kind of thing twice: once, when the tool was binary pacthed to death (and just crashed), and once when the "patch" was "replace binary entirely", and the replacement was older and did not support a command line argument I knew that it should.

[ultimoo]

Nice. So both times that you noticed this, was it malice on someone's part due to which the tools were patched? I have just never heard or encountered such a situation and am frankly paranoid about something like this happening to one of the tools I use.

[saurik]

I've been pwned with 0-days in various email servers: sendmail over a decade ago, and exim4 more recently (still many years ago, though). The patched copy of ssh on one of my boxes was then distributing passwords to someone, and which then was used to gain access to another machine.

What I'm always paranoid about is that I work in a community of security researchers that sit on and occasionally drop 0-days: I have very little trust that much software is actually remotely "secure". Meanwhile, the only reason I had noticed those other attacks is just how sloppy they were... a more targeted-to-me run by a more careful attacker would have maybe never been noticed.

It has drastically changed the way I think about security, FWIW; as one example: I don't every store logs on a box being logged anymore. Instead, logs are immediately transported to another machine whose only purpose is to accept and store logs (and so is listening for incoming log packets, OpenSSH, and nothing else. The first thing anyone does is attempt to patch themselves out of logs (one attack I noticed because wtmp was mysteriously damaged).

[saurik]

(When I said "more obscure tools like top" I meant to say "pstree" but edited the statement one too many times before posting.)