[Eiriksmal]

Because they never gained root access. I trust the logs in this case because their actions were immediately made known: Access logs show failed login attempts from that ip for a range of usernames, a successful login on the compromised account, then nothing but reams of mail being churned out.

If it had been done for more nefarious purposes, wouldn't "they" have been more discreet, carefully wiping traces of their activity from the logs? Not doing something that immediately throws red flags like sending thousands of email messages?

In all honesty, I certainly don't have the skills to detect an NSA-level attack that doesn't involve brute-force attempts on accounts. I can erase or alter logs, but then there are logs logged of me vi'ing logs, so I erase the shell history, but then that gets logged when I log out. It's a weird loop I don't know how to defeat, but some people do.

The heart of our problem was a misconfigured sshd that permitted remote logins (not root logins) on all user accounts. A disaster in the making. We got lucky that it was a spammer who compromised the system and not a competitor.

[zerohp]

> I can erase or alter logs, but then there are logs logged of me vi'ing logs, so I erase the shell history, but then that gets logged when I log out. It's a weird loop I don't know how to defeat, but some people do.

This is trivial, but you need to be familiar with the environment variables used by bash. unset HISTFILE

[yourad_io]

Or kill the shell from within, avoiding history write:

    $ vi /var/log/*.log
    $ kill -9 $BASHPID

[PeterisP]

The solution for log issue can be a remote specialized machine that does append-only logs and nothing else - it should be possible to lock down such a service so that if you're compromised, then at least you have unaltered data from the initial part of the attack, before they disable all logging.