How would you know for sure that it didn't dump the database to somewhere in Asia once "they" have your server under control? Serious question, because how can you trust the logs? (Mind you, I'm not that technical)
Once the servers been comped, you can't tell what's been accessed on that machine. You could possibly find out if it accesses other machines within your network (logging depending) - however if someone were to root a public facing server that had a bunch of files on it, you have to assume they've been seen/duplicated.
And, also, we operate in a low-tech service industry where simply having a database of customers is considered moderately cutting edge. We're not a software company producing hacking tools for evil governments and their puppets. There's nothing interesting on the server for anyone save our competitors. That leads me to logically deduce that the "hacking" attempts the internet-facing servers experience simply fall into the net of trolls searching for more machines to add to their botnets.
All the logs over the years simply show spam from bots idly probing for pirated SIP lines/extensions on our VoIP box, attempts to send mail through our mail server, and open PHP MyAdmin/Django/Wordpress login pages--none of which are present because none of that software's in use.
Because they never gained root access. I trust the logs in this case because their actions were immediately made known: Access logs show failed login attempts from that ip for a range of usernames, a successful login on the compromised account, then nothing but reams of mail being churned out.
If it had been done for more nefarious purposes, wouldn't "they" have been more discreet, carefully wiping traces of their activity from the logs? Not doing something that immediately throws red flags like sending thousands of email messages?
In all honesty, I certainly don't have the skills to detect an NSA-level attack that doesn't involve brute-force attempts on accounts. I can erase or alter logs, but then there are logs logged of me vi'ing logs, so I erase the shell history, but then that gets logged when I log out. It's a weird loop I don't know how to defeat, but some people do.
The heart of our problem was a misconfigured sshd that permitted remote logins (not root logins) on all user accounts. A disaster in the making. We got lucky that it was a spammer who compromised the system and not a competitor.
> I can erase or alter logs, but then there are logs logged of me vi'ing logs, so I erase the shell history, but then that gets logged when I log out. It's a weird loop I don't know how to defeat, but some people do.
This is trivial, but you need to be familiar with the environment variables used by bash. unset HISTFILE
The solution for log issue can be a remote specialized machine that does append-only logs and nothing else - it should be possible to lock down such a service so that if you're compromised, then at least you have unaltered data from the initial part of the attack, before they disable all logging.