[gopi_ar]

Funny thing is we don't do uploads anywhere and there's no CMS whatsoever.. Which leads us to believe it's an OS vulnerability.

Would you how we could hire professionals to investigate this for us? And report it to appropriate groups..?

PS: These are dedicated servers :-/

[throwbsidbdk]

You could hire a firm that specializes in this sort of thing, but it's going to be expensive. Look for the guys that build security tools, like a company that worked on metasploit or has submitted multiple bug bounties. I know some of the makers of anti virus software will investigate this kind of thing for a (steep) price.

Basic computer forensics needs a copy of the drive as unaltered as possible so you should start with that before running or installing anything. Basically don't run the server if you want to be able to get anything our of it.

If it's not a user data breach its par the course to reintsall and sweep it under the rug lol.

Next time make sure the server takes snapshots and dumps logs to an external place where they can't be deleted.

[flukus]

>Which leads us to believe it's an OS vulnerability.

Have you ruled out an internal source that decided to use some of your "spare" capacity? Or a previously internal source that might not have had all their privileges revoked?

[nodesocket]

Can you answer the following for us?

    Is ssh only allowed by public key? (in /etc/ssh/sshd_config => PasswordAuthentication no)
    Is Apache or NGINX running on the server?
    Is PHP/Ruby/Node/Python running apps?
    What ports are open in iptables (iptables -L)
    What does /var/log/auth.log say?

[gopi_ar]

Thank you for responding.

We searched the whole system for authorized_keys files and found one created in a /var/lib/redis/ of a staging container (with no firewall) on this host. We then came across the redis vulnerability https://kevinchen.co/blog/postmortem-server-compromised/ . A junior dev had spawned this container without help from dev-ops and hence left ports open.

What doesn't make sense to us is how this daemon (yam) was running under a statd username when the container doesn't have such a user, but the host does? Are LXC containers able to run daemons on the host?

[arpa]

Well, it is always possible that attacker broke out of the container, as the container is still running under the same kernel, only its process(es) are chrooted + namespaced. Containerized "root" supposedly has its' privileges cut down, but if the kernel is exploitable...

[jmgao]

> What doesn't make sense to us is how this daemon (yam) was running under a statd username when the container doesn't have such a user, but the host does? Are LXC containers able to run daemons on the host?

This is because usernames don't exist, as far as the kernel's concerned. ps is resolving the process's UID to the corresponding name for the outside context, not the one inside the container.

[gopi_ar]

This makes sense, we can rest easy knowing they didn't break out of the container. Thanks!

[stevekemp]

Is SSH available to the outside world? Many many compromised hosts spend all their time trying to login to remote hosts with dictionaries of usernames/passwords to try.

If your (root) password is weak then I'd not be surprised if that was the source of the infection.

You might see logins via "last", or via the system logs.