Hacker News new | ask | show | jobs
by VWWHFSfQ 2234 days ago
The intruders had root access to every server in a salt deployment for who knows how long and yet everyone is claiming there's no evidence that any data or secrets (customer's or otherwise) were exfiltrated from the network. This is a very dangerous assumption. Nobody has any idea what was run on the servers since it seems that once the initial attack script was deployed it downloaded and executed new scripts every 60s and then removed themselves. Pretty standard C&C ops. It may have started as a mining operation, but that doesn't mean it was the only thing it was doing.
3 comments

> ... and yet everyone is claiming there's no evidence that any data or secrets (customer's or otherwise) were exfiltrated from the network.

A number of people have carefully reviewed the payload that was deployed to servers, especially during what we're calling v1-v4 of the attack. (v5 onwards got more complex, but that wasn't until Monday (with variability for timezone).

> Nobody has any idea what was run on the servers ...

Well that's not true - there's a number of victims that have useful IDS tools, including auditd, plus the review of binaries and shell scripts deployed, etc.

Some of us also have netflow collection at the edge, and can review connections initiated from within our networks.

> ... once the initial attack script was deployed it downloaded and executed new scripts every 60s and then removed themselves.

I don't think any of us have found scripts that removed themselves. While that may sound naive, there's a few researchers that have been analysing these tools, including via large honeypot networks, and this just hasn't (at least for the first 2-3 days) been a profile of the attack.

Thankfully - and I appreciate it's very weird to say this - the initial attacks were very much vanilla crypto currency mining opportunities. It could have been a lot worse, and algolia's assessment matches a lot of other independent assessments on this front.

I hope for everyone's sake that it was just a naive crypto mining operation. But given the length of time this vulnerability was available, and the extent of access it allowed, I just find it very hard to say with any certainty that we know everything that it was doing. Exploits like this get passed around in nefarious circles pretty regularly. One of the scripts I saw went to great lengths to eliminate competing crypto miners from the systems so they could run their own. That tells me there were multiple people (or groups) exploiting this in competition with each other.

You said the v5 of the attack got more sophisticated. How do we know there wasn't a "v0" that was even more sophisticated and innocuous? You can't trust the server logs. Firewall tables were flushed, SELinux was disabled. It's just really hard to say the full extent of damages.

You're absolutely right that we can't be 100% confident, and best practice dictates a full rebuild from known sources, as usual after IOCs especially of this magnitude.

However, the number of public and non-patched salt servers might be considered a sufficiently small volume for bad actors to have investigated, who can say why it took so long to see genuinely malign attacks.

> One of the scripts I saw went to great lengths to eliminate competing crypto miners from the systems so they could run their own. That tells me there were multiple people (or groups) exploiting this in competition with each other.

It wasn't very sophisticated - just a series of kill statements. This tells me that the author of that script picked up an existing script that's probably been around for years and adjusted it to their needs.

The script also tried to kill confluence, amongst a handful of other large, relatively rare applications, which further suggests this was old fashioned copy-pasting by some non-sophisticated script kiddies ... or someone just wanting to do a PSA and draw attention to this exploit, and making a few BTC for their troubles. Who can say.

We don't know there wasn't a 'v0' - but we're fairly confident. Unless it was disabled as soon as 'v1' popped up, you'd expect honeypot systems to identify non-benign variants - and honeypot systems were identifying modest, reversible changes and nothing in the way of data exfiltration.

By Tuesday or Wednesday of this week I expect there were more (and worse) exploits than could be tracked, though, and some people are really going to suffer as a result.

Hello,

I'll try to give you some insight as I'm a security engineer at Algolia.

Your concern is valid, and it's true, we cannot know for sure. That's the reason why, as explained in the blog post, we are reinstalling all impacted servers and rotating our secrets. If our assumption is false, this should contain the issue.

That being said, we have good reasons to make that assumption.

- Our analysis of the incident and how the malware behaved on our systems didn't find any evidence towards access and transfer of data.

- There are other public analysis of the malware. Other companies hit have the same analysis than us, and you can have a look at https://saltexploit.com/ which is maintaining an interesting list of what is known on the attack, how it behaved, and how it's evolving fast to adapt.

I hope this answers your concern.

I agree. I would like to seem more details of how they determined it was only crypto mining. Finding only mining scripts in your logs doesn't mean they were not running other code once they had root.
It seems bizarre to me that a crypto miner got in. It wouldn't make much money on regular CPUs, and the high processor usage would immediately draw attention. So it looks like a low-effort botnet, which is embarrassing to get pwned by.

(The coin mining could be a cover like you mention, but it seems unlikely since it naturally draws attention.)

It’s easier to sell Monero for cash than... some random data from some random company.
I once worked at a place where a minor piece of cloud infra got exploited. All the attacker did was run a monero miner on it.
Heh, in a way it makes a good bug bounty. Like if popping calc got you a trickle of income.
> It wouldn't make much money on regular CPUs

Not true; some PoWs such as Random-X are designed to be most efficient CPUs.

running the virus code in a container / vm and checking what gets modified