[jrockway]

Yeah. I've dealt with definitely-not-nation-states before, and their pattern was to sign up for free/cheap CI services (CircleCI, Github Actions, that sort of thing) and launch their attacks from there. The VPS thing also sounds very very plausible to me, I figured there was a long tail, but until I was looking up every network that was attacking us, I really had no idea how deep the long tail goes. I now feel like half the world's side hustle is to rent a server that they never update and host a couple of small business websites there.

[bredren]

> I now feel like half the world's side hustle is to rent a server that they never update and host a couple of small business websites there.

Do you mean people are offering build / host services for small biz, and leaving their servers in such a state they can be owned and used as jump points for intrusion?

Reason I ask is long-hosted small business websites are sometimes established with the intent to legitimize some future unrelated traffic.

[outworlder]

> Do you mean people are offering build / host services for small biz, and leaving their servers in such a state they can be owned and used as jump points for intrusion?

Probably not what's happening.

I've tried to build a cloud CI service a while ago. Per their nature, you _have to_ allow arbitrary commands to be run. And you also have to allow outbound connectivity. So you don't need to 'own' anything in order to be dangerous. They will not run with heightened privileges but that's of little help if the target is external.

It is pretty difficult to reliably secure them against being used as a source of attacks as there's a lot you can do that will mimic legitimate traffic. Sure, you can block connections to things like IRC and you can throttle or flag some suspicious traffic. You can't really prevent HTTPS requests from going out. Heck, even SSH is pretty much required if you are allowing access to git.

Generally speaking, a build service provider will try to harden their own services and sandbox anything that is run in order to protect themselves from being compromised. Most providers won't want to be known as a major source of malicious activity, so there's some effort there. AWS and other large providers have more resources and will easily ban your ass, but that doesn't matter if it happens after a successful attack was launched.

[jrockway]

That's exactly right. CI providers are good anonymizers for unsophisticated attackers because they provide an extra layer of obfuscation. But if they were doing something significantly harmful, I'd obviously be talking to those providers and asking for their own logs as part of the investigation, and then it would clearly link back to the actual culprits. So that was one popular technique to use to circumvent IP bans after abusing our service.

The whole hosting provider thing was another type of problems. I would always look at who owned the IPs that malicious sign-ups were coming from, and found a lot of ASNs owned by companies like "hosturwebsite4u.or.uk" and things like that. Those I assumed were just forgotten-about Linux boxes that the attackers used to anonymize through.

Ultimately, this was all to get a "free trial" of our cloud service, which did let you run arbitrary code. We eventually had a fairly large number of ASNs that would get a message like "contact sales for a free trial" instead of just auto-approving. That was the end of this particular brand of scammers. (They did contact sales, though! Sales was not convinced they were a legitimate customer, so didn't give them a free trial. Very fun times ;)

I should really write up the whole experience. I learned so much about crypto mining and 2020-era script-kiddie-ing in a very short period of time. My two favorite tangents were 1) I eventually wrote some automation to kill free trials that were using 100% CPU for more than 12 hours or something like that, and so they just made their miner run at 87% CPU. 2) They tried to LD_PRELOAD some code that prevented their process from showing up in the process table, but didn't realize that our tools were statically linked and that they were running in an unprivileged container, so the technique doubly didn't work. But, good old `ps` and `top` are linked against glibc, so they probably fooled a lot of people this way. They also left their code for the libc stub around, and I enjoyed reading it.