Hacker News new | ask | show | jobs
by finnlab 111 days ago
Nice work, but I feel like it's not required to use AWS for this. There are small hosting companies with specialized servers (50gbit shared medium for under 10$), you could probably do this under 100$ with some optimization.
2 comments

I did some crawling on hetzner back in the day. They monitor traffic and make sure you don't automate publically available data retrieval. They send you an email telling you that they are concerned because you got the ip blacklisted. Funny thing is: They own the blacklist that they refer to.
This. I tried to run a very slow DHT scraper I was writing on a Hetzner server and within minutes they were on my ass. I don't want to make an enemy of them so I killed it immediately, but they are clearly very sensitive to anything outside of "normal".
If Hetzner actually puts their own customers on their blacklist then that list becomes more trustworthy.

They were right to blacklist you, they were right to complain to you, and they were right not to assume malice and kick you off their platform/shut down your server.

Yes I wasn't banned or anything, they aren't barbarians. Also, explain your opinion don't just put it out there. This is not a football match.
I felt like the arguments behind my opinion were implicit.

Consistently not showing bias makes for trustworthy lists. Nobody (well, nobody reasonable I'd argue) will trust an IP blacklist from a major hosting company that actively excludes their own customers.

As for banning customers that get blacklisted, this does actually happen, especially on the more affordable cloud hosts that get plagued with massive scraper and bot loads. Anything from suspended, inaccessible servers that require manual intervention to network caps or CPU load caps. This is a rather extreme measure, but not one uncommon or even unacceptable depending on the exact IP blacklist you managed to trigger.

Maybe you didn't get the nuance here:

Hetzner says (not exact wording) "according to this public blacklist you seem to be performing something they consider bad".

What they don't mention is that they operate the list.

This. AWS is like a cash furnace, only really usable for VC backed efforts with more money than sense.