[phoronixrly]

So is this the crawler that has been constantly hammering all my applications searching for these files from the very second I first issue a TLS cert for them? Thanks to you I've had to put fail2ban on all my public-facing web servers...

How about you be a good netizen and make it so people can request to be scanned and don't proactively do it, let alone constantly keep hammering them with requests?

[bblb]

I need to protect against the malicious good guys, the shodans of the world. Peeping inside your windows and trying your front door handles. Querying every string imaginable and port pinging all 65536 of them.

And I need to protect against the actual criminals already inside my house looking for something to steal. Scouting out the easy target to setup their ransomwares.

[graemep]

Lots of crawlers do this. I have never seen a webserver that does not get a variety of these from obviously different sources. Even just an ssh port will get a lot of malicious login attempts.

[phoronixrly]

I don't get what you mean by that. Everyone does it so it's OK?

No - scanning without consent is abuse and anyone doing this is malicious regardless of what they tell themselves or anyone else.

[graemep]

I am not saying its OK. I am saying it is probably not this crawler that is hammering your site, and definitely not only this crawler. I specifically mentioned malicious attempts at ssh logins.