[vbernat]

I often run a linkchecker on my blog and substitute broken URL with links to the Wayback machine. Unfortunately, this is becoming quite difficult to detect broken URL as everybody is fighting bots. I am using linkchecker <https://github.com/linkchecker/linkchecker/> and it respects robots.txt but many sites are now serving 503 or various other codes.

[sph]

I built a commercial link checker SaaS and this was one of the reason I decided to abandon the endeavour. Had plenty of customers complaining that some links had 403 or similar errors, when in reality it's because of Cloudflare. If you add heuristics to ignore Cloudflare's errors, your link checker is pretty much useless as CF is in front of every bloody website on the internet nowadays.

You can't win. Also I learned that people and companies don't really care if their links break, let alone pay for a link checker service. I am guilty of this on my personal website.

[vbernat]

I don't think CF is that much present, but yes, the situation is dire.

[sph]

Well, I disagree and I have seen the data.

[vbernat]

It's easy to disprove. This website is not behind CF. Also: https://backlinko.com/cloudflare-users.