[Ndymium]

For some reason, Facebook has been requesting my Forgejo instance's robots.txt in a loop for the past few days, currently at a speed of 7700 requests per hour. The resource usage is negligible, but I'm wondering why it's happening in the first place and how many other robot files they're also requesting repeatedly. Perhaps someone at Meta broke a loop condition.

[antonyh]

As facebookexternalhit is listed in the robots.txt, it does look like it's optimistically rechecking in the hope it's no longer disallowed. That rate of request is obscene though, and falls firmly into the category of Bad Bot.

[RobotToaster]

My guess is it's dutifully obeying it, not storing anything from the site and then exiting, without clearing the site from the crawl queue.

[mghackerlady]

That is probably the dumbest yet most genius solution to getting your scraper blocked I've ever seen