[bostik]

Yup, according to these "researchers" having robots.txt on your website is enough to warrant a CRITICAL vulnerability.

No, I'm not joking. That's one of the reports I saw in November. I've also had to triage the claim that our site supposedly has a gazillion *.tar.xz files available at the root. All because the 404 handler for random [non-production relevant] paths is a fixed page with 200 response.

As far as I'm concerned, running a bulk vulnerability scanner against a website and not even checking the results has as much to do with security research as ripping wings off of flies has to do with bioengineering.

[wiredfool]

Oh god. One client I work for does automated scans, and we had an s3 bucket set up as a static site.

They freaked out when /admin/ returned permission errors, essentially a 404, because it was information leakage about admin functions of the website.

[glandium]

That happens when you disable directory enumeration (or whatever name that has) on S3. In that case, it sends 403s (permission denied) instead of 404s.

[wiredfool]

I know, but try explaining that to someone in very small words. There is no admin. There is no login. The api has open CORS because we want reuse and there’s no risk because there’s literally no concept of identity in the app. Everything is public data or f(public).

Scanners see things through their eyes, and they’re not used to static/public.

I the end, It was easier just to rewrite 403 into 404.