I would agree in principle, but that would require an extensive parser for files it thinks it finds... it's easier to deal with this on the server side than the client side.
No it's not. Dealing with it on the server side means that all tools and servers in the world have to go along with this decision. Posting on a blog saying that people 'SHOULD' do something and actually making it so that this is a reasonable expectation are two very different things.
A vulnerability scanner having to implement hairy heuristics to decide what's a vuln and what's a common false positive is literally its whole job.
There is a standard for HTTP status codes that has dictated what web servers “should” do for quite a while, now. Most web servers respond properly with a 404 “out of the box”, when asked to serve up content that doesn’t exist.
I think that's still somewhat beside the point though. in the case of Prometheus yes it should return a 404. but what if it was nginx routing all paths to some app? or even just some actual file being served on that path? in either case the vuln scanner says you have an exposed home directory, and that's a false positive.
Almost any web server can be configured to provide a generic response to a specific request, for example by ignoring some or all of the url path, and in practice I would bet that a majority of actual instances do this for at least some sets of requests. (To confirm my theory in 20 seconds I checked if https://news.ycombinator.com/user?id=nonexistent_user_1620 returns a 404 or a 200 - it's the latter.)
It's silly to pretend that the use of a 404 in this type of circumstance is either clearcut in the standards or ubiquitous in practice.
> It's silly to pretend that the use of a 404 in this type of circumstance is either clearcut in the standards or ubiquitous in practice.
The standards seem pretty clear to me.
I would point out that technically, the path portion of the HN URI does indeed point to a valid endpoint, it is the query portion of the URI (usually not used by the server to do any routing) that points to a non-existent resource.
Still, HN is wrong here and should be returning a 404 status.
> Posting on a blog saying that people 'SHOULD' do something
To be fair, it's not about some rando on a blog post. The HTTP response codes and their semantics are in the RFCs. And while it's true the RFCs say SHOULD and not MUST, there are also now ~30 decades of experience and expectation that a non-existent resource is more likely a 404 not 200.
Sure a server can always do what it likes but can't expect it to play well with the outside world if it goes against both convention and documentation.