[Roark66]

Personally I think the root cause of this was bad documentation practices. If the old system was properly documented they would've scanned the right folder.

Likewise with the certificate, if there was documentation to indicate when that cert expires (or monitoring to alert few weeks in advance) they would have a functioning ids and these web shells would be found immediately.

Unfortunately, out of half a dozen fortune 500 companies I worked for perhaps 2 had doc practices good enough to prevent this.

[yuliyp]

That feels like the wrong conclusion. Assuming documentation will be followed properly is not a reasonable security strategy. Validation and monitoring is needed. That their NIDS gracefully degraded to a "don't monitor the payloads" when it was expected that it would be monitoring those and nobody noticed is a problem. A scan of a system which misses a web server running it without erroring is a problem.

[hn_throwaway_99]

Couldn't agree with this more. While I think it's important to have good documentation, it is nearly always a very bad idea to rely on that documentation being 100% correct. Businesses simply have way too many moving parts to assume the state of the world is always up-to-date in the documentation.

You also highlight a very good point. Things like security software should "break loudly", i.e. beyond just sending alerts (which can be ignored), there should be some explicitly "painful" steps that occur if the security system is in a broken state for long.