This only makes sense for vulnerabilities that can actually be exploited in your particular use-case and configuration of the library. A lot of vulns might be just noise and not exploitable so no need to patch.
Problem is code bases are continuously evolving. A safe decision now, might not be a safe decision in the future. It's very easy to accidentally introduce a new code path that does make you vulnerable.