The answer to your question is WebKit (because iOS), kernels (XNU, Linux, Windows) etc. In case you are not familiar with the domain I'd start with user-space exploitation and relevant write ups to get my feet wet. You'll find plenty of write ups, blogs etc. so I'll skip those.
Some of the books I generally found interesting are [1],[2], [3]. There's more to that, including fundamental concepts of CS (e.g., compilers and optimization in JITs, OS architecture etc.). I believe also https://p.ost2.fyi/dashboard has some relevant training.