To piggy back on this comment: what build system are you using? I've found that bazel has greatly reduced compile times, even when using boost and opencv.
I have used qbs with great success. It's by far the fastest and most correct¹ build system for C++ that I've used so far, and it's configuration ain't so bad (tm).
¹ I used it in a couple projects and never had a single incorrect build. With most other build tools, be it CMake or Gradle, these happen regularly for me.
I just took a very quick look at Bazel and saw that Windows support is "Highly Experimental" which concerned me. I've been looking around in this space recently as I'm getting into C/C++ again (ARM uC stuff) and all three platforms are fairly important to me for what I'm doing. I'm a big Gradle fan and Bazel seems like a natural next step if it's not problematic.
¹ I used it in a couple projects and never had a single incorrect build. With most other build tools, be it CMake or Gradle, these happen regularly for me.