Even in my plugins, I don't actually use JUCE DSP primitives. It's mostly a product shell and distribution framework, two things I have no need for in this codebase.
I understand .. but its not 'mostly a product shell' and distribution framework, there are a lot of things JUCE brings to the table that might've been useful in this context, such as being able to host plugins and adjudicate resources (shared memory primitives, extremely easy parameter setup and management across DSP and MIDI frames, easy integration of common codebases into both DAW and standalone versions, etc.) that are generally good for DSP .. you're not obliged to use JUCE's DSP incidentally, I have my own synth project based on JUCE with both ARM and x86 SIMD implemented side-by-side .. JUCE' primary value is as a runtime environment, which I guess you are more interested in setting up, yourself, and I get that, so yeah ..