In this case, "too high" = "we haven't figured out how to commoditize it well enough yet because the skill floor/ceiling is too high to scale with cheaper talent." This was one of the primary reasons I began to specialize partially in tech domains that are regarded as "hard" (compilers/perf): much smaller market means employees have a bit more leverage.
Do you feel there is much demand in those areas? I'm interested in more theoretical things but have doubts that many jobs would be available or for them to be valued even though its more difficult
Definitely a tradeoff: much fewer companies, but the ones that are left are higher quality if they are seeking out that sort of thing. It really depends on what you're looking for.
I settled into a role of research software engineer, where I do both applied research and development, applying a lot of compiler-ish stuff to different domains within cybersecurity, such as building out control flow graphs from binaries, thinking about how to instrument assembly code efficiently, fast pattern matching, and static analysis, where I am currently. The role fits me like a glove, but it isn't for everyone. In my job search, I started at, "I want a job doing compiler work," and eventually broadened scope a few times until I landed on, "I want a job where compiler-type approaches are on the table of possibilities." This offers a wider variety of work, which I like.
I can discuss more over email (check my HN profile) if you'd like, but most of what I know is US-centric due to how funding works for these types of research. Larger orgs like FAANGs also have it, but the pool is much more competitive, as you'd expect.
I'd love to have some more discussion especially on what areas within cybersecurity may have good demand :) Although I can't find the email in your profile or github!
Those parts you mentioned like the static analysis or control flow graphs sounds cool