[aerotwelve]

I find myself asking the same things when it comes to the lower-level software side of things. I find this area fascinating as well, but it's all so "invisible" in modern systems.

I graduated from a CS program (BS) a few years ago, and the projects I'm still most proud of existed "lower" in the stack: implementing FAT16, writing compilers/interpreters (I still have my Brainfuck interpreter!), playing around with paging in MULTICS, and the like.

These were all very toy-like (with good reason, a semester is only so long), but since I enjoyed them I've found myself asking things like "I wonder how the process scheduler/virtual memory manager/in OS X is handling [whatever I'm doing at the moment]? What does my stack look like right now? How are all of these threads communicating with each other, what are they saying?"

You can occasionally see this in the console when something is going wrong, but usually not when the system is operating normally. When the OS is handling a heavy load brilliantly, that's kind of when I'm most impressed, and therefore most interested.

(maybe this kind of procedural output would be dreadfully boring or unreadable due to the complexity of a modern system if I actually saw it, I don't actually know of course.)

[exikyut]

`perf top` on Linux, and doing a (really) deeeep dive on the perf and BPF APIs, may be of interest.

The possibly-incorrect impression I get of the performance-analysis related areas of the kernel is that they're a bit siloed (in the same way that X11 is siloed, and only a very small group of people look at it), which may render it bit functionally academic. (It's newer code, though, so there's less chance of eccentricity, FWIW.) Isolation does have benefits - with less chaos to keep up with and less bus factor, the code is changing less and the maintainers have more mental bandwidth to post on mailinglists :), so you have more opportunity to get a good understanding of what's going on.

I don't have the same low-level experience you do, but I do share the same interest in wanting to understand "what's really going on" - and I incidentally want to make a Linux system monitor tool that vacuums up as much information that the kernel is willing to make available to it. `htop` and friends surface a very caricatured picture from maybe 1-10% of the data the kernel has to offer at any given moment.