[moyix]

Once a rootkit is installed, it can completely bypass system call monitors in all sorts of ways – communicating with a kernel component via a shared user/kernel memory page, or adding a new device and communicating using custom ioctls, or "backdooring" an existing system call when some userland parameter is set to a magic value, or ...

I am not at all confident that one could find such malware without human intervention.

[nickpsecurity]

Not if it's hypervisor-based monitoring with IO mediation. This is still a weak defence. Stronger model is kernel integrity + syscall restriction + MAC or capability protection for usage details.