[destructionator]

> The fact that any Xorg client can become a key logger without any user input or authentication is a pretty big security hole imo.

This "hole" doesn't exist. For an X client to capture input, it must be authenticated by either the unix user permission or by an access control list (where the default is to deny). Individual clients can also be marked untrusted which sandboxes them to some extent (though not as much as using a separate X server of course).

I'll grant that in practice, most the time these restrictions are very lax... in part because they can break some applications. But at the same time, in practice, it doesn't seem to matter that much since either you're running things you trust anyway or if a malicious application has access to your X connection they also have access to all your other files so you're in trouble anyway.