Have you tried playing a video? Or moving a window around fast? Without a compositor there will be tearing on X. With a compositor, X is just a useless middleman.
There's absolutely no reason to write an X server from scratch to add native compositing support.
You're also massively overstating the complexity of an X server. Writing X servers that are advanced enough to be usable is something a lot of individuals have done. It's nothing like matching a modern browser. Incidentally, if I were to try to do this today, I'd gut a proxy like Xephyr and make it rootless to support older clients, and make the main server itself only support the subset of the X protocol modern clients speak.
> You're also massively overstating the complexity of an X server. Writing X servers that are advanced enough to be usable is something a lot of individuals have done.
If so, why are we observing apparent shortage of developers to maintain the actual Xorg? I can't believe there are no people interested in maintaining it in the whole world. I thought it's so complex it just requires much more expertise and dedication than anybody is ready to invest.
Because Xorg is a huge beast with plenty of legacy, and it's also not a very fun project to hack on because it's largely maintenance of stuff people only care about when it breaks.
E.g. support for a bunch of backends of which most are not worth supporting if you want to implement a server from scratch, but which Xorg has been saddled with, as well as support for a bunch of legacy X functionality that nobody would bother implementing if implementing a server from scratch because most clients never use it.
E.g. a major source of complexity is supporting various X visuals that are irrelevant today when you can decide to only support truecolor natively. Another is support for a bunch of drawing primitives that you might as well just ditch all but the very simplest of, and support for legacy clients via a proxy like Xephyr. That said, most of those drawing primitives are also supported by any 2d canvas style library like Cairo or Skia that does client side 2d rendering, so if you want to implement them for an X server, just picking up one of those libraries would be time far better spent than implementing it yourself.
It'd certainly be a lot of work, and a lot more work to write an X server from scratch that is a complete replacement for e.g. Xorg - you'd need to support a number of X protocol extensions, but most clients backends will fall back on core functionality if extensions are not available (you'd definitely want to support some of them, like Xrender and SHM though, or your server would be largely unusable with modern clients).
At the same time, if someone wants to write a new X server, it'd be silly to do the work of writing their own backends when the could just grab wlroots and build an X server on top of that. "Modern X11" can be reduced to a relatively narrow superset of Wayland functionality if you defer most of the weirder legacy functionality that is rarely but very occasionally used to a proxy like XWayland, Xephyr or XPRA (all of which do, that said, rely on Xorg code) and make people use those for clients that need it.
Honestly, I think this is the most likely/viable future for X on Linux. It'd drastically reduce the amount of code to maintain and share that burden with other projects, and would make it far more viable to keep maintaining X support for those of us who wants (part of) it.
I really don't understand this line of thinking or what you are trying to accomplish. If you are dropping things from the X protocol to the point where it's only XInput2.2, SHM and DRI3, and then farming the rest out to Xephyr, then it's no longer X anymore. That is pretty much entirely what Wayland already is. So you might as well just build it as a Wayland server and call it a day.