My impression is that Wayland planned to support X as a compatibility option. The difference is that X isn't the low level mediator of all input/graphics by default. As I understand.
I can imagine this being a real issue if it happens at the toolkit level; for example, GTK dropping X11 support. But until Wayland proves itself in the field as a real and superior competitor to X, it's absurd to suggest that even a significant proportion of distros would abandon X altogether. And as long as any distro of note (Linux or otherwise) still ships with X, it seems really unlikely that the major toolkits would indulge in such madness.
I wouldn't worry about it for another decade or so.