Answering the updated question regarding the security of running what is described in the post, but using Xephyr and not the desktop's existing X11 session.
It is as secure as the individual components, that is whether there are security vulnerabilities in LXD, and in Xephyr. There are currently no pending security vulnerabilities to fix in either (as far as I know).
So including Xephyr this is a reasonable way to sandbox a browser? To the same level of reasonable as browsing in any virtual machine is.
Is this sort of thing possible with Wayland? If so does Wayland already enforce the necessary process isolation or does something like Xephr for Wayland need to be developed first?
The X11 protocol has been around several decades. The Wayland protocol has been around several years. There are more tools available for X11 and it's used extensively in all distributions. Even in those that are based on Wayland, those have XWayland as well meaning that X11 works there as well.
Gradually, the Linux desktop will be moving towards Wayland and it's a good thing that this happens.
Xephyr is the appropriate tool for what we do (it's a display server for the X11 protocol, can reuse the acceleration of the desktop). There are equivalent tools for Wayland, it's just not required yet. It is possible though to do these things with Wayland as well and probably there is already a tool that I am not aware of.
There is already process isolation with the containers. The lingering issue is with the graphical output on either X11 or Wayland. That needs some extra care. With X11, choice is Xephyr. With Wayland, there should be something equivalent and is probably simpler.
In terms of security of Xephyr, there is an issue. It is a tool that is not used very much and may have some unreported security vulnerabilities. But the same goes with qemu, the hardware emulator. qemu is big and has too much functionality which makes it likely to have yet unreported security vulnerabilities. Have a look at https://cloudplatform.googleblog.com/2017/01/7-ways-we-harde... which specifically mentions the risk in points 2 and 3.
Nevertheless, it should be very important to also implement an option of using Xephyr as part of the application isolation efforts.
This is a convenience tool, that uses your existing X11 session. You would not use it when testing malicious programs because they would be able to attack the X11 session (but not your host's filesystem).
If you want to test programs that might be malicious, then you would set up a separate X11 server like Xephyr and get the output to get directed over there. In that way, both the filesystem and X11 session would be separate from those of your host.
I mention the use of Xephyr in the Conclusions of the post. There is some new functionality in LXD that is being released soon that will make it very easy to use Xephyr as well.
It is as secure as the individual components, that is whether there are security vulnerabilities in LXD, and in Xephyr. There are currently no pending security vulnerabilities to fix in either (as far as I know).
Of course, the same goes with VirtualBox. It is as secure as there are no pending known security vulnerabilities (https://www.techrepublic.com/article/10-new-vm-escape-vulner...).