> This is obviously not a great solution, and VNC in general feels like ancient technology
It's not obvious to me. This looks like software ageism. :)
There are at least 2 things that could be done to improve performance in the author's setup:
1. Use the --gpu flag which enables dmabuf capture and h264 encoding via vaapi.
2. Connect using a dedicated VNC client like TigerVNC. This reduces latency by roughly 50 ms.
That being said, noVNC has gained hardware decoding of h264 via WebCodecs since the writing of that blog post, so performance will have improved some. It's unlikely that the 50 ms latency has gone down, but it might be worth measuring again since browsers may opt to render video through a dedicated overlay plane that bypasses regular rendering. However, it would be silly not to synchronise frame presentation between overlays.
Well software has gradually improved and VNC hasn't so of course it's "software ageism". In any case if you were seriously not aware of VNC's shortcomings compared to modern alternatives:
1. Hilarious security. Passwords silently truncated to 8 characters, etc.
2. Inefficient graphics. Very laggy on all but the best connections. Just sending an H.264 video stream is better these days.
3. No seamless window mode.
4. I've never needed it tbh but I doubt it supports remote sharing of USB, printing, audio, GPU etc
As the author of wayvnc, I would say that I'm pretty well aware of what VNC can and cannot do. The RFB protocol (VNC) has improved over time like other protocols.
1. Wrong. There are many extensions that improve upon authentication and encryption.
2. This mostly depends on the implementation, and not necessarily the protocol and/or its extensions.
3. This would require deep integration with the window system on both sides, so I doubt that this is even a thing with RFB on Linux. There certainly aren't any wayland protocols to do this.
4. No one has ever asked me to implement any of that with the exception of audio, but there is also an extension for that.
a) connect to multiple vnc instances
b) display them in a single window
c) act as the "window manager" / "desktop environment" - doing this by enumerating menu entries from each instance and creating a composed whole. Enumerating the active applications from each instance and composing them into a single "task bar"/"window switcher" interface.
To add to the long list of caveats: This also breaks assumptions GUI toolkit and applications make about only one of their windows being focused at a time: `gtk_application_get_active_window`, `QApplication::activeWindow`, etc.
I have never liked split keyboards but I might if each keyboard was also a mouse with its own cursor, it seems fun to develop the skill to work that way. Would also be useful at times if the the Thinkpad trackpoint and trackpad could each have their own cursor and I could leave one somewhere. Or maybe a cursor stack like pushd/popd, registers would probably work better; instead of virtual desktops just give me one massive desktop with cursor locations I can jump between, switch to cursor 3 it saves the location of the current cursor and jumps.
Wayland seats are sets of inputs, to be handled together, each seat having its own cursor position, focused element, and so forth. That’s not what you want, I believe.
… unless you do want the left half of your keyboard to type into one window, and the right half of your keyboard into another.
I have a UHK split keyboard with a keycluster module (which has a small trackball) on one the left half and a larger trackball module on the right half.
Even though there is only one cursor it's been a really powerful way to work, being able to control the mouse with either thumb.
Then there are the alternative keybindings that allow moving the mouse using the wasd keys as well.
The original title, "State of multi-player Wayland" is a much better summary of the article. This ain't just about the pointers! In fact, the author is very explicit about that:
> I think there could be a term that’s describing this concept even better – in the title of this post, I picked “multi-player”, to make clear that it’s mostly about multiple people who are collaborating. I’ve also seen “multi-cursor” or “multi-pointer” (which are missing the keyboard component).
In fact, you can also have a different touch input devices per seat!
I'm not aware of actual hardware that would make this feasible, but if you invent a touchscreen which recognises between two people's different fingers, you could have a client which tracks who clicks where without any protocol changes.
I'm building a compositor now, and I've been thinking about multi-seat from the beginning, but I know there are lots of places that assume a single seat, and so this sort of thing won't work for me. It's actually a lot more work than you'd think to support this...
One very nice use case that I haven't seen anyone else talk about is touchscreens.
I have assigned my touchscreen to be in a separate seat. This allows me to interact with apps without moving my cursor or changing my window focus.
For instance: web browser on the left, with a document. Text editor on the right. I am writing some notes with my keyboard, scrolling the document with my touchscreen. The editor stays focused, no back-and-forth with the mouse. It feels very natural this way, instead of having your cursor jump where you touched the screen and change your M/K focus.
Given that a lot of the use case is interacting with two different programs simultaneously, even just supporting it at the compositor level is pretty valuable: then it should "just work" for that use case. You can't really do a whole lot about the fact that most Wayland clients are making assumptions about seats; well, you can, but that's a lot more work...
Multiple keyboards is trivial - unless you meant multiple active cursors. (If you've used a USB keyboard attahced to your laptop - you've already experienced multiple keyboards. Both work.)
It's not obvious if your core statement is "multiple keyboards is trivial".
Assuming that multiple mice obviously means multiple cursors while in the same comment assuming that multiple keyboards obviously does not imply multiple input cursors is in fact pretty confusing to me
Of course you're correct re: laptop and aux kb - I was thinking of "seat" (and multiple seats) being: "Wayland seat refers to one 'seat' at which a user sits and operates the computer."[0]
So a keyboard being attached to a seat, whose input might be associated w one active window, while another kb is associated with a different seat and different active window. Perhaps I'm just overthinking it.
Back when we were working in the office, this was honestly a preferred pair programming setup for us. Two people, and each just had their keyboard and mouse.
Interestingly enough, it seemed that the inputs from multiple mice onto the same cursor is summed up by default. So you could fight cursor movement you disagreed with by moving the mouse in the other direction.
It's not a direct answer, but Nucleus Coop (Windows only) allows playing multiple instances of the same videogame with different savefiles and allow using multiple keyboards and mice (is it mice for the hardware thing?)
It's an incredible hacky piece of software that does work by putting together like 10 oss. Incredible and cool.
I keep thinking of trying to see how far I get re-doing Niri in River.
The one other constraint that I'd really like to see relaxed is about displays, about being able to merge and subdivide screens as I please. Having a huge ultra-wide display really would be so much better if I could divide it up somehow. This is kind of the headline feature of stilch, which is a pretty neat compositor,
https://github.com/wegel/stilch .
There's been all kinds of rough incredibly hard things going on with Niri just trying to figure out how to handle some of the more interesting copy-paste scenarios, and with really weird sequencing issues across basically ephemeral inputs, that have really plagued the project and made life incredibly bad for downstream apps. This is just my opinion, man, but I think the lack of ambition about these ridiculous over the top features eventually starts hampering what turns out to be relatively prosaic asks. Better models for multi-seat, multi-cursor help scenarios like remote-desktop, and video-chat remote control, that at first don't seem needed, but are fundamentals that pay dividends if you shoot for them.
It’s good to see the minimal drag to border behavior coming to all the OSs and DEs these days. I’m a weirdo and use i3 so I relate to this concern and am very comfortable with tiling for lots of applications. However there’s sometimes I wish I could just tell my display manager to treat the single WH wide monitor as two separate displays with dimensions, e.g. W(H-x) and W*(H+x).
Not sure the best level to implement this. There’s probably already solutions for this now that I think about it…
I wish libinput gave the option for pointer inertia, where the pointer glides to a gradual rather than a hard halt after you stop moving it. I’m limited to a trackpad these days and it’s a really nice feature on my Steamdeck that I’d love to have on my Thinkpad.
Come to think of it, Steamdeck is Wayland? How do they implement it?
>I wish libinput gave the option for pointer inertia, where the pointer glides to a gradual rather than a hard halt after you stop moving it
Why? That seems like it would make it a lot harder to move the cursor precisely. (in fact the main implementation of it on linux I know of was a joke in IOCC)
How precisely can you scroll on your phone? I don't think anyone finds it awkward.
I get the impression that you're thinking of a much jankier implementation of "inertia" than that, and to be honest I wouldn't be surprised if that's what we'd get anyway.
An unambitious inertia describes the linux community of the past decade pretty well.
Does this mean the inertia only applies if you remove your finger at the end of the movement, and so non-flicky movements are interpreted normally, with the cursor stopping as soon as your finger stops? That implementation sounds a lot better than how I interpreted the original comment; I thought they meant it would apply to all movements, even when your finger stays down the whole time.
You'd think so but as the person above mentioned, this is how the steam deck/controller does it and it works well. It mostly affects faster movements like flicking the mouse across the screen, leaving smaller movements unaffected.
The opposite applies to my use. Humans are generally great at predicting where moving things will land - we evolved for that. It feels great flicking the cursor around and stopping it at the right point. It’s just like scrolling a list on your phone, I for one get great precision doing that. It’s especially great on small touchpads.
I highly agree. The synaptics driver on an Asus Eee PC I had had this feature and I really liked it. It just makes sense: It is a thing you touch and flick around. Having it immediately stop is like the list you’re scrolling immediately coming to a hard stop once you release the finger.
Semi-related - I think that controllers would be a very nice single input device for many kinds of workloads, especially if you can occasionally just dictate text. It's kind of sad that it's not supported all that well.
one interesting use case is surface management in ways that cant be solved easily by swapping desktops or which is clunky this happens most on single-screen where have to swap apps out that have related actions and want to rearrange or copy paste between or scroll sync and native app so cant scroll sync by extension think rapid fire macro mouse events except not just serial which can run the experience eg browser tab and electrion app that sync scroll or scroll c/p and where macro recorder might come close but not really
You’ve been led astray by the altered title on this submission. The web does not support the concept described in this article (seats) at all. It supports tracking multiple pointers, but it’s all still single-player, in the terminology of this article: you’re still limited to single focus, single typing, single pointer.
I've always thought this was a cool idea. It's obviously been on the minds of Wayland protocol developers since the beginning, otherwise it wouldn't be possible!
I thought it was the other way around, X11 has supported multiple cursor since 2009 and Wayland didn't add support for it since toolkit developers weren't making use of it anyway.
Enabling in X works via:
xinput list # find second mouse id
xinput create-master second
xinput reattach <mouse-id> "second pointer"
Works great for clicking, but keyboard doesn't work in some applications when enabled. Dragging windows might attach them to the wrong cursor and other issues. Playing around with xinput can also crash some apps (libgdk-3 here) or leave you in a state without a keyboard or mouse.
Which desktop are you running where you can't figure out how to take a screenshot? Maybe some of the more tech-savvy folks here at HN can help you figure it out.
Since you edited your post I'll rephrase my question: Which desktop are you running where you can't figure out how to share your screen? Maybe some of the more tech-savvy folks here at HN can help you figure it out.
I have been dreaming about a split keyboard with dual trackballs and two cursors for the past few months. This may be the kick i need to actually get on it.
They didn't call them mouse cursors with Ataris and Commodores because almost nobody had heard of a mouse, but there was more than one joystick port so multi-player was built-in from the beginning.
I hadn't heard of a mouse either but when the trackball appeared, mainly to play Centipedes and Missile Command, I used it as a single-button mouse anyway, not only to point & click but to drag & drop which I had never seen demonstrated.
So I made a little 2-player game, not fancy, each player stayed in their own window.
Of course nobody had ever heard of "Windows" either since MS-DOS wasn't even out yet.
I immediately recognized the avatar of this person. They are the one with the DIY soft drinks which has been posted on HN before: https://blinry.org/diy-soft-drinks/
While op uses systemd/udev's internal property mutation mechanism to configure seats, not that sway (and likely other compositors) allow you to just configure the device-to-seat assignment in your configuration or even at runtime.
I use two mice just to be able to switch from the fatigued hand to the other without shifting the mouse repeatedly. I wish it was easy to have two cursors in KDE.
The more "typical" use for a feature like this is drawing tablets and touch screens. You don't necessarily want a touch/drawing tablet input to move your actual "main" mouse cursor.
That’s unrelated. The typical device will only ever have one seat. From Wayland core, that seat may have a pointer, keyboard and touch; and from tablet-v2 <https://wayland.app/protocols/tablet-v2>, you can add tablets and their tools and such, all still in the one seat.
Interestingly, for me this usecase isn't typical at all. Ever since I switched from X11 to Wayland (on GNOME) I've had the bug that when I'm drawing with my Wacom tablet, there's two mouse cursors and the tablet doesn't influence the main cursor anymore.
I thought this is a bug in the Wayland implementation: I find it utterly unintuitive to work like this. (I just have no idea with whom to file a bug report, which is why I haven't.)
Are you implying that this is intentional behavior?
The typical GTK application out there can't handle multi-touch and can rarely even handle several mouse events (e.g.: mouse-wheel scroll while dragging), so I doubt they have full blow support for multiple seats.
This is actually something I've always wanted, but people told me it's dumb. I want two cursors and two keyboards, with two "focused window" panes. If Mouse A clicks Pane1, keyboard A presses should go into Pane1. I think this would be so fun for multiplayer games on the same computer for example (think two Minecraft instances running).
It's possible with wayland seats, if the compositor supports it. I've used sway this way. Assign M+K A to seat A and M+K B to seat B. I only wish I could change the focused window decoration depending on the seat (say: one blue seat and one yellow seat).
It's not as weird as you might think: laptops may have touchscreens (which can be multi-touch) on top of the "normal" cursor. IIRC, X.org can't really handle such cases cleanly.
Sometimes that's true, but sometimes (imo the superior approach) the touchscreen is independent from the cursor. I think touchscreen Chromebooks are like that, for example.
It depends on the DE and the application to some extent. The touch events are seperate to mouse events and the mouse cursor moving, but AFAIK usually there is an emulation layer for compatibility with applications that don't support touch events which also moves the cursor, and this behaviour can change depending on the application (either it sets some flag or responds to the touch events in some way). Digitizer pens are the same way.
This is cool. I feel like I shouldn’t be the only one who as a kid connected up an extra mouse to their computer with their buddy and was disappointed to still only have one cursor.
The way screen sharing solutions like Teams handle the multi cursor thing is to just keep the main cursor jumping around, which works, but meh.
Having multiple cursors within the same environment theoretically enables such interesting collaborative experiences. Imagine a large surface in which everyone has their own workspace (cluster of windows), but you can seamlessly go between them …
On the note of the two different kinds of seats, a tangent: better multi physical seat support is something I am passionate about, specifically, I want Windows-style "Fast User Switching" on Linux with RDP. Today's Linux RDP support usually acts like screen mirroring; you get a mirror of a physical seat. With GNOME I believe there is also now support to do remote login: you can log in remotely, and even disconnect and reconnect to a remote login session. But what you can't do is continue a physical session remotely or a remote session physically, and the reason why is sorta to do with seats. Sort of.
See, when you start a graphical session on modern systemd-based Linux, a new logind session is created, and it is immutably tied to a seat. So you can't just remove the seat from a running logind session and then make it remote, or add the seat to a running logind session and make it physical. I actually tried patching systemd-logind to do this, but when I proposed it upstream, Lennart Poettering himself actually took the time to explain that this wasn't the intended way to do it: instead, you would actually just create a new logind session and drop the old one entirely, or literally whatever you want.
Which to me sounded perposterous, but then I realized that it really wasn't that ridiculous. As it was, when you boot a desktop using systemd-logind, the desktop itself doesn't typically run much of anything directly in the session context anyway - the compositor and almost all services actually run as user services, as in systemd-land having more than one active desktop session per user isn't really supported. Okay so we can do this!
And so I set out to prototype this with KDE, but there are still a few issues...
- First and foremost, $XDG_SESSION_ID. Believe it gets set by the systemd PAM module somehow? I don't fully recall. Either way, it gets used by various programs to determine the current logind session, but if our graphical session can "roam" between them we need a mechanism that can handle changing over time. I propose we stuff the current logind session ID into a file in $XDG_RUNTIME_DIR instead.
- Secondly, compositors need to be redesigned to handle this. Compositors today can hotplug displays and input devices - that's a good start. Unfortunately, at least kwin can't switch backends at runtime. You can really go one of two ways, and I tried a bit of both and either can work but both are a bit ugly: one is to make a single backend that can handle either case and the other is to allow switching the backend at runtime. Either way, when running remotely the card* DRM nodes are unavailable so you need to switch to renderD* nodes instead. You can also use both, and only use card* for scanout when available, but it seems a bit trickier.
- Finally, I genuinely have no idea how this could or should be handled in non-systemd environments. I have admittedly forgotten how that all works. It would be rather lame to simply require logind for this to work, but sadly it is the path of least resistance at least at first.
And maybe some more, but otherwise the concept does indeed broadly work. I mainly have only tested it in virtual machines with DRM native context, but I can see everything working switching in either direction at least once before things start to get screwed up. Even OpenGL clients and audio playback can move seamlessly! It's just like what Windows accomplished over 20 years ago!
Aside from convenience and being cool, it would also solve a couple problems. No need to worry about the limit of only one session per user: just move your session with you. No need to worry about your physical session being unlocked when remotely connected: when you log in this way your physical seat gets disconnected from the session entirely and the display manager takes it back over.
I actually would really like to at least make some progress on ground work that would be needed to make this happen, although I'm not really that sure that anybody cares. But either way this post's brief discussion about physical vs logical seats activated a few neurons so I couldn't help but dump this out.
I would appreciate this as well, TBH. Window's handling of remote desktop has been well ahead of linux's for a long time, but there's an irritating artificial limit of one active user on most versions of windows. Last I looked, even just headless wayland was still very awkward to use.
It's not obvious to me. This looks like software ageism. :)
There are at least 2 things that could be done to improve performance in the author's setup:
For more details on 2. see second paragraph in the following blog post https://andri.yngvason.is/measuring-latencies.htmlThat being said, noVNC has gained hardware decoding of h264 via WebCodecs since the writing of that blog post, so performance will have improved some. It's unlikely that the 50 ms latency has gone down, but it might be worth measuring again since browsers may opt to render video through a dedicated overlay plane that bypasses regular rendering. However, it would be silly not to synchronise frame presentation between overlays.