[flohofwoe]

IME, working with an immediate mode UI framework automatically gets rid of most such "architecture astronaut" problems.

But I found that it's almost impossible to describe to someone used to event-/callback-driven UIs why exactly that is. You really need to try it yourself on a non-trivial UI to "get it".

[Const-me]

I used and implemented immediate mode GUIs, and I think it’s only good for simple stuff.

Accessibility is hard. The OS needs to know visual tree to be able to conclude things like “this rectangle is a clickable button with Hello World text”.

Complex layouts are hard. Desktop software have been doing fluid layouts decades before the term was coined for web apps, because different screen resolutions, and because some windows are user-resizable. Layout can be expensive (one reason is measuring pixel size of text), you want to reuse the data between frames.

Animations are hard. Users expect them because smartphones introduced them back in 2007: https://streamable.com/okvhl Note the kinetic scrolling, and soft “bumps” when trying to scroll to the end.

Drag and drop is very hard.

[flohofwoe]

Apart from accessibility, isn't this mostly caused by the "immediate mode API" versus "immediate mode implementation" confusion I described in another sub-thread?

There's nothing in the idea that forbids immediate mode UI frameworks from keeping any amount of internal state between frames to keep track of changes over time (like animations or drag'n'drop actions), the difference to traditional UI frameworks is just that this persistent state tree is hidden from the outside world.

Layout problems can be solved by running several passes over the UI description before rendering happens.

For accessibility, the ball is mainly in the court of the operating system and browser vendors. There need to be accessibility APIs which let user interface frameworks connect to screen readers and other accessibility features (this is not a problem limited to immediate mode UIs, but custom UIs in general).

[Const-me]

> There's nothing in the idea that forbids immediate mode UI frameworks from keeping any amount of internal state between frames to keep track of changes over time

When you call ImGui::Button("Hello World") it has no way of telling if it’s the same button as on the previous frame, or a new one. You’re simply not providing that data to the framework.

It’s often good enough for rendering, they don’t really care if it’s an old or new button as long as the GPU renders identical pixels. When you need state however (animations certainly do), the approach is no longer useful.

A framework might use heuristics like match text of the button or relative order of things, but none of them is reliable enough. Buttons may change text, they may be dynamically shown or hidden, things may be reordered in runtime.

> Layout problems can be solved by running several passes over the UI description before rendering happens.

You’re correct that it’s technically solvable. It’s just becomes computationally expensive for complex GUIs. You can’t reliably attach state to visual elements, gonna have to re-measure them every frame.

> There need to be accessibility APIs which let user interface frameworks connect to screen readers

Windows has it for decades now. For Win32 everything is automatic and implemented by the OS. Good custom-painted GUI frameworks like WPF or UWP handle WM_GETOBJECT message and return IAccessible COM interface. That COM interface implements reflection of the complete GUI, exposing an objects hierarchy. I’m not saying it’s impossible to do with immediate rendering, just very hard without explicit visual tree that persists through the lifetime of the window.

[flohofwoe]

> When you call ImGui::Button("Hello World") it has no way of telling if it’s the same button as on the previous frame

ImGui does have stable widget ids, otherwise many features which require keeping state across frames wouldn't work. In case of your button example, the widget id is hashed from the label string "Hello World":

https://github.com/ocornut/imgui/blob/61b19489f1ba35934d9114...

This is just the default behaviour, there's also a convention to define the id separately from the visible label string behind a '##':

https://github.com/ocornut/imgui/blob/61b19489f1ba35934d9114...

Of course other immediate mode UIs might have a different way to define widget ids.

[shrimpx]

> When you call ImGui::Button("Hello World") it has no way of telling if it’s the same button as on the previous frame, or a new one.

Can you solve this by adding stable IDs to the API, so you'd call `ImGui::Button("my-button-id", button_text)`?

[Const-me]

To consistently assign these IDs you gonna have a tree on your side of the API. You’ll then have two visual trees, one on your side on the API, another one on the framework’s side of the API. And then you gonna be debugging things.

Who and when should destroy backend visual nodes? If you won’t use an ID in some frame, does it indicate the backend should drop the related state? What if an animation is still playing on the element in question? What if the missing control re-appears later i.e. was only hidden temporarily? What should backend do if you reuse same ID for different things?

One can implement simple stuff using any approach, immediate GUI is often the simplest of them. It’s when use cases are complicated you need a full-blown retrained mode OO framework. Win32, HTML DOM, MS XAML, QT, UIKit are all implemented that way, that’s not a coincidence.

[enqk]

these APIs are UI Automation on windows and NSAccessibility on macos

[Chris_Newton]

When building more complicated UIs, an architectural pattern I have often found useful is to have two distinct levels of state/data behind the rendering code, so my hierarchy looks something like this:

1. Application data

2. Presentation data

3. Presentation rendering

The first is the single source of truth for your application state, what we often call a “model” or “store”. It’s where you represent the data from your problem domain, which is usually also the data that needs to be persistent.

The second is where you collect any additional data needed for a particular way of presenting some or all of your application data. This can come from the current state of the view (list sort order, current page for a paginated table, position and zoom level over a map, etc.) or the underlying application data (building a sorted version of a list, laying out a diagram, etc.) or any combination of the two (finding the top 10 best sellers from the application data, according to the user’s current sort order set in the UI). This is often a relatively simple part of the system, but there is no reason it has to be: it could just as well include setting up a complicated scene for rendering, or co-ordinating a complicated animation.

The final part is the rendering code, which is a translation from the application and presentation data into whatever presentation is required. There isn’t any complicated logic here, and usually no state is being maintained at this level either. The data required for rendering all comes ready-prepared from the lower layers. Any interactions that should change the view or application state are immediately passed down to the lower layers to be handled.

The important idea is that everything you would do to keep things organised around the application data also applies to the presentation data. Each can expose its current state relatively directly for reading by any part of the system that needs to know. Each can require changes of state to be made through a defined interface, which might be some sort of command/action handler pattern to keep the design open and flexible. Each can be observable, so other parts of the system can react to changes in its state.

It just happens that now, instead of a single cycle where application data gets rendered and changes from the UI get passed back to the application data layer, we have two cycles. One goes from application data through presentation data to rendering, with changes going back to the application data layer. The other just goes from the presentation data to the rendering and back.

I have found this kind of architecture “plays nicely” with almost any other requirements. For example, data sync with a server if our GUI is the front end of a web app can easily be handled in a separate module elsewhere in the system. It can observe the application data to know when to upload changes. It can update the application data according to any downloaded information via the usual interface for changing the state.

I have also found this kind of architecture to be very flexible and to fit with almost any choice of libraries for things like state modelling, server comms, rendering the actual view, etc. Or, if your requirements are either too simple to justify bringing in those dependencies or too complicated for a ready-made library to do what you need, you have a systematic overall structure in place to implement whatever you need directly, using all the same software design and scaling techniques you normally would.

[Const-me]

What you talking about is very similar to MVVM as implemented in various MS XAML frameworks: https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93vie... IMO that’s very good approach to the problem.

> laying out a diagram

I think layout belongs to the views (you call them presentation rendering).

View models provide data to be visualized, but it’s rarely a good idea to compute pixel positions, or handle low-level input there. These things are coupled with views too much.

Similar to animations. Unless the app being developed is a video editor or similar where animation is the core functionality, animations don’t affect models nor view models, they merely prettify GUI state transitions or provide progress indication. No need to wire them all the way into view models. Some frameworks even run them on a separate thread called “compositor thread” so they play fine even if the main thread is blocked or starved for CPU time.

[Chris_Newton]

What you talking about is very similar to MVVM

At first when I adopted this architectural pattern, it was a development of classic MVC so already used the terms “model” and “view”. I actually did adopt the term “view-model” for what I’m referring to here as the presentation data layer. After all, it was the connection between the view and model, so what else should it be called? :-)

However, while there certainly are similarities between the architecture I described above and MVVM — the VM can collect data from the Model and reformat it for the View, and the VM might manage some aspects of the view state — they also differ in some important ways.

Most significantly, MVVM has a linear V–VM–M relationship between the components. The V and M never communicate directly in either direction.

In the architecture I described, the rendering code might depend on both state layers, accessing both application state directly and derived or view-specific data from the presentation state. This avoids any need for boilerplate in the middle layer if nothing special needs to be done with the application data before it’s used for rendering.

Likewise, in response to events triggered from something in the rendered output, messages might be sent to the application and/or presentation state layers so they could update accordingly. The distinction is always clear: any event that can affect application data gets sent there, while anything that affects view state goes to the presentation data layer. In practice, it’s unusual for a single UI event to affect both, so usually only one message is needed. (There is an implicit assumption being made here that the starting point for handling events triggered by user interactions is defined as part of the rendering, which is not necessarily always the case, but let’s gloss over those details for now or this comment will get insanely long.)

In MVVM, there may also be direct two-way data-binding between the V and VM. The architecture I described never assumes that kind of 1:1 relationship between something in the rendered output and the underlying data.

I think layout belongs to the views (you call them presentation rendering). View models provide data to be visualized, but it’s rarely a good idea to compute pixel positions, or handle low-level input there. These things are coupled with views too much.

This is another place where I think the responsibilities in the architecture I described are allocated differently. I want any non-trivial calculations to be in the presentation data layer, and I want the presentation rendering layer to be as dumb as possible. This separates data processing from I/O, which is something I favour as a general principle for organising programs.

In this case, it has the specific advantage that you might have very different strategies for testing them. The presentation data layer is pure computation and can be easily unit tested. The I/O is all about rendering and, in some cases, simulating actions on the rendered output that trigger some sort of message to be sent, so for this you might want some sort of snapshot-based testing or simulated UI environment.

You might also want to incorporate totally different third party libraries or depend on different platform APIs to implement each step. For example, switching out one UI rendering library for another or updating to integrate with some new-style platform API probably shouldn’t affect any of the data (including presentation data like diagram layouts or animation logic) unless we’re specifically shifting responsibility for something like handling animations to a dependency rather than our own code or vice versa.

Some frameworks even run them on a separate thread called “compositor thread” so they play fine even if the main thread is blocked or starved for CPU time.

Sure. One of the nice things about the architecture I described is that it’s entirely neutral about these things. If your requirements so dictate, there is nothing to stop you implementing a more elaborate design at any level of the system. For example, that might include running time-consuming parts of the presentation data logic in separate threads, having changes in the presentation data triggered by timers or other system events, delegating some aspects to hardware acceleration where available, or using any sort of sophisticated caching or scheduling implementation for better performance.

Much of this would add unnecessary complexity in a simple CRUD application GUI. If you’re building a complex, real-time visualisation of an incoming data stream or you’re implementing a game engine that needs to draw part of a large game world with many moving parts, presumably you might have a different perspective on how much extra complexity in the design is acceptable if it gets you the performance you need.

[Const-me]

> The V and M never communicate directly in either direction.

There’s nothing wrong in dropping a layer there.

That intermediate VM layer makes sense when the data being presented is not precisely what’s stored in the model. Good place to implement filters or pagination. Good place to validate user input. However, in other cases one doesn’t need any of that, in which case there’s nothing wrong with data binding directly to models.

> the rendering code might depend on both state layers, accessing both application state directly and derived or view-specific data from the presentation state.

In .NET with MVVM that’s easily doable, expose a property on VM that returns the model, and this will work.

> In practice, it’s unusual for a single UI event to affect both

In the apps I work on this happens all the time. User clicks “do something” button, the model needs to start doing that, in the meantime the view needs to disable that button and show “doing something, please wait…” with a progress bar.

For this reason, I normally handle such higher-level events in VMs and call models from there.

> The presentation data layer is pure computation and can be easily unit tested.

I think strongly typed languages handle 90% of what people usually unit test for, with better developer’s UX. If you change something that broke 33 places of other code, a compiler will tell you precisely which 33 places you gonna need to fix.

> switching out one UI rendering library for another or updating to integrate with some new-style platform API probably shouldn’t affect any of the data

I think that only works when switching to something extremely similar. I did when porting iPhone code to iPad, or WPF XAML to UWP XAML, but these were very similar platforms. UI libraries are complicated; replacing them with something else is huge amount of work.

> including presentation data like diagram layouts or animation logic

I don’t believe these pieces are portable in practice. Both coupled with UI libraries too much. For the diagram, half of the 2D rendering libraries are immediate mode other half are retained mode. Animations are totally different as well, CSS animations in HTML don’t have much in common with visual state manager in XAML.

[Chris_Newton]

No doubt there are many different architectures that can work well in this sort of situation. I’m just arguing that, contrary to your earlier comment that I first replied to, an immediate mode UI can be one of them even when the requirements are not so simple.

In the apps I work on this happens all the time. User clicks “do something” button, the model needs to start doing that, in the meantime the view needs to disable that button and show “doing something, please wait…” with a progress bar. For this reason, I normally handle such higher-level events in VMs and call models from there.

I used too strong a word before. As you point out, situations where you’d want to update both the application and presentation data aren’t really unusual. You gave one good example. Another might be controlling a modal UI, like closing a dialog box or moving to the next step of a wizard, when maybe you also want to commit data user has entered in a temporary form to the permanent application state. In these cases, handling a UI event would indeed need to get the relevant information to both of the other layers one way or another. This still easily fits within the architecture pattern I was describing, though.

I think strongly typed languages handle 90% of what people usually unit test for, with better developer’s UX.

I am a big fan of using expressive type systems to prevent defects, particularly as an alternative to huge unit test suites full of boilerplate that still don’t do as good a job.

However, no mainstream type system will verify that for a given list in your application data and a given sorting option chosen in the UI, a function in your presentation data layer has generated the correct top 10 items in order. (No unit test can ever verify that property in general either, of course, but at least we can test some representative example cases.)

I think that only works when switching to something extremely similar. I did when porting iPhone code to iPad, or WPF XAML to UWP XAML, but these were very similar platforms. UI libraries are complicated; replacing them with something else is huge amount of work.

I suppose that depends on what you consider extremely similar. Around 10–15 years ago, when what we call “single page applications” today were starting to gain serious traction, if you wanted an interactive visual representation of your data, you probably used one of the proprietary embedded technologies like Flash, Java or ActiveX. But if you were drawing, say, an org chart, you were still ultimately just putting boxes and lines and text in certain places within the allocated area on the page. Today we have tools like SVG, canvas and WebGL available handle the drawing part, but everything we’d want to display in that drawing might still be the same.

I don’t believe these pieces are portable in practice. Both coupled with UI libraries too much. For the diagram, half of the 2D rendering libraries are immediate mode other half are retained mode. Animations are totally different as well, CSS animations in HTML don’t have much in common with visual state manager in XAML.

I think perhaps we have very different types of animation in mind. From the above, I’m guessing you’re thinking of things like pulsing a button when it’s pressed or a little progress spinner while downloading some data? I’m thinking of things like smoothly moving all of the boxes on that org chart to their new positions, including animating the shapes of the paths between them, when the user does something that moves the chart around. Another example might be animating weather data overlaid over a map graphic through the next few hours when the user presses a “play” button. That is, I’m talking about animations where the data to be shown at each step needs to be determined according to complex requirements, not just playing a simple, predetermined effect that a UI library or platform API might handle for you almost for free anyway.

[badsectoracula]

I've tried it at the past and "get it" but that doesn't mean i "agree with it" - to me immediate mode GUIs are the graphical equivalent to...

    Running = -1
    WHILE Running
      PRINT "1) Add record"
      IF HasRecords THEN
        PRINT "2) Delete record"
        PRINT "3) Edit record"
      END IF
      PRINT "H) Help"
      PRINT "X) Exit"
      INPUT "Choice: ", I$
      IF I$="1" THEN AddRecord
      IF (I$="2") AND HasRecords THEN DeleteRecord
      IF (I$="3") AND HasRecords THEN EditRecord
      IF I$="H" THEN ShowHelp
      IF I$="X" THEN Running = 0
    WEND

...which, sure, it is ridiculously simple (like imguis) but it can very quickly become unwieldy the more you ask from your user interface. There is a reason why UIs moved beyond that.

[flohofwoe]

I think when you look at projects like Tracy, there's no question that immediate mode UIs also work well for non-trivial use cases:

https://github.com/wolfpld/tracy

[badsectoracula]

My point isn't that you can't make them work for non-trivial cases, after all as long as you can draw something on screen you can do pretty much anything. The point is how well that is done.

For example from a quick look at the code it looks like this project needs to draw in a bunch of unrelated to each other libraries just to get some basic GUI functionality you'd find even in toolkits from the 80s provide (e.g. drawing text or using file dialogs).

And check these sources [0] and [1], this is doing at least as much bookkeeping as in a "retained" GUI toolkit - except it also has to do things that such a toolkit would do automatically, like explicitly drawing the widgets [2]. People at the past were complaining how tools like Visual Basic were mixing presentation and logic, yet the worst you could do in these tools is to have the logic in event handlers (...which if you think a bit about it at least that does make a bit of sense), yet here is actual application logic being part of the drawing code in [3] (this is inside a method View::DrawPlayback, scroll a bit upwards to find it).

Now sure, someone might say that this isn't the best example of immediate GUIs... but this is the one that was brought up as a good example. And TBH all it convinced me is that if (for some reason) i had to use ImGui (the library), to spend some time building a retained GUI wrapper around it :-P.

[0] https://github.com/wolfpld/tracy/blob/master/server/TracySou...

[1] https://github.com/wolfpld/tracy/blob/master/server/TracyVie...

[2] https://github.com/wolfpld/tracy/blob/d8c1dae9e120c27801e762...

[3] https://github.com/wolfpld/tracy/blob/d8c1dae9e120c27801e762...

[jcelerier]

> , there's no question that immediate mode UIs also work well

I sincerely hope no one looks at the screenshot on that page and thinks "this is something that works well". This UI screams "I've been made that way because that was the easiest way in the tool I'm built with" (which is bad - good tools should not dictate the form of the resulting product)

[coldtea]

>I sincerely hope no one looks at the screenshot on that page and thinks "this is something that works well". This UI screams "I've been made that way because that was the easiest way in the tool I'm built with"

That's where you'd be wrong. Profilers traditionally have looked like this regardless of the tool/GUI lib/GUI paradigm they'd been made with.

It's the domain need (to show lots of information, graphs, timelines, stack trees, etc, to compare) that asks for this...

[jcelerier]

> Profilers traditionally have looked like this regardless of the tool/GUI lib/GUI paradigm they'd been made with.

the profiler I use most (hotspot) definitely looks much cleaner: https://www.youtube.com/watch?v=6ogEkQ-vKt4

[coldtea]

It's the same thing split in N tabs.

Not a distinction made because of immediate vs retained GUI.

[flohofwoe]

How would a profiling tool UI need to look differently in your opinion?

And why do you think the UI looks that way because of restrictions of the UI framework?

I'm quite sure if the UI would have been written with (for instance) Qt or React, it would look exactly the same, because the UI requirements would be the same (minus the different default style). The question is whether this could have been achieved with less, or "cleaner" code (which I doubt).

[jcelerier]

example with Qt: https://www.youtube.com/watch?v=6ogEkQ-vKt4

[flohofwoe]

And how is that different except that the same information is spread over more screen space and hidden inside tabs? White space and tabs also works in immediate mode UIs.

[kuraja]

Probably a decent chunk of the perceptual issue you're seeing comes from their choice of font rendering library, which is noticeably "the IMGUI font rendering library", and looks amateurish. This is stb_truetype, which is a very primitive library, but is easy to install and incorporate. The other library is freetype+harfbuzz, which produces higher quality output, faster, and with lower memory. The cost of freetype is that installation is absurdly difficult, the maintainer is offensive, and the code is ancient, unreadable, and architecturally a disaster.

I don't think the chosen GUI library causes the other UI issues you perceive. The author probably hasn't put much time into learning visual design, and thus fell into whatever the UI library supported. But to produce a better one, the author would have had to do a lot more learning and thinking, which is independent of the library choice.

[keeganpoppen]

seems pretty information dense to me. tufte would approve. but surely he's no match.

[panic]

Immediate mode is great, but it can be hard to implement nice custom event handling on top of it. For example, say you have a collection of buttons. You want to be able to click on one button (the button highlights), drag onto another button (that button highlights and the first button unhighlights), and release on that second button to trigger its action. This is a real issue I ran into while building an immediate-mode UI... I found it hard to implement an interaction like this without a reified "layout" that I could query as the mouse moved around.

[boterock]

I was doing something like that with Godot [1], the approach I took was setting some global "dragged_item" when the drag started, and then each drop target would check if the dragged_item was compatible and if the mouse was within its bounds. This way each drop target would do the check instead of some drag manager having to check whatever was under the cursor.

[1] https://github.com/0xafbf/aether/tree/master/addons/godot_im...

[amelius]

Yes, but immediate mode causes more CPU load (or draining of batteries) since you have to redraw everything all the time.

[panic]

It's easy to skip drawing frames when no state changes, and for frames where state does change, you can use techniques like rxi's cached software rendering to avoid redrawing the entire screen: https://rxi.github.io/cached_software_rendering.html

[layer8]

That doesn‘t seem too different anymore to the invalidate/paint paradigm of traditional UI frameworks (e.g. win32 GDI).

[amelius]

How does it handle scrolling? This is typically very fast and smooth (performed by bitblt and redrawing only the newly exposed part on every frame), but in immediate mode you'd have to redraw the entire screen on every frame.

[flohofwoe]

I think this whole efficiency thing is a common misconception. Only the public API appears "immediate mode", the internal implementation doesn't need to be, and usually isn't (e.g. it keeps mutating state between frames instead of building everything from scratch again).

The user-side code basically describes what the UI should look like in the current frame, those "instructions" are recorded, and this recording is reasonably cheap.

The UI backend can then figure out how to "diff" the new instruction stream against the current internal state and render this with the least changes to the screen.

However some immediate mode UI systems came to the conclusion that it might actually be cheaper to just render most things from scratch instead of spending lots of processing resources to figure out what needs to be updated.

In conclusion: "Immediate Mode UI" doesn't say anything how the UI is actually rendered or generally how the internals are implemented, it only describes how the public API works.

[amelius]

Perhaps I should give a different example: a listbox filled with 1 million items, with a scrollbar.

If the public API requires you to give a new paint command on every frame (everytime the scrollbar is dragged), then regardless of whether the underlying rendering engine performs each of these paint commands, you still have to run through every item of the list (and so does the diff'ing code), making this a O(N) operation on every frame.

[flohofwoe]

From what I've seen (in Dear ImGui), this is solved by not giving the 1 million items to the API, instead you can query what range of the list is currently visible, and only describe those to the API (in Dear ImGui this is called a ListClipper).

But I guess different UI frameworks have different solution for this. Creating and updating a 1 million item list wouldn't be a cheap operation in a traditional UI system either.

[wdfx]

It's not difficult to implement a 'virtual' list which renders only the items which are visible. It's a little trickier if your list items are not all the same height, but not impossible.

[panic]

Modern renderers typically draw scrollable content into a separate texture (or collection of smaller textures that tile the scrollable region) and use the GPU to scroll rather than bitblting on the CPU. You can use the same technique to render the scrollable part of the UI in an immediate mode system.

[forrestthewoods]

> immediate mode causes more CPU load

Does it? Computers are FAST. Webdev stacks are many things, but it ain’t fast or performant.

Has anyone built a moderately complex UI in a good immediate mode UI and in a good retained UI? I’d be very curious to know the actual results.

[jameshart]

What if you have your ‘immediate mode’ produce not a direct rendering of the GUI, but a virtual object model describing what the structure of the GUI should be at that moment? Then you diff that object model against the actual object model of the GUI and apply the changes....

That’s how react works. It’s effectively an ‘immediate mode’ abstraction above a component based UI.

That avoids the component connecting and wiring problems, and creates the simple determinism that makes immediate mode systems easier to reason about.

[erwincoumans]

Agreed, in particular Dear Imgui is pretty nice, and high performance.