[MrJohz]

One thing that I've learned over time is that if I can't understand why people are taking some particularly obtuse or circuitous route, then that's probably because I don't yet understand what they're doing properly. I never really understood why people would use terminal-based text editors until I tried it out for myself, and while I still have my doubts, I understand the tradeoffs much better now. If frontend development feels as insane to you as the analogy you're describing, and yet it's one of the largest sectors of the software industry, then it seems likely that you're missing something, rather than everyone else going mad.

As you say, it has always been possible to build things where the browser-based client is separated from the backend via an explicit API. Consider an application that shows you a list of flights coming into an airport. You want it to automatically update whenever new flights come in, so you need some polling to fetch that data, and some Javascript to clone elements, fill in the details of the new flights, and insert them into the DOM. But you also want sort functionality, so you need to be able to rearrange all the elements as well. And you want to filter, so you need to be able to store all the flight information internally as state, even if those flights aren't currently being shown. And your can click on each flight to expand it and view more details about that flight, so while you're rearranging and rewriting all your flights, you also need to be keeping track of which ones are currently open or not.

This was the exact application where I first realised that all the jQuery-based DOM manipulation techniques I'd learned were not going to be enough, and that there was a lot of value in having a tool that manages keeping the DOM in sync with the application state, allowing you to focus on keeping track of that state.

There are other approaches, but in my experience, they mostly come down to building your own framework, one way or another. They don't have to be as complicated as React, or as heavyweight as Angular, but you need some system in place to help you sync your state to the DOM.

I'm not establishing here that all web development needs to take this approach, just that there are a lot of situations where it is very useful. I've ended up building and maintaining variants in the above filter/sort/details/live updates system three or four times in different contexts and with different parameters, to say nothing of other, similarly complex web apps where the system works have been unmanageable without some sort of frontend framework.

Once this starts, though, you end up with frontend developers who specialise in this kind of development, and I suspect this results in a kind of siloisation, splitting web development into frontend teams and backend teams, who can each specialise further. Backend developers no longer need to worry about that occasional time where they need to tweak some CSS or JS, and frontend developers don't need to understand databases. This in turn develops into frontend developers using their tools in a wider variety of situations, because it turns out they make a lot of web development a lot easier, not just complex state management. It's a bit like how people use git to track their dotfiles. Even though git is designed to coordinate between multiple people syncing complex branches and forks together, it turns out it also works well as a simple backup and rollback mechanism.

The result is that tools like React are probably somewhat overused, but that allows developers to specialise into broad fields like "frontend" or "backend", while also remaining very general within those fields (because most work that is frontend will use similar sorts of tools and layouts, just like how MVC will apply to most backend development, regardless of language or framework). And this also isn't universal - there are still plenty of smaller, more boutique design operations that will build websites in the more traditional way.

[caspper69]

I hope you understand that a fair bit of my second post was tongue-in-cheek.

I appreciate the concrete examples, and yes, that makes perfect sense when you start getting into data-heavy frontend applications. I do remember jQuery being useful for AJAX and for DOM selection primarily, so I get it.

And of course I knew I was missing something, that's why I asked lol, the question was serious and I knew HN wouldn't let me down.

I'm still a bit mystified about the necessity of a shadow DOM, it just seems counterintuitive to me, but perhaps it's because I'm not familiar with browsers' DOM handling conventions. Perhaps me gaining some insight there would make the purpose of a shadow DOM more clear.

Honestly, I really did figure that it had something to do with the event model (not the DOM or shadow DOM). Something about providing a more consistent apparatus for wiring and handling events.

I still think there's a lot of unnecessary spinning of the wheels on the front end, but that gets beat to death around here, so I'll let it be.

[MrJohz]

By shadow DOM do you mean the virtual DOM? The shadow DOM is also a thing, but it's largely unrelated to web frameworks.

The idea behind the virtual DOM is that it's usually easier to render everything in a functional sort of way. You treat rendering like a function that takes state as an argument, and returns your desired UI. Every time an event handler fires and changes some state somewhere, you rerun all the functions with the new state and that produces a (potentially different) UI.

For various reasons, it's not practical for these functions to actually return a new DOM every time the state changes, so instead they return the VDOM, which is a description of what the DOM should look like, and React diffs the real DOM against the VDOM and updates it if there are any changes. This way, if you've got an input that the user is typing in, it doesn't get replaced completely every time the component updates, but React might update properties on it as they change.

Originally the React team talked about how the VDOM made React faster, and this claim got stuck in everyone's heads, but it's not really true. The VDOM allows React to separate the rendering phase (i.e. when all the component render functions are called) and the DOM update phase, which can have some performance improvements against more naive approaches, but if you could optimally update the DOM directly whenever state changes, then this would still always be quicker. (And a lot of frameworks are moving towards this model, and away from the more React-like VDOM approach.) The main benefit of the VDOM is that it simplifies the programming model: it allows the "UI as a function of state" concept, and makes it easier to reason about what's happening while a render function is being executed.

A lot of more modern frameworks are moving away from VDOM-based approaches, although React is staying very much where it is. Like I said, there are faster approaches, but these arguably come with tradeoffs about complexity. Personally, I tend to avoid React because of the VDOM approach, in large part because it's a layer of abstraction that I don't really need.

[caspper69]

Yes, virtual DOM. My mistake. Not even sure why I used the term shadow DOM.

That makes sense. I understand the rationale now.

I appreciate the detailed and thoughtful response.

[MrJohz]

It's a very common mistake! No worries, I always enjoy getting to explain these details to people, I find them really interesting.

[gbuk2013]

I'm a BE dev that has do to some FE work now and again, but IIRC the shadow DOM thing is about performance: it is in theory more performant to modify the shadow tree and then send it to the browser for rendering in one go than to manipulate individual elements and have the updates render immediately.

Back in the day I had to use an escape hatch in Vue 2 (which does not have shadow DOM) to work with the DOM directly to render a fast changing component (a structured log browser window with live logs flowing in rapidly). No idea if React would have worked better though. :)