[seanwilson]

How does this manage to scale? With even a modest number of drivers, you'd think the maintenance burden would be unreasonably high in terms of needing to have someone available who understands how a set of drivers work, knows how to test them, and keeps them up-to-date.

[adrian_b]

That is why everybody attempts to push their modules into the kernel source.

Those who cannot do that because their modules are not open source, like NVIDIA, have a lot of work to do for module maintenance.

Nevertheless, there are also open-source modules that cannot be pushed into the kernel because they are for uncommon hardware with few users, and those also require a lot of work for maintenance, so they are frequently abandoned and no longer brought up-to-date, to be compatible with the latest kernels.

[seanwilson]

> That is why everybody attempts to push their modules into the kernel source.

> Those who cannot do that because their modules are not open source, like NVIDIA, have a lot of work to do for module maintenance.

Say someone gets a driver for a WiFi USB stick into the kernel source and leaves it for others to maintain. Wouldn't the maintainers need the hardware device for testing changes and an understanding of the hardware's quirks to keep the driver updated? Obviously the system works but I don't get how it doesn't suffer from software rot, especially when physical hardware is required for robust testing including testing the hardware against different motherboards.

[toast0]

What I've seen is (more or less) the mainline kernel maintainers for whichever kernel have the objective of making sure the drivers compile.

They can't test what they don't have, and I haven't seen evidence than any open source kernel has a testinfg time with large fleet of hardware for tests. There's more or less an incidental distributed test fleet as users have various testing setups, etc; but driver mistakes in obscure hardware tends to get found late.

It's not unusual for drivers to be found to have been broken for years, and then unless the reporter can fix it, or the fix is obvious from the report, the driver gets removed.

Another path towards driver removal is some interfaces get updated in multiple steps: a new interface is added, drivers are updated to the new interface, the old interface and any drivers that haven't been updated are removed. If nobody has stepped up to update the driver before removal, that's an indication of disuse; sometimes they get added back after removal though.

For the record, Windows isn't necessarily better. I had a printer where there was a working driver, but Windows would prefer the non-working driver from Windows Update, and it was impossible to get that fixed.

[Veserv]

You are correct. The answer is that it does suffer from software rot.

[aseipp]

> How does this manage to scale?

"Okay" to "very poorly", depending on your view. And for the record, many times the drivers go untested, frankly. I've absolutely found completely broken drivers in the kernel (as in just obvious lock imbalances where unlock() can't possibly ever get called, null pointer derefs; simple things you can spot in a code review) in the kernel where the break was introduced as some part of a refactoring and the person doing the work made a mistake. Because they also refactored 30 other drivers in the same patch series to match an API change. (The filesystem developers have said many times in patch reviews that certain things are massive chores because they have to go fix 40+ filesystem drivers every time some API break happens.)

Linux actually has tons of hardware regressions because of this whole design choice. When you totally rewrite a piece of code that is shared among multiple components, that can be OK, as long as you preserve the external behavior that the previous interface exposed. An easy way to do that is to establish a contract between the implementation and call sites, which is often reflected as a stable API. The stable API makes some contracts explicit, by construction. But many times API refactorings will happen at the same time, and that in practice typically introduces new behaviors into the downstream code that previously didn't exist before. These new behaviors, when introduced into an existing driver that was not developed for them, tends to cause bugs when not tested fully.

A big reason Linux chooses not to have stable internal APIs is for agility, more or less. But nothing is free and this is the price that is paid as the project grows.

My personal poster child for this stuff is the amdgpu driver. I use a Navi workstation card (WX5500) in my server and whether or not the amdgpu driver functions correctly on updates is a crap shoot. A while back it went completely headless; when I upgraded to 6.5 like 2 weeks ago, and I had to attach a monitor to look at the kernel logs (network config snafu), my dmesg had 7 kernel faults in its log from amdgpu. Seven! For a 3.5 year old card! With no desktop environment! Despite the fact that the card isn't changing, the driver is changing; new hardware support, expanded interfaces, new features, and those cause regressions. Many subsystems get overhauls and behavioral changes every release, so these things are bound to happen. Testing is not uniform; many parts of the kernel are far more well tested than others. Peripherial drivers are another example of easily broken code (Xilinx code upstream is frankly broken garbage half the time.)

The Linux Kernel is a pretty amazing project in many respects but I'm frankly astonished half the time it actually boots to a working desktop successfully.

[anelson]

My company maintains an out-of-tree kernel module for changed block tracking. It requires a full-time Linux driver developer to keep it compiling and working on new kernels.

Our module is GPL FWIW but I doubt it would be accepted into the kernel tree as it is. We need to have a driver developer on staff anyway to support it so it works out okay for us. But if we ever stop maintaining it, it will bitrot quickly and stop working on new kernels.

[rwmj]

Coccinelle semantic patching is pretty cool for making systematic changes across the kernel (eg. adding a parameter to the same function everywhere).

https://coccinelle.gitlabpages.inria.fr/website/