[l1k]

Thunderbolt devices appear in the OS as a PCIe switch, so you need two additional bus numbers (one for the Switch Upstream Port and one for the Switch Downstream Port). If the device is hotplugged to a port which has run out of bus numbers, you'll get this error message.

Mika Westerberg is constantly fine-tuning the allocation of PCI resources in the Linux kernel to avoid such scenarios. Some recent patches:

https://lore.kernel.org/linux-pci/20220905080232.36087-1-mik...

https://lore.kernel.org/linux-pci/20221130112221.66612-1-mik...

On macOS, it's possible to pause the PCI bus, reallocate resources and unpause the bus:

https://developer.apple.com/library/archive/documentation/Ha... (search for "Supporting PCIe Pause")

We don't have that on Linux unfortunately, so we depend on getting the initial resource allocation right.

Sergei Miroshnichenko has worked on such a reallocation feature for Linux but it hasn't been accepted into mainline yet and he hasn't posted a new version of his patches for almost two years, so the effort seems stalled:

https://lore.kernel.org/linux-pci/20201218174011.340514-1-s....

[nneonneo]

It sounds like the pause/unpause might be the way to fix this properly, since trying to be heuristically smarter sounds like a recipe for never-ending corner case bugs like the OP’s issue.

The patch for pausing and unpausing seems quite reasonable, except that it does require driver support (unsurprising - you’re literally reallocating the resources used by the driver!). I suppose if you had at least a few movable devices then you should be ok in the event of a hotplug event, so you’d have to hope that enough drivers bother to support the feature.

I wonder what is necessary to get people to care about the patch enough to fix it up and mainline it? I suppose the problem it fixes is still niche enough that not so many people are clamoring for the fix.

[l1k]

The PCI resource allocation code is fairly intricate and everyone is scared that changing it may cause regressions. Sergei's patch set is quite intrusive and it would be necessary to somehow break it up into smaller pieces that are slowly fed into mainline over several release cycles, always watching out for regression reports. So, the problem is known, but the engineers working on PCI code in the kernel are given higher priority stuff to work on by their employers, hence the issue hasn't gotten the attention it deserves.

Actually I forgot to mention there's another solution: A PCIe feature called Flattening Portal Bridge (PCIe Base Spec r6.0 section 6.26). That was introduced with PCIe 5.0. It's more likely that FPB support is added in mainline than the pause/unpause feature. It's supported by recent Thunderbolt chips and it's an official feature of the PCIe standard, so companies will prefer dedicating resources to it rather than some non-standard approach.

[sylware]

In the dynamic use cases, the PCIe specs is kind of shabby on the addressing space: it is theorically fixed by FPB.

I guess this is sorry for those niche hardware use cases.

Isn't FPB into PCIe 4.0? (I am not a SIG member, cannot read the specs).

[sylware]

I meant, I know about PCIe addressing (from the web, linux code, and a book I read years ago), but I cannot read the modern specific FPB specs.

[bombcar]

Would a workaround be that whenever the kernel detects this happening (and it did, it dmesg printed it) that it somehow increases an internal counter so on next reboot there will be more resources?

This would require the kernel being able to either update its own command line somehow, or having some permanent storage somewhere it could store it.

Or this could all be done by systemd - detect that message, increase the resource, next reboot will fix it.

[smashed]

Kernel state does not survive reboots afaik.

That would need help from userland, which is not involved in the early boot process.

You could I guess change kernel init parameters and save that in your boot loader, but that is very hackish.

[planede]

Maybe it can be introduced gradually, making the reallocation an optional feature that a driver might support. Then drivers can independently implement the resource reallocation feature.

Mainline drivers can move gradually. If they want to be nice for out-of-tree drivers then they can describe a timeline for deprecating and removing the support for non-reallocating drivers.

[AnIdiotOnTheNet]

What is the point in having all of the drivers be open sourced and mainlined if we're not willing to fix them to support this?

[lostdog]

> What is the point in having all of the drivers be open sourced and mainlined if we're not willing to fix them to support this?

With open source and mainlined drivers, it's very difficult to change all the drivers and ensure they work.

Without open source and mainlined drivers, it becomes impossible.

[bee_rider]

Possibly it is hard, tedious, or the people able to fix it don’t think it is worth the effort.

Open source projects rely on volunteers mostly so it isn’t like there’s some outside force to appeal to. If nobody volunteers a solution, then it isn’t important enough to solve. The point is that, if it were important enough to fix, anybody with the requisite skills could do so.

[mrb]

The key takeaway, especially for the author who wants things to "just work", is that he should have used a USB external disk, instead of a Thunderbolt external disk. OSes tend to have more issues with Thunderbolt disks (such as this issue you explained in details) than with USB disks, because the latter is more common, so more corner case bugs have been eliminated.

[bhewes]

Yeah, first thing I thought was oh that drive looks esoteric.

[fabiensanglard]

Any book or documentation you recommend to read to someone interested in getting involved?