Do you have real time priority configured for the Pipewire (and possibly bluetoothd) process? I had annoying stutter issues before setting that up (Gentoo could not do it automatically at the time, but that may have changed).
I haven't look into it yet. I was trying to get a Bluetooth 5.0 adapter first before trying again, but can't find any at the time. There are plenty of bluetooth 5.0 audio transmitter though, which sidesteps the issue completely.