Bluetooth audio devices that I use tend to change the protocol as soon as it switches to headset mode (with microphone enabled), which works terribly for music. I imagine the protocol used when the microphone is enabled might have completely different latency characteristics than the one used purely for audio, so a chirp might be measuring completely different thing
You could use a different device in the swarm for measurement, but yeah it seems pretty quickly complicated! I have no idea as well how stable the latency is