That's what a 16-sample buffer would get you. In reality, there are many devices on the market that can get close to 2.5ms. For example, Line 6 claims 2.8 ms end-to-end, BOSS claims 2.3 ms [2], the NUX B8 is at 2.5ms [3].
Longer buffers allow for jitter in processing further upstream. If you get a buffer every 0.6ms, you need to be able to process it always within 0.6ms.