I googled "opus packet loss" and found https://www.asterisk.org/asterisk-opus-packet-loss-fec/, which says "an Opus encoder can embed redundant data about the preceding packet in-band in the current packet". That method would cause 1 packet of delay. Are you using something different, or avoiding the delay through another method like tiny packets?