The latency of iTerm has traditionally been much worse than Terminal.app. It got a bit better with GPU rendering, but it still trails behind Terminal.app quite a bit.
https://danluu.com/term-latency/
https://www.lkhrs.com/blog/2022/07/terminal-latency/
I tried iTerm for a while because Terminal.app does not support true color. But the difference in latency is definitely noticeable to me.