I didn't do arbitrage but running bots for statistical data via API. I think latency is not even the big problem, although of course you have to account for it when doing arbitrage. The latency can be quite high even during calm periods (also don't forget that exchanges can be on different sides of the globe, that already adds quite a bit of latency).
During times of high volumes as in the period of the last rally, the websites of many crypto exchanges break easily and even the API are not that stable.
You can also see that on reddit. Every time the price spiked, you got people complaining about exchanges breaking. There's still a long way to go for crypto exchanges.