Yes, physics (and most disciplines on arXiv) is relatively special. Although for some reason physicists still mostly submit there work to the journals in addition to placing them on arXiv...
(That said, typesetting and writing are more and more often offloaded to academics, but there's also things like e.g. making HTML versions and system-readable versions [1] available, which is a cumbersome manual process.)
It depends on the field, but you're forgetting that arXiv does not do peer-review. Just because publishing is broken and yes peer-review has issues doesn't mean peer-review isn't needed.
On the one hand that's an important point, but on the other hand, the formal process of peer review as currently arranged by journals has many flaws, and the reason arXiv is so popular in fields like physics is that they want their results "out there" for others to use as soon as possible - i.e. people start to use it before it's been peer reviewed. In fact, the comments one receives after publishing it in arXiv often make its way into the submitted manuscript.
So yes, peer review is useful, but it also often happens after publication at arXiv, outside of the journal submission.
(That said, typesetting and writing are more and more often offloaded to academics, but there's also things like e.g. making HTML versions and system-readable versions [1] available, which is a cumbersome manual process.)
[1] https://en.wikipedia.org/wiki/Protocol_for_Metadata_Harvesti...