Unfortunately, it's pretty slow on macOS, as well. I love Magit, but there are times when I feel like it's hindering me more than helping due to the performance.
That, and I often have multiple projects open at once. There's a note in the Magit manual about this and how it affects performance, but unfortunately, the workaround means that buffers don't get refreshed and you might see stale information unless you perform a manual refresh.
I only work on Emacs in small projects though... so maybe that's why it's fast for me?!