I work at a 100% remote company, and we do this as well (before covid). We're a smaller company, so I'd be curious to hear how it scales, but at least for us I've found it to work pretty well.
I think it scales because, at a larger company, everyone isn't realistically going to get together anyway. So you get together in groups of various sizes. (And there are usually opportunities to get together with an even larger cross-section at things like user conferences.)