If the "You may not use the material for commercial purposes" clause applies to the wire protocol itself, does MongoDB itself get an exception to that for Atlas/MongoDB Enterprise?
The copyright owner/holder can do whatever they want. That's essentially how you're able to have things like Gitlab Enterprise or CockroachDB Enterprise use their open source base in an extended not-open-source premium product.
It does require you manage outside contributions to the open source part carefully and get approval for those contributions to land in the not-open-source part.
The license is for the published version available publicly only. There's no reason why it can't be separately licensed to an individual/entity to grant commercial use on a case by case basis. This is no different from negotiating a contract - the license defines the terms under which you are allowed to use the provided software.
It does require you manage outside contributions to the open source part carefully and get approval for those contributions to land in the not-open-source part.