The contraction shouldn't feel weird unless all contractions feel weird. The grammticalized "have" in the modal is almost never pronounced the same way as the verb meaning "possess"; it has become a schwa-vee or often even a simple schwa (rendered as musta) in ordinary (not explicitly emphatic) speech. It has really become more of an affix than a separate word, so maybe it is time for the written rendering of the language to begin to reflect that.
Are you proposing a common spelling of "musta" ("woulda", etc)?
I only find "must've" weird as it's not a very commonly seen contraction. Much like "mustn't". They're definitely used in speech, but people rarely type them out, which is how you end up with people using "of" when they mean "have" or "'ve".
I don't think I've come across anyone using "must" and "not" in a contractional form. I shudder to think what the "of" crowd would type for "mustn't".