Mailgun: Public Security Disclosure

[throwaway2346mg]

TLDR: Inbound email routes don't have SPF/DKIM/DMARC protection, meaning any inbound email sent to a webhook can be trivially spoofed / phished. Mailgun describe inbound routes as:

"complex incoming messages are simplified and parsed into all of the data you need with Inbound Routes." [1]

Mailgun refers to DKIM/SPF/DMARC with:

"Mailgun leads the charge in email authentication by requiring DKIM and SPF records by default before you can begin sending." [2]

However, nowhere does it say that Mailgun does not provide SPF or DKIM validation checks on inbound emails. Confusingly, these headers appear only for a subset of inbound emails.

Attack scenario:

1) Pick any domain using mailgun for inbound email processing (trivially found via MX records).

2) Spoof an email from any address you want to impersonate (eg. [e-mail]).

If that address is handled via an inbound route to a webhook (eg. CRM system / other), then it will appear as from the spoofed sender without any DKIM/SPF flags. This makes phishing via Inbound Routes trivial.

Requested fix:

Include SPF (X-Mailgun-Spf), DKIM (X-Mailgun-Dkim-Check-Result), and Spam Assassin headers (eg. DMARC_QUAR / DMARC_REJECT / DMARC_NONE / DMARC_MISSING) headers for ALL inbound routes to a webhook. The application can then decide what to do.

Mailgun Response:

"Our security team has confirmed that our routes act as an open relay and that this is not considered a security vulnerability with Mailgun."

Nowhere in the Mailgun documentation / sales pages are inbound routes described like this, and in fact, they pretend to have protection as headers are sometimes present!

Note A: Using throwaway to not give information away for our own exposed systems.

Note B: The spoofed email must not trigger above a certain spam assassin threshold (this is fairly trivial to do and openly testable by any attacker), otherwise it might get blocked by higher level Mailgun spam handling.

[1] - https://www.mailgun.com/products/send/inbound-routing/

[2] - https://www.mailgun.com/blog/deliverability/implement-dmarc/...

[sandstrom]

Just did a quick reading of their docs.

They seem to provide raw headers in the JSON payload, and if they don't, they also have another mechanism where you can do the MIME parsing yourself.

https://documentation.mailgun.com/en/latest/user_manual.html...

Either of those two should allow you to do the SPD/DKIM/DMARC validation on your own.

I wouldn't call this a security vulnerability.

At best, is a lack of functionality on their part, where a (reasonable) wish would be that they as a email receiver would perform the SPF/DKIM/DMARC validation and provide the results of that check as headers or other metadata.

[elif]

a lack of security functionality by default is categorically a vulnerability. you are mincing words.

having a redis exposed to the internet is a vulnerability, even though theoretically you could use lua scripting yourself to do some kind of authentication for every request. same stretch of logic applies there...

[junon]

Ehhhh not convinced that's a fair comparison.

In Chalk (node.js library for terminal coloring via escape codes) we had a "vulnerability" reported to us that inputs to our library that had malicious escapes were passed through and thus our library allowed RCE.

Never in the docs did we claim we sanitized inputs, nor should we - the net result is that the library would be slower and bulkier for the 99% case and protect only a fraction of the remainder from any sort of real attack - and those users should probably be doing their own sanitization anyway.

Is that really our fault? No, not really. I don't think that if mailgun doesn't support it, should it be considered a vulnerability. GP had it right - could be a cool feature but this is not some oversight unless they market that as being a security measure.

[throwaway2346mg]

Agreed. I think the point here is that what % of Mailgun users will be doing this additional processing? I suspect it's basically 0%.

Why? It's not outlined in their specs, their sales copy implies they are handling it, and sensible headers are sometimes there which is extra misleading.

[willcipriano]

The mouse and keyboard are a vulnerability.

This definition makes the word pretty meaningless. I'd stick to things not doing what they promise to do. To do email doesn't require handling these other, tacked on later, protocols as well. They probably also fail to scan for Nigerian king scams but that isn't really a problem email solves so we don't fault them for it.

[Padlock4543]

I don't fully understand what the role of the inbound service is on "setting headers".

I believe SPF/DKIM/DMARC are mostly (not fully) DNS verifications. Shouldn't these be setup by the domain owner (sender/outbound)?

Is the problem that they don't do the verifications for SPF/DKIM/DMARC for inbound emails?

[throwaway2346mg]

> Is the problem that they don't do the verifications for SPF/DKIM/DMARC for inbound emails?

Yes - the result of the checks aren't passed through for inbound emails (when sent to webhooks).

[TheDong]

Let me get this straight:

Your attack is "I can spoof an email from address". Is that not true for literally anything that accepts email?

The person who receives your spoofed message gets a link to the original message (message-url) and can download it, right? Can't they download it and validate spf, dkim, sender ip, whatever else?

This really sounds like it's working as intended. You asked to receive email. You receive email. You can perform filtering on the email to get rid of spam if you want.

You said "If that address is handled via an inbound route to a webhook [and they don't do any validation] [then phishing is trivial]"

Is that not identical to "if a company runs an SMTP server, you send a spoofed email, and they don't do any validation then phishing is trivial"?

[throwaway2346mg]

> Is that not identical to "if a company runs an SMTP server, you send a spoofed email, and they don't do any validation then phishing is trivial"?

Yes. Except in this case, the company is paying Mailgun to process inbound mail and attach SPF/DKIM/DMARC headers, which they don't do. And this is counter to their own API spec.

If you are running your own SMTP server, then you wouldn't be relying on headers from Mailgun.

Essentially though you are right, using Mailgun is akin to having an SMTP server without any spam protection in place, and limited ability to put that spam protection in place. You are better off running the server yourself.

The point here is that Mailgun customers won't be aware of this, and as such, it's a vulnerability.

[elif]

"I can spoof any email address" leads to an open relay elevated to mailgun's exit ips, presumably with authentication headers added by mailgun's dkim-enabled servers. because it is possible (and likely) that many users will have an inbound email trigger some form of outbound email.

Essentially, vulnerability reporter is saying "hey, you've created a landmine field of open relays, which hackers only have to discover to abuse" and mailgun is saying "no we haven't, our USERS have created those"

that is why this is different from 'literally anything that accepts email.'

[nickphx]

Ehhh, mailgun is NOT relaying the mail.. Their MTA is accepting the message and re-publishing it via a webhook HTTP post to the mailgun user's configured webhook URL(s).

[elif]

right... and if any single user's implementation has inbound mail cause outbound mail, merge variables or multiple RCPT lines will likely cause this user to be an arbitrary mail relay using that user's auth.

[nickphx]

Not sure how that would be the fault of mailgun or how spf/dkim validation enforcement on their end would help with poor design decisions of their clients.

[nubinetwork]

If I'm reading this right, the recipient has to be a mailgun user, and the sender has to be another (yet spoofed) mailgun user.

[throwaway2346mg]

The sender can be anyone. They don't need to be a mailgun user.

The recipient has to be a mailgun user, yes.

[throwaway2346mg]

As pointed out on Reddit [1], if you want to trivially see companies using Mailgun it's as simple as looking at:

https://securitytrails.com/list/mx/mxb.mailgun.org

https://securitytrails.com/list/mx/mxa.mailgun.org

[1] - https://www.reddit.com/r/sysadmin/comments/14vtszl/comment/j...

[throwawaway44]

This is a bit weird.

A friend reported a similar vulnerability at our company about 2 days ago.

Basically there was a insecure SMTP server that you could use to send emails from anyone (internal) to anyone (internal or external).

Literally every employee/contractor on the company network had access to exploit the vulnerability. All you had to do was know how to use the Send-MailMessage function in powershell or something similar.

We work at a publicly traded company, so you could imagine how bad this could've been.

I have no idea if they're using mailgun though. But I wonder if this is related or merely a coincidence?

Note: I'm being intentionally vague about the details for obvious reasons.

[throwaway2346mg]

I don't think this is related.

The issue here is with inbound emails using Mailgun's inbound routes functionality.

Protecting your sending servers from abuse isn't an issue with Mailgun as far as I'm aware.

[caladin]

As far as I know, inbound emails did include the headers you mention.

> Confusingly, these headers appear only for a subset of inbound emails.

Is it a predictable subset? Some unifying characteristic?

[throwaway2346mg]

Yes - it's emails that hit a certain pre-determined spam assassin threshold (see Note B). But this is fairly easy to circumvent (spammers/phishers are especially good at it) and the threshold is reasonably high.

All in all this means the vast majority of emails don't have these headers included in webhooks, and we have plenty of examples of spoofed emails for our own domain (invalid SPF/DKIM and failed DMARC) that get through without triggering spam assassin.

This simplest way to verify this is to store the value of these headers along with all inbound emails hitting your webhooks, you'll quickly see they are missing the majority of the time.

[throwawaaarrgh]

Afaik, AWS is still vulnerable to this too for all CloudWatch alerts. Go ahead and spoof a CloudWatch email as there is no way to ensure its authenticity.

[nickphx]

I'm not sure I understand the issue. What are you doing with the emails received via webhook that SPF/DKIM is needed? Are they being imported into a CRM and re-displayed? I treat any email message sent to the subdomain(s) configured for sending transactional / marketing messages as untrusted and act accordingly.

[throwaway2346mg]

Yep - there are a number of scenarios:

- CRM system (obviously an issue) - Inbound email automation (eg. action based on reply from user / admin / etc)

But really, any inbound action where you don't want someone to be able to trivially spoof the sender, when the sender has SPF/DKIM/DMARC all configured.

For people using Mailgun purely for marketing email purposes, this is unlikely to be an issue, as you're unlikely to be using inbound routes for automation/processing.

[TheCycoONE]

Do you know if they verify the sender for ARF (abuse) emails that should be coming from the recipient's hosting provider e.g. yahoo. If not this could potentially be used for targeted denial of service by submitting fake spam reports on behalf of other recipients to add them to the suppression list?

[nickphx]

If you look at the headers of email originating from mailgun, you will notice several headers they've added that include unique identifiers that identify the sending account and recipient.

ESPs receive FBLs/ARF from email providers through various delivery methods, "webhooks", ARF via SMTP..

So to pull off an attack someone would need to generate matching identifiers and know where to deliver the ARF with the forged data..

[throwaway2346mg]

I'm afraid I haven't checked this - are you a Mailgun user and want to report back on this? Alternatively, hopefully Mailgun themselves will spot this and can respond directly.

[caladin]

Did you ever find an alternative vendor that did this correctly in a manner that you were satisfied with?

[Coronon]

I just implemented a custom little SMTP server with full DMARC authentication yesterday. If what you describe is true, then it’s truely baffling to not acknowledge that as a vulnerability. Aren’t the customers paying to not have the hassle of implementing these things? ^^

[lumpa]

Your comment was dead, as are all your comments and submissions. Probably another case of a spam filter gone bad. As dang says[1], email [e-mail] and they'll solve it.

[1] https://news.ycombinator.com/item?id=36577441

[throwaway2346mg]

Exactly this!