How Threadable solved the DMARC problem

Update: Steve Atkins of Word to the Wise points out that the solution described below is lacking a critical detail and, as originally described, could train users in a way that makes your list exploitable to phishers. I've updated the "How it works" section to add details about the safest implementation.

What Happened?

Over the weekend, Yahoo changed a setting that affects the deliverability of messages sent from yahoo.com addresses. Specifically, Yahoo published a DMARC record with a policy to reject all yahoo.com mail that fails DMARC. 

The result is that mail sent from yahoo.com addresses will now bounce when sent via a mailing list, like Google Groups or mailman. Worse, these bounces can result in the senders being unsubscribed from the list altogether. As John Levine of IETF explains:

Since Yahoo mail provokes bounces from lots of other mail systems, innocent subscribers at Gmail, Hotmail, etc. not only won't get Yahoo subscribers' messages, but all those bounces are likely to bounce them off the lists.

He recommends that all list administrators immediately stop delivering mail from Yahoo addresses to limit damage, and encourage members to move to a more friendly provider.

How Threadable Fixes This

Today, Threadable released an update that fixes the DMARC issue for our users. Rather than bounce a message that fails DMARC (or worse, unsubscribe the sender), Threadable will proactively check the sender domain's DMARC settings and generate a message with correctly aligned identifiers, resulting in a successful delivery. 

Better still, this fix doesn’t just work for the yahoo.com problem, but for any domain owner who might change their DMARC policy in the future. This happens more than you'd think: what if you subscribe your mailing list to Twitter notifications? What if you have a procmail rule that forwards important notices from your bank to your accounting department? These are the correct application of a DMARC 'reject' policy, and all of these examples will break your average mailing list.

One Step Further

Because of Threadable's unique design, we were able to re-deliver all the messages that failed using the new DMARC-friendly headers. So, no Threadable users missed email: it was just a bit delayed for some of them.

This is possible because on the backend, Threadable is very different from other mailing lists. Rather than using a mail reflector and a separate archive, emails actually pass through Threadable's message store as part of the remailing process. Each message can be customized and delivered on-demand to a single user. In some ways, it's more like an email client than a list server. In this case, it was trivial to identify the individual rejected outgoing emails and re-send them.

How does it work?

The DMARC FAQ suggests three options for mailing list operators. We've implemented the third, "take ownership of the email."

The minimum standard here is to rewrite the From header to point to an address at your own domain. There's even a patch for Mailman 2.1 that implements this option by replacing From with the list address. However, maintaining a good user experience requires a few additional tweaks.

Threadable checks the From address domain's DNS to determine if the DMARC policy is too strict to interfere with delivery ("p=reject"), and if so, does the following:
  1. Rewrite the From address to use the local domain: "Some Guy <someguy@example.com>" becomes "Some Guy via Threadable <placeholder@threadable.com>". I like this because it preserves the name part of the address, which is super useful when browsing my inbox.
  2. Sign the new message with DKIM, and send using an SMTP envelope From at threadable.com. This gets the message delivered.
  3. If Reply-to munging is enabled for the recipient (this is a user-level setting on Threadable, not a list-level setting as on most mailing lists):
    1. Add the original From address to the CC, so that it's available to the recipient
    2. Remember, Reply-to was already being set in this case, so there's no need to change it
  4. If reply-to munging is disabled for the recipient
    1. Set the reply-to address to the original From address
This preserves the reply semantics. That is, if the recipient expects a reply to go to the list, it will. If they expect the reply to go to the sender, it will do that instead. If they hit reply-all, the presence of the list in the To header ensures the message goes to both the sender and the list.

We're considering an implementation that makes the From address deliverable, but that's probably not necessary: setting Reply-to should protect the recipient from ever using our placeholder address, or getting it added to their address book.

Important note: If you're going to implement this, make sure to never include the sender domain in your rewritten From address. Doing so could teach your users to trust that domain to indicate the sender's identity, and make your list vulnerable to phishing. Here's specifically what Threadable does when rewriting From:
  • Grab the name part if it's present, so "Some Guy <someguy@example.com>" becomes "Some Guy via Threadable <placeholder@threadable.com>"
  • If there's no name part, grab only the local part of the email. So, "no-name@example.com" becomes "no-name via Threadable <placeholder@threadable.com>"
  • Whatever you do, do not write an address that looks like "no-name@example.com via Threadable <placeholder@threadable.com>".
Other solutions

If you don't want to do it this way, there are two other options offered in the FAQ:
  • Operate as a strict forwarder, so as to not break the original message's DKIM signature. This works as long as your list has no features. As soon as you add a link to a footer, change a header, or enable web-based posting, you'll need to generate your own DKIM signatures.
  • Add the Original Authentication Results (aka XOAR) header. It's totally worthwhile to do this. However, it only makes a difference if the destination ESP has listed you as a trusted sender. If you have that luxury, awesome! Most mailing list operators don't.
Was this the right thing for Yahoo to do?

Not a chance. A restricted DMARC policy makes sense for domains on which phishing is a serious risk, and who are not also email service providers. An ESP's job, first, is to interoperate with existing Internet infrastructure. It will be a couple years at least before DMARC support is widely deployed enough in mailing lists to make this feasible. Furthermore, this makes it difficult if not impossible to use a yahoo.com address with many 3rd-party mail clients.

Because lists tend to unsubscribe addresses that generate bounces, Yahoo is not only breaking email for their own customers, but for everyone else.

I expect that they'll eventually figure out what they've done and change it. It was a fun challenge to work around this, and exercise the power of our platform, but it's bad for the Internet. Yahoo: please fix this soon.

References