Email Address Disclosures, Preliminary Report

Head of Let&#x27;s Encrypt here. Our automated mail system had a bug that accidentally exposed about 1.9% of subscriber email addresses to the same 1.9% of recipients.<p>Our sincerest apologies for this mistake. We will be doing a thorough postmortem to determine exactly how this happened and how we can prevent something like this from happening again.<p>There is a preliminary report on the issue here:<p><a href="https:&#x2F;&#x2F;community.letsencrypt.org&#x2F;t&#x2F;email-address-disclosures-preliminary-report-june-11-2016&#x2F;" rel="nofollow">https:&#x2F;&#x2F;community.letsencrypt.org&#x2F;t&#x2F;email-address-disclosure...</a>

The reason for this screw up was guessed by a Twitter user, and his theory was confirmed by Josh from Let&#x27;s Encrypt [1].<p>The whole mess was caused by the Python `email` package, and specifically the behavior of the `MIMEMultipart` object [2]. When you reuse the same `MIMEMultipart` object for multiple emails, each destination address is appended. The same problem takes place when you use Python 3 [3].<p>[1]: <a href="https:&#x2F;&#x2F;twitter.com&#x2F;0xjosh&#x2F;status&#x2F;741487697059946497" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;0xjosh&#x2F;status&#x2F;741487697059946497</a><p>[2]: <a href="https:&#x2F;&#x2F;docs.python.org&#x2F;3&#x2F;library&#x2F;email.mime.html#email.mime.multipart.MIMEMultipart" rel="nofollow">https:&#x2F;&#x2F;docs.python.org&#x2F;3&#x2F;library&#x2F;email.mime.html#email.mime...</a><p>[3]: <a href="http:&#x2F;&#x2F;i.imgur.com&#x2F;XwWlUXv.png" rel="nofollow">http:&#x2F;&#x2F;i.imgur.com&#x2F;XwWlUXv.png</a>

Since both users mentioned were the last in the list of addresses for the email they received, my money&#x27;s on a trivial mistake like:<p><pre><code> getEmailBody(users[:i]) </code></pre> instead of<p><pre><code> getEmailBody(users[i]) </code></pre> I typically prefer a high level of polymorphism in my code&#x2F;APIs (sensibly handling single inputs vs. arrays) but this is a great counter-example even if not the actual root cause. Every feature is also a liability. Double edged sword. Etc.

Preliminary report out from Let&#x27;s Encrypt: <a href="https:&#x2F;&#x2F;community.letsencrypt.org&#x2F;t&#x2F;email-address-disclosures-preliminary-report-june-11-2016&#x2F;16867" rel="nofollow">https:&#x2F;&#x2F;community.letsencrypt.org&#x2F;t&#x2F;email-address-disclosure...</a><p>as far as I know, all emails starting with 0-9, A-Z and at least part of &#x27;a&#x27; were exposed. I did not get one starting with &#x27;g&#x27;, so it&#x27;s somewhere between &#x27;a&#x27; and &#x27;g&#x27; that it got stopped.<p>Edit: &quot;7,618 out of approximately 383,000 emails&quot; were sent out

It sucks this happened but I don&#x27;t really care. You guys are providing such an amazing and sorely need service I have no problem cutting you some slack. I hope others will too. Of course those working for companies who&#x27;s lunch you&#x27;re eating will likely run with this as far as they can.

Interesting that since the list of addresses was sequentially prepended to (if I understand the wording of the notice correctly), anyone who anonymously shares the list will incriminate themselves ,though to a smaller and smaller pool of peer customers.

This reminded me exactly of Python&#x27;s mutable default arguments:<p><a href="http:&#x2F;&#x2F;docs.quantifiedcode.com&#x2F;python-anti-patterns&#x2F;correctness&#x2F;mutable_default_value_as_argument.html" rel="nofollow">http:&#x2F;&#x2F;docs.quantifiedcode.com&#x2F;python-anti-patterns&#x2F;correctn...</a>

Sounds like the popular &quot;append e-mail address to e-mail text with each iteration of the loop while keeping the previous ones&quot;.

To clarify: the email addresses are in the body of the message, not the To field.

The Hyatt hotel in Switzerland did a similar thing a few weeks ago. They sent a mail shot to everyone using the CC function not BCC. I complained and their response was that they&#x27;d recalled the mail so &#x27;that was that&#x27;. Of course a recall means nothing to the hordes of gmail addresses,etc. that the mail shot was sent to. It&#x27;s a common problem and a big incentive to use throw away addresses.

Interesting. Slightly embarrassing. Not a huge deal. Handled well.

For the curious this was the content of the email. Pretty generic.<p>&quot;Dear Let&#x27;s Encrypt Subscriber,<p>We&#x27;re writing to let you know that we are updating the Let&#x27;s Encrypt Subscriber Agreement, effective June 30, 2016. You can find the updated agreement (v1.1) as well as the current agreement (v1.0.1) in the &quot;Let&#x27;s Encrypt Subscriber Agreement&quot; section of the following page:<p><a href="https:&#x2F;&#x2F;letsencrypt.org&#x2F;repository&#x2F;" rel="nofollow">https:&#x2F;&#x2F;letsencrypt.org&#x2F;repository&#x2F;</a><p>Thank you for helping to secure the Web by using Let&#x27;s Encrypt&quot;

Planned email blast accidentally cc&#x27;d other recipients, allowing users to see each other&#x27;s email addresses. They caught it after &lt;8,000 emails went out and are fixing the problem.

I think it&#x27;s positive that they own up to it and actually apologize.<p>One would also think that most subscribers of this newsletter has a positive attitude towards the general concepts of privacy and security, so I&#x27;m also positive in thinking that a list of these disclosed addresses will never see the day of light (hoping I&#x27;m not too naive).

I received one of these emails (most likely because my address begins with 73 and the emails are sorted alphanumerically). It looks like this: <a href="http:&#x2F;&#x2F;pastebin.com&#x2F;vpPU5sLj" rel="nofollow">http:&#x2F;&#x2F;pastebin.com&#x2F;vpPU5sLj</a>

I wonder if people whose email addresses start with the letter &#x27;a&#x27; also get more spam.

A reminder: If you got a copy of this e-mail (as I did), please don&#x27;t repost it -- or if you do, don&#x27;t include the e-mail addresses.

Curious to see the reply-all thread that ensues.

Not of those that did not specify any, I would imagine; you can use the ACME service without providing any email address at all.

Good to know, but really I don&#x27;t care if they send my email address to every other registrant. I run a public web server, I already receive junk email that must be filtered, so I see no problems. It has zero impact on the free certificate service they provide.

It&#x27;s disappointing to see this level of incompetence from a group responsible for such great leaps in web security. Let&#x27;s Encrypt should take appropriate steps to ensure this never happens again lest they erode users&#x27; trust any further.

Does this suggest that the first person got sent 7,618 e-mails?

Note to self: use a new email account when using this service.

Hm, I didn&#x27;t get anything.<p>Did you sign up in the earlier or later stages?

Security is not just a product.

What I&#x27;m asking me myself everytime I see a post about data leak, could you sue a company for the leak?

Note to self: keep using HTTP, and provide HTTPS for important website content (like shop payment) and use a SSL&#x2F;TLS Cert that lasts 1 year.

Directly below the apology for leaking emails addresses, I get this message prominently displayed:<p>&gt; <i>&quot;Hey there! Looks like you&#x27;re enjoying the discussion, but you&#x27;re not signed up for an account.</i><p>&gt; <i>When you create an account, we remember exactly what you&#x27;ve read, so you always come right back where you left off. You also get notifications, here and via email, whenever new posts are made. And you can like posts to share the love.&quot;</i><p>&gt; <i>[Sign up] [Remind me tomorrow]</i><p>No thanks :)

The jaded, bitter part of me hopes this will be another nail in the coffin for XaaS and the recent trend of centralizing everything onto Web services.<p>The rest of me which is more jaded and bitter knows that it won&#x27;t.