Head of Let's Encrypt here. Our automated mail system had a bug that accidentally exposed about 1.9% of subscriber email addresses to the same 1.9% of recipients.<p>Our sincerest apologies for this mistake. We will be doing a thorough postmortem to determine exactly how this happened and how we can prevent something like this from happening again.<p>There is a preliminary report on the issue here:<p><a href="https://community.letsencrypt.org/t/email-address-disclosures-preliminary-report-june-11-2016/" rel="nofollow">https://community.letsencrypt.org/t/email-address-disclosure...</a>
The reason for this screw up was guessed by a Twitter user, and his theory was confirmed by Josh from Let's Encrypt [1].<p>The whole mess was caused by the Python `email` package, and specifically the behavior of the `MIMEMultipart` object [2]. When you reuse the same `MIMEMultipart` object for multiple emails, each destination address is appended. The same problem takes place when you use Python 3 [3].<p>[1]: <a href="https://twitter.com/0xjosh/status/741487697059946497" rel="nofollow">https://twitter.com/0xjosh/status/741487697059946497</a><p>[2]: <a href="https://docs.python.org/3/library/email.mime.html#email.mime.multipart.MIMEMultipart" rel="nofollow">https://docs.python.org/3/library/email.mime.html#email.mime...</a><p>[3]: <a href="http://i.imgur.com/XwWlUXv.png" rel="nofollow">http://i.imgur.com/XwWlUXv.png</a>
Since both users mentioned were the last in the list of addresses for the email they received, my money's on a trivial mistake like:<p><pre><code> getEmailBody(users[:i])
</code></pre>
instead of<p><pre><code> getEmailBody(users[i])
</code></pre>
I typically prefer a high level of polymorphism in my code/APIs (sensibly handling single inputs vs. arrays) but this is a great counter-example even if not the actual root cause. Every feature is also a liability. Double edged sword. Etc.
Preliminary report out from Let's Encrypt: <a href="https://community.letsencrypt.org/t/email-address-disclosures-preliminary-report-june-11-2016/16867" rel="nofollow">https://community.letsencrypt.org/t/email-address-disclosure...</a><p>as far as I know, all emails starting with 0-9, A-Z and at least part of 'a' were exposed. I did not get one starting with 'g', so it's somewhere between 'a' and 'g' that it got stopped.<p>Edit: "7,618 out of approximately 383,000 emails" were sent out
It sucks this happened but I don't really care. You guys are providing such an amazing and sorely need service I have no problem cutting you some slack. I hope others will too. Of course those working for companies who's lunch you're eating will likely run with this as far as they can.
Interesting that since the list of addresses was sequentially prepended to (if I understand the wording of the notice correctly), anyone who anonymously shares the list will incriminate themselves ,though to a smaller and smaller pool of peer customers.
This reminded me exactly of Python's mutable default arguments:<p><a href="http://docs.quantifiedcode.com/python-anti-patterns/correctness/mutable_default_value_as_argument.html" rel="nofollow">http://docs.quantifiedcode.com/python-anti-patterns/correctn...</a>
The Hyatt hotel in Switzerland did a similar thing a few weeks ago. They sent a mail shot to everyone using the CC function not BCC. I complained and their response was that they'd recalled the mail so 'that was that'. Of course a recall means nothing to the hordes of gmail addresses,etc. that the mail shot was sent to. It's a common problem and a big incentive to use throw away addresses.
For the curious this was the content of the email. Pretty generic.<p>"Dear Let's Encrypt Subscriber,<p>We're writing to let you know that we are updating the Let's Encrypt Subscriber Agreement, effective June 30, 2016. You can find the updated agreement (v1.1) as well as the current agreement (v1.0.1) in the "Let's Encrypt Subscriber Agreement" section of the following page:<p><a href="https://letsencrypt.org/repository/" rel="nofollow">https://letsencrypt.org/repository/</a><p>Thank you for helping to secure the Web by using Let's Encrypt"
Planned email blast accidentally cc'd other recipients, allowing users to see each other's email addresses. They caught it after <8,000 emails went out and are fixing the problem.
I think it's positive that they own up to it and actually apologize.<p>One would also think that most subscribers of this newsletter has a positive attitude towards the general concepts of privacy and security, so I'm also positive in thinking that a list of these disclosed addresses will never see the day of light (hoping I'm not too naive).
I received one of these emails (most likely because my address begins with 73 and the emails are sorted alphanumerically). It looks like this: <a href="http://pastebin.com/vpPU5sLj" rel="nofollow">http://pastebin.com/vpPU5sLj</a>
Good to know, but really I don't care if they send my email address to every other registrant. I run a public web server, I already receive junk email that must be filtered, so I see no problems. It has zero impact on the free certificate service they provide.
It's disappointing to see this level of incompetence from a group responsible for such great leaps in web security. Let's Encrypt should take appropriate steps to ensure this never happens again lest they erode users' trust any further.
Directly below the apology for leaking emails addresses, I get this message prominently displayed:<p>> <i>"Hey there! Looks like you're enjoying the discussion, but you're not signed up for an account.</i><p>> <i>When you create an account, we remember exactly what you've read, so you always come right back where you left off. You also get notifications, here and via email, whenever new posts are made. And you can like posts to share the love."</i><p>> <i>[Sign up] [Remind me tomorrow]</i><p>No thanks :)
The jaded, bitter part of me hopes this will be another nail in the coffin for XaaS and the recent trend of centralizing everything onto Web services.<p>The rest of me which is more jaded and bitter knows that it won't.