The handling of this has been quite terrible. The article is the first "good" communication about an event that started 7 hours prior. And according to the communication will take another 4 days to solve completely.<p>Before this, the Technical Solutions Director tweeted solutions that did not work for end users, but highlighted a typical IT centric approach to problem resolution ("Works, What's the problem?") [1]<p>For anyone not already aware, check out Let's Encrypt. I am evaluating it for about 200 domains now in earnest after having it on my horizon for some time. At least to have it ready as a fallback. [2]<p>Getting 200 EV certificates in a hurry from a different CA has been costly this morning.<p>[1] - <a href="https://twitter.com/vanbroup/status/786548172864626690" rel="nofollow">https://twitter.com/vanbroup/status/786548172864626690</a>
[2] - <a href="https://letsencrypt.org/" rel="nofollow">https://letsencrypt.org/</a>
In case you can't access their website<p>Dear Valued GlobalSign Customer,<p>As most of you are aware, we are experiencing an internal process issue (details below) that is impacting your business. While we have identified the root-cause, we deeply apologize for the problems this is causing you and wanted to ensure you that we are actively resolving the issue.<p>GlobalSign manages several root certificates and for compatibility and browser ubiquity reasons provides several cross-certificates between those roots to maximize the effectiveness across a variety of platforms. As part of a planned exercise to remove some of those links, a cross-certificate linking two roots together was revoked. CRL responses had been operational for 1 week, however an unexpected consequence of providing OCSP responses became apparent this morning, in that some browsers incorrectly inferred that the cross-signed root had revoked intermediates, which was not the case.<p>GlobalSign has since removed the cross-certificate from the OCSP database and cleared all caches. However, the global nature of CDNs and effectiveness of caching continued to push some of those responses out as far as end users. End users cannot always easily clear their caches, either through lack of knowledge or lack of permission. New users (visitors) are not affected as they will now receive good responses.<p>The problem will correct itself in 4 days as the cached responses expire, which we know is not ideal. However, in the meantime, GlobalSign will be providing an alternative issuing CA for customers to use instead, issued by a different root which was not affected by the cross that was revoked, but offering the same ubiquity and does not require to reissue the certificate itself.<p>We are currently working on the detailed instructions to help you resolve the issue and will communicate those instruction to you shortly.<p>Thank you for your patience.<p>Lila Kee
Chief Product Officer
GMO GlobalSign<p>US +1 603-570-7060 | UK +44 1622 766 766 | EU +32 16 89 1900
www.globalsign.com/en
This worked for me on OS X:<p><pre><code> sqlite3 ~/Library/Keychains/*/ocspcache.sqlite3 'DELETE FROM ocsp WHERE hex(serialNum) IN ("040000000001444EF03E20", "040000000001444EF04247");'
</code></pre>
What a pain in the behind :/
How does their SSL warranty play into this? Will they have to pay $1.250.000 for each OrganizationSSL certificate? <a href="https://www.globalsign.com/repository/globalsign-warranty-policy.pdf" rel="nofollow">https://www.globalsign.com/repository/globalsign-warranty-po...</a>
I bought new certificates, for a new set of domains, through AlphaSSL today. One hour later customers starts calling, complaining about revoked certificates. Initially I assumed they had screwed up somehow and revoked our old certs, but after reports saying that it worked with some browsers and failed on others I started googling for recent related issues, and found out about GlobalSign.<p>Man, do they suck at communication. We're now 14 hours into the incident. 6-7 hours ago they posted a trouble shooting guid, promising new intermediate certificates for AlphaSSL and I've just been informed by their support that it'll be another hour before they're ready.<p>It's now 02:00 in dk, so I can expect the new certs at 3 and be done by 4.<p>Fun night.<p>Thanks GlobalSign.<p>P.S. Also thanks to the guy who made their marketing department stop tweeting iot crap while this is going on.
That pissed me off.
Since this may take days to resolve completely, here's a temporary workaround for Chrome users -- launch your browser with this flag:<p><pre><code> chrome --ignore-certificate-errors
</code></pre>
You'll still know when sites have bad certificates due to the red line drawn through the <a href="https://" rel="nofollow">https://</a> portion of the URL. But you will be able to access these sites. But be sure to stop using this flag as soon as you can. It could leak secure cookies to a MitM with a fake cert. <i>Very</i> slim odds of that, but still undesirable. Yet Chrome left us with no other option here.<p>I must say though, I'm increasingly frustrated by software vendors trying to strip away control over our own machines. There is no option at all from the standard error message, even under advanced, to indicate that you know about this problem and wish to proceed. And I'm sure it's only a matter of time before they remove this command-line option as well.<p>I get that novices probably need some protection, but I really wish there were a way to say that, "yes, I <i>really</i> do know what I'm doing, please stop treating me like a toddler." So instead, I'm forced to use a much less safe, hidden command-line option or be locked out of various sites for four whole days.
>some browsers incorrectly inferred that the cross-signed root had revoked intermediates, which was not the case.<p>So it's not their failure but browsers'? Which ones then and what versions?
This killed one of our main webapps, but since the site was hosted on AWS, I provisioned an ACM certificate (free) in about 5 minutes and manually applied it to the ELB listener. Couldn't have been easier.<p>Today's weirdness and communication around it has made me trust Globalsign a lot less now.
Affecting SoundCloud, though I'm not seeing any ssl issues on my end.<p><a href="http://status.soundcloud.com/day/2016/10/13" rel="nofollow">http://status.soundcloud.com/day/2016/10/13</a>
Revoking keys (and the necessary checking that it requires) will never work IMHO. The only way to solve this problem is short key signature lifetimes, automated signatures and, if compromised, just no re-signature.<p>Let's Encrypt is one way, although the lifetime with 3 months is a bit too long. One month or even less would be better. Additional verification and checks via DANE/DNSSEC help to shorten the impact of a compromised key. Constant checking for revocations do not. Again: IMHO.
This issue affected me as soon as I upgraded my chrome to V54 yesterday. It broke all my CDN hosted files which were using the AlphaSSL Wildcard certificate. We were experiencing low traffic and realized this may have been the issue. Got into Chat support with ssl2buy who provided me with a Comodo Wildcard certificate. It was a pain to recreate and install the certs everywhere. But we didn't want to lose any more traffic.
<a href="https://support.globalsign.com/customer/portal/articles/2599710-ocsp-revocation-errors---troubleshooting-guide" rel="nofollow">https://support.globalsign.com/customer/portal/articles/2599...</a><p>They are in the process of fixing certificates. I know of many that are now ok. Too bad I already bought new ones, won't be going back,.
it looks like this is impacting sites hosted on google cloud using the load-balancer (you upload your cert, and I'm using a globalsign cert). I am getting 502 errors via mobile but via desktop it's fine.<p>anyone else use globalsign via google load balancer who can confirm?
I've been wanting to switch all our certificates to Let's Encrypt for almost one year. But, there was always something else which needed my attention more urgently.<p>So, I guess, thank you Global Sign for forcing me to finally make the switch!
Luckily I don't have many tenants yet: I replaced my wildcard AlphaSSL with a couple of Lets Encrypt certs to fill the 4 day gap. Four days of inaccessibility is just not acceptable.<p>This is a real mess.