Email from Apple<p>Apple Developer Website Update<p>Last Thursday, an intruder attempted to secure personal information of our registered developers from our developer website. Sensitive personal information was encrypted and cannot be accessed, however, we have not been able to rule out the possibility that some developers’ names, mailing addresses, and/or email addresses may have been accessed. In the spirit of transparency, we want to inform you of the issue. We took the site down immediately on Thursday and have been working around the clock since then.<p>In order to prevent a security threat like this from happening again, we’re completely overhauling our developer systems, updating our server software, and rebuilding our entire database. We apologize for the significant inconvenience that our downtime has caused you and we expect to have the developer website up again soon.
Here's my semi-educated guess for how the attack started: from casual observation (view source, URLs ending with .action, etc) a good chunk of the ADC is written in Java and uses WebWork/Struts2, a framework I helped create years ago.<p>Late last week a security advisory came out that allows for executing malicious code[1]. Atlassian, which uses similar technology, also issued announcements around the same time[2]. My wild speculation is this was the attack vector.<p>Sadly, I feel some responsibility for this pretty major security hole. There have been a few like this and they are all rooted in the fact that almost 9 years ago I made the (bad) decision to use OGNL as WebWork's expression language. I did so because it was "powerful" but it opened up all sorts of extra binding trickery I never intended. I haven't been contributing to the project in 5+ years, but this is a good reminder how technology choices tend to stick around a lot longer than you ever imagine :)<p>[1] <a href="http://struts.apache.org/release/2.3.x/docs/s2-016.html" rel="nofollow">http://struts.apache.org/release/2.3.x/docs/s2-016.html</a>
[2] <a href="https://confluence.atlassian.com/display/BAMBOO/Bamboo+Security+Advisory+2013-07-16" rel="nofollow">https://confluence.atlassian.com/display/BAMBOO/Bamboo+Secur...</a>
> <i>Sensitive personal information was encrypted and cannot be accessed, however, we have not been able to rule out the possibility that some developers’ names, mailing addresses, and/or email addresses may have been accessed.</i><p>So they can't rule out the possibility that sensitive personal information, which cannot be accessed, has been accessed. Got it.<p>Apparently our intelligence, which cannot be insulted, has been insulted.
I downloaded the CRL for developer certificates [1] and quickly looked at it using grep:<p><pre><code> grep -E "Revocation Date: Jul 17 .{8} 2013" wwdrccrl.txt | wc -l
3065
grep -E "Revocation Date: Jul 18 .{8} 2013" wwdrccrl.txt | wc -l
2289
grep -E "Revocation Date: Jul 19 .{8} 2013" wwdrccrl.txt | wc -l
2
grep -E "Revocation Date: Jul 20 .{8} 2013" wwdrccrl.txt | wc -l
0
grep -E "Revocation Date: Jul 21 .{8} 2013" wwdrccrl.txt | wc -l
0
</code></pre>
These are the two certificates that were revoked on the 19th<p><pre><code> grep -A 3 -B 1 -E "Revocation Date: Jul 19 .{8} 2013" wwdrccrl.txt
Serial Number: 2628C7F90970D227
Revocation Date: Jul 19 03:14:04 2013 GMT
CRL entry extensions:
X509v3 CRL Reason Code:
Key Compromise
--
Serial Number: 1A51ABFA4844BD45
Revocation Date: Jul 19 03:24:03 2013 GMT
CRL entry extensions:
X509v3 CRL Reason Code:
Key Compromise
</code></pre>
To generate the wwdrccrl.txt file I used:<p><pre><code> openssl crl -inform DER -text -noout -in wwdrca.crl > wwdrccrl.txt
</code></pre>
Just to be clear -- every entry there I see lists the reason as Key Compromise, just interesting that they usually seem to revoke at least 2000 certificates a day but suddenly stopped on the 19th with just revoking 2.<p>[1]<a href="http://www.apple.com/certificateauthority/" rel="nofollow">http://www.apple.com/certificateauthority/</a>
"Completely overhauling our developer systems, updating our server software, and rebuilding our <i>entire</i> database."<p>That does not sound like an intruder "attempt" by any means.<p>They got hacked, and they got hacked bad if they're rebuilding databases and overhauling entire enterprise-class systems over there.<p>Transparent my ass. They're deep in the gutter, 3-days and counting no fix, engineers are probably working 24 hours a day and the entire site is still down. This isn't a small time breach folks. They had to go public considering it will probably be down for a few more days...
A little more info from TC:
<a href="http://techcrunch.com/2013/07/21/apple-confirms-that-the-dev-center-has-potentially-been-breached-by-hackers/" rel="nofollow">http://techcrunch.com/2013/07/21/apple-confirms-that-the-dev...</a><p>Update — Just got off the phone with an Apple rep, who confirmed a bit more:<p>- The hack only affected developer accounts; standard iTunes accounts were not compromised<p>- Credit card data was not compromised<p>- They waited three days to alert developers because they were trying to figure out exactly what data was exposed<p>- There is no time table yet for when the Dev Center will return
There is an interesting comment at techcrunch:<p><a href="http://fyre.it/tjlVmC.4" rel="nofollow">http://fyre.it/tjlVmC.4</a><p>"[...] One of those bugs have provided me access to users details etc. I immediately reported this to Apple. I have taken 73 users details (all apple inc workers only) and prove them as an example.<p>4 hours later from my final report Apple developer portal gas closed down and you know it still is. I have emailed and asked if I am putting them in any difficulty so that I can give a break to my research. I have not gotten any respond to this.. [...] "
I read the comments dismissing apples handling of this. What would you have expected them to do? There is a LOT of forensics going on probably even now trying to get a handle on this. A massive corp isn't going to make an announcement until they have some idea what they're talking about. In my books 4 days is a very quick first announcement from a company of this size.
These details are befuddling. "Personal information was encrypted and cannot be accessed". It can't be accessed because it's somehow stored elsewhere, or it can't be accessed <i>because</i> of the encryption? That is, does the intruder currently own my encrypted data?<p>I'm also disappointed that it took them 72 hours to tell us <i>anything,</i> and that the update doesn't even have a timeline for when the site may be back. "Soon" is meaningless.
> "In the spirit of transparency, we want to inform you of the issue."<p>Ha, what a joke, I can't help laughing at that.<p>With so many third-party Apple developers drinking the kool-aid, and dreaming of becoming rich, I'm not surprised Apple treat them like fools.<p>Just yesterday on Twitter, some developers were speculating that the site was taken down to be updated with new SDKs for exciting new features and product lines.
Hmm so it only takes a few days to "completely overhaul" their developer systems? Not sure I believe this is what they're actually doing. And why haven't they updated their server software before? I know mistakes can never be completely avoided, but this seems slightly amateurish for a company with so much cash.
I understand everyone's frustrations with this, and the fact that Apple haven't been immediately clear on exactly what happened. As a developer, I too am alarmed by what has happened.<p>But these things are complex, and it takes time (i.e. a few days) to fully and properly evaluate what has happened and what information leaks/security breaches have occurred.<p>Let's give this a reasonable amount of time, and only then pass judgement on their handling of the case.<p>I don't want to appear like an Apple apologist - and maybe it is a serious fault on their side. But in fairness I do think it's reasonable we give them time to evaluate & respond appropriately.
Uh, how does this "encryption" work?<p>For the website to show these details (and it does, in part, use these details in the interface) it must be able to decrypt these on the web applications side. Ergo the keys for decryption must also be on the server or derived from the users passwords, both of which make the use of encryption a fairly worthless venture.<p>ED: As another commenter mentioned in an earlier thread, lots of other AppleID facing applications are gone as well ( <a href="https://ecommerce.apple.com/" rel="nofollow">https://ecommerce.apple.com/</a> ), so it would be interesting to find out how far this all goes. The websites don't seem that far disconnected from the information in iCloud.
I got this email about an hour ago. I feel sorry for the folks who are "updating our server software, and rebuilding our entire database". Songs will be sung in the opsen bars about about this battle.<p>From the sound of the email it suggests they have records of some data (perhaps not sensitive data :-) being compromised but no root cause on how it was compromised, so they are re-building systems from the ground up validating, configuring, and then moving to the next step.There are times where this is faster than spending time trying to root cause the exploit.<p>That said, this is where privacy and security collide. Since logs going back months of what everyone has done on every system really helps reconstruct things, but of course if you have those logs it means that someone else can abuse them.
Good to see some transparency on Apple's part here.<p>I understand this must be a very challenging situation for them to deal with, and I appreciate the notification. As I'm sure many developers feel, I'd like to know more details, but I'm sure these will come in due course.
Theres a security researcher commenting on techcrunch claiming he's responsible for the breach here <a href="http://fyre.it/tjlVmC.4" rel="nofollow">http://fyre.it/tjlVmC.4</a><p>His proof uploaded to youtube: <a href="http://www.youtube.com/watch?v=q000_EOWy80" rel="nofollow">http://www.youtube.com/watch?v=q000_EOWy80</a>
For what it's worth, Wednesday morning at 4am I had an email account associated with my developer account compromised(they both stupidly used the same password). This account was used for almost nothing but accessing my developer accounts at Apple. At the time, I thought my Apple accounts might be in trouble and I immediately changed all my Apple related passwords as well as regained control of my email account. I'm now wondering if the breach might have gone the other direction...
> In order to prevent a security threat like this from happening again, we’re completely overhauling our developer systems, updating our server software, and rebuilding our entire database.<p>I am wondering what was the thought process behind this gem. I think this looks like a knee jerk reaction and it's particularly lacking polish coming from Apple. I mean clearly Apple knows that "overhauling" systems and updating software is no guarantee for future security. It's not a one time fix - it's an ongoing process. And rebuilding entire database - that's just crazy talk! This is especially inexcusable because the target of this update are developers!<p>Security is hard - you've got legacy crap, 3rd party/unsupported code, you've got open source code and then you have your own code that has evolved to be a Frankenstein. I don't have a problem with Apple getting it wrong once - but the statement does nothing to make developers confident that Apple will finally get web services right.
Could it be related to CVE-2013-2251 which was released on 07/20? The URL developer.apple.com/devcenter/ios/index.action seems struts alike..
Jeez people, a company identifies a hack attempt, stops it, and makes sure it never happens again. How often do you hear that one? Most companies don't even tell you anything happened and if they are forced to, they don't even admit anything bad happened (we only exposed 80,000,000 credit cards, no biggie).<p>If my employer suffered this I doubt they'd even tell the employees.<p>What do all of us do when we find a security issue?
This may explain some strange occurrences I had yesterday.<p>Starting at 7am, I received an Apple ID password reset request every 4 hours and 19 minutes, ending last night at midnight.<p>This Apple ID is also the login for my personal developer account (several years old). My developers IDs used for work never received a password reset request.
I can't feel too bad for Apple. They use WW/Struts but when was the last time they contributed to the project? They never have. Open source volunteers do their best but unless big corporations want to spend their own money, and do their own security assessments, and contribute back anything they find, what do you expect? It's great when you get things for free, but when you're sitting on billions, send some back to the community you're using code from.
Can it be related to the similar attack on the ubuntu forum? Maybe it was a single group of hackers targeting the servers in which they know a lot of developers have an account
Is the encryption not good enough (and I mean in general when sites get bcrypt'd passwords stolen, etc) when owners are worried the encrypted data is in the hands of intruders?<p>As a developer I'd still be concerned if I lost such data when encrypted - so I understand - but what measures can be put in place so that as a developer/site owner you're without uncertainty that the encrypted data will never be encrypted by the attacker (eg, would take trillions of years).
If anyone thinks this is the complete truth, well be prepared to be fooled many times more. I mean the thing is down for 3 days now. This must be a huge breach.
Imagine what you could do here:
- break into facebook or twitter or any other high profile dev account
- reissue new code signing keys
- crack the latest public app and patch in a backdoor
- code sign with new keys and submit as an app update
I'm more interested in the identity of the intruder for some reason. Who/what are they? Presumably there are easier targets to steal credit card numbers from, for example.
>> and rebuilding our entire database.<p>maybe someone dropped or polluted the database after hacking it, so they need to rebuild the entire database from other sources?
Manage your Apple ID/password/security questions here: <a href="https://appleid.apple.com" rel="nofollow">https://appleid.apple.com</a>
wow if they're "overhauling" everything that means Apple knows that hackers got some or all developers' info so it's not just that they can't "rule it out" they just don't want to publicly announce it.
How is a developer's mailing address not a sensitive information for that developer? How does a tech company get away making a blanket assumption like that?
I got kicked out of an Apple store. I questioned a Managers
managatorial expertise. I took his angry picture at the
door(Eric in Corte Madera). I am tempted to post it on youtube, but feel punishment enough is working there?
Oh yea, the reason he was furious at me, is because I
didn't like the way he was treating my salesman. I've
never understood people who let a title go to their head?
Off topic, just venting.