Writeups and actions like this from cloudflare are exactly why I trust them with my data and my business.<p>Yes, they aren’t perfect. They do some things that I disagree with.<p>But overall they prove themselves worthy of my trust, specifically because of the engineering mindset that the company shares, and how serious they take things like this.<p>Thank you for the blog post!
> Analyzing the wiki pages they accessed, bug database issues, and source code repositories, it appears they were looking for information about the architecture, security, and management of our global network; no doubt with an eye on gaining a deeper foothold.<p>For a nation state actor, the easiest way to accomplish that is to send one of their loyal citizens to become an employee of the target company and then have the person send back "information about the architecture, security, and management" of the target company.<p>Fun (but possibly apocryphal) fact: more than a decade ago in a social gathering of SREs at Google, several admitted to being on the payroll of some national intelligence bureaus.
> The one service token and three accounts were not rotated because mistakenly it was believed they were unused.<p>Eh? So why weren't they revoked entirely? I'm sure something's just unsaid there, or lost in communication or something, but as written that doesn't really make sense to me?
> Even though we believed, and later confirmed, the attacker had limited access, we undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials), physically segment test and staging systems, performed forensic triages on 4,893 systems, reimaged and rebooted every machine in our global network including all the systems the threat actor accessed and all Atlassian products (Jira, Confluence, and Bitbucket).<p>> The threat actor also attempted to access a console server in our new, and not yet in production, data center in São Paulo. All attempts to gain access were unsuccessful. To ensure these systems are 100% secure, equipment in the Brazil data center was returned to the manufacturers. The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway.<p>They didn't have to go this far. It would have been really easy not to. But they did and I think that's worthy of kudos.
Thing about a data breach is once the data is out there - source code in this case - it’s out there for good and you have absolutely no control over who gets it. You can do as much post incident hardening as you want, and talk about it as much as you want, but the thing you’re trying to protect against, and blogging about how good you’re getting at preventing, has already happened. Can’t unscramble those eggs.
> The threat actor searched the wiki for things like remote access, secret, client-secret, openconnect, cloudflared, and token. They accessed 36 Jira tickets (out of a total of 2,059,357 tickets) and 202 wiki pages (out of a total of 14,099 pages).<p>In Atlassian's Confluence even the built-in Apache Lucene search engine can leak sensitive information and this kind of access (to the info by the attacker) can be very hard to track/identify. They don't have to open a Confluence page if the sensitive information is already shown on the search results page.
>The one service token and three accounts were not rotated because mistakenly it was believed they were unused.<p>This odd to me - unused credentials should probably be deleted, not rotated.
So after the Okta incident they rotated the leaked credentials...<p>But I think they should have put honeypots on them, and then waited to see what attackers did. Honeypots discourage the attackers from continuing for fear of being discovered too.
They mention Zero Trust, yet you can gain access to applications with <i>just</i> a single bearer token?<p>Am I missing something here?<p>There’s no machine cert used? AuthN tokens aren’t cryptographically bound?<p>This doesn’t meet my definition of ZT, it seems more like “we don’t have a VPN”
This is an excellent report, and congratulations are due to the security teams at CS for a quick detection, response and investigation.<p>It also highlights the need for a faster move in the entire industry away from long-lived service account credentials (access tokens) and toward federated workload identity systems like OpenId connect in the software supply chain.<p>These tokens too often provide elevated privileges in devops tools while bypassing MFA, and in many cases are rotated yearly. Github [1], Gitlab, and AZDO now support OIDC, so update your service connections now!<p>Note: I’m not familiar with this incident and don’t know whether that is precisely what happened here or if OIdC would have prevented the attack.<p>Devsecops and Zero Trust are often-abused buzzwords, but the principles are mature and can significantly reduce blast radius.<p>[1] <a href="https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect" rel="nofollow">https://docs.github.com/en/actions/deployment/security-harde...</a>
Cloudflare being compromised would be enormous. Something between 5 and 25% of all sites use CF in some fashion. An attacker could literally hold the internet hostage.
What I don't understand is how they got access to Jira yet you still insist there was no compromise.<p>The very nature of Jira and Confluence (both terrible products, btw) is to collect documentation. I'm assuming it was an internal Jira/Confluence for engineering teams, but still. There have got to be addresses, passwords, service account info, all kinds of info. If it was a tech support server then it's impossible to assert that you didn't lose customer data.<p>So we have this double standard where you pay for this product that is designed to house your deepest secrets and most cherished organizational information, that's so important to you that you run on premises servers to keep it safe, but it's not important enough to constitute a real "beach".<p>You're lying. Either the server contained junk of no value in which case it wouldn't have existed in the first place, or you actually did lose something of value that you won't identify to us. Nobody sets up on-prem Jira just to leave it empty and never put secrets in it.
Am I the only one who just sees a totally blank page?<p>Viewing the HTML shows it's got an empty body tag, and a single script in the <head> with a URL of <a href="https://static.cloudflareinsights.com/beacon.min.js/v84a3a4012de94ce1a686ba8c167c359c1696973893317" rel="nofollow">https://static.cloudflareinsights.com/beacon.min.js/v84a3a40...</a>
> To ensure these systems are 100% secure, equipment in the Brazil data center was returned to the manufacturers. The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway.<p>The thoroughness is pretty amazing
Such a beautiful report and beautiful ownage.<p>Whenever some shitty Australian telco gets owned, people are angry and call them incompetent and idiots; it's nice to see Cloudflare gets owned in style with much more class and expertise.<p>Like the rest of the HN crowd, this incident has only increased my trust in Cloudflare.
>They did this by using one access token and three service account credentials that had been taken, and that we failed to rotate, after the Okta compromise of October 2023. All threat actor access and connections were terminated on November 24 and CrowdStrike has confirmed that the last evidence of threat activity was on November 24 at 10:44.<p>Okta hitting everywhere
> The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway.<p>This seems incredibly wasteful.<p>Replacing an entire <i>datacenter</i> is effectively tossing tens of millions of dollars of compute hardware.
> Then, from November 27, we redirected the efforts of a large part of the Cloudflare technical staff (inside and outside the security team) to work on a single project dubbed “Code Red”.<p>Why didn't they start this effort BEFORE there was an incident?<p>> we undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials<p>Bearer credentials should already be rotated on a regular basis. Why did they wait until an incident to do this?<p>> To ensure these systems are 100% secure<p>Nothing is 100% secure. Not being to see and acknowledge that is a huge red flag.<p>> Nothing was found, but we replaced the hardware anyway.<p>Well that is just plain stupid and wasteful.<p>> We also looked for software packages that hadn’t been updated<p>Why weren't you looking for that prior to the incident?<p>> we were (for the second time) the victim of a compromise of Okta’s systems which resulted in a threat actor gaining access to a set of credentials.<p>And yet they continue using Okta. The jokes just write themselves.<p>> The one service token and three accounts were not rotated because mistakenly it was believed they were unused.<p>Wait, wait, wait. You <i>KNEW</i> the accounts with remote access to your systems were <i>UNUSED</i> and yet they continue to be active? Hahahahaha.<p>> The wiki searches and pages accessed suggest the threat actor was very interested in all aspects of access to our systems: password resets, remote access, configuration, our use of Salt, but they did not target customer data or customer configurations.<p>Totally makes sense, I'm sure the attacker was just a connoisseur of credentials and definitely did not want them to target customer data.
Reading this 2 months after the fact feels a bit late, but I guess it’s better for your stock price if these revelations happen with remediation already in hand?<p>Since they didn’t really have reason to believe my data was accessed, maybe that’s ok. I know from firsthand experience how hard rotating all your credentials across the whole org is.
I am trying to learn and understand the attack. Can someone please help me brainstorm some possibilities? (please note, I do not intend to question Cloudflare's business practices or security program; my intention is to learn and understand).
--<p>Blog: The threat actor (TA) accessed Okta’s customer support system and viewed files uploaded by Cloudflare (CF) as support cases.<p>Why was the session token part of the support files uploaded to Okta support? Does Okta require it for troubleshooting?<p>TA hijacked a session token of a CF employee from a support ticket.<p>Blog: Using the token extracted from Okta, the TA accessed Cloudflare’s Okta and compromised two separate Cloudflare employee accounts within the Okta platform.<p>How did this happen? Was the stolen token privileged? Also, why only 2 employee accounts? Were these different employees, or was one the same one whose token was compromised? What does Okta employee account compromise mean - did TA reset the password and MFA, and how or was there no MFA?<p>Blog: TA used stolen credentials to get access to our Atlassian server and accessed some documentation and a limited amount of source code.<p>Did the stolen employee credentials only have access to Atlassian?<p>Blog: TA gained access to a set of credentials<p>Does this mean multiple credentials got uploaded to the Okta support system?<p>Blog: The Okta compromise was in October, but the threat actor only began targeting CF systems using stolen credentials from the Okta compromise in mid-November.<p>Does this mean the compromised token was long-lived?<p>Blog: We failed to rotate one service token and three service accounts (out of thousands) of credentials that were leaked during the Okta compromise.<p>I didn’t get this. Does this mean that over time, CF employees had uploaded support info for 1000s of apps managed by Okta? Which credentials did CF rotate initially after the Okta compromise?<p>Leaked Credentials:
1. Moveworks service token that granted remote access into our Atlassian system.<p>Is this service token a bearer token? And without expiry. Is this like an API key?<p>TA accessed Atlassian Jira and Confluence using the Moveworks service token to authenticate through the gateway.<p>2. A service account used by the SaaS-based Smartsheet application that had administrative access to our Atlassian Jira instance,<p>So here, the Smartsheet Saas was given access to the on-prem Atlassian Jira instance? What kind of trust is it? Is this as well managed through Okta? And how does support case filing include a service account? Here, does the Service account mean again some kind of hardcoded API key without expiry<p>TA used the Smartsheet service account to gain access to the Atlassian suite. They used Smartsheet credentials to create an Atlassian account that looked like a normal Cloudflare user. They added this user to a number of groups within Atlassian so that they’d have persistent access to the Atlassian environment.<p>Since the Smartsheet service account had administrative access to Atlassian Jira, the TA was able to install the Sliver Adversary Emulation Framework, which is a widely used tool and framework that red teams and attackers use to enable “C2” (command and control), connectivity gaining persistent and stealthy access to a computer on which it is installed. Sliver was installed using the ScriptRunner for the Jira plugin.
This allowed them continuous access to the Atlassian server, and they used this to attempt lateral movement. With this access, the Threat Actor attempted to gain access to a non-production console server in our São Paulo, Brazil data center due to a non-enforced ACL. The access was denied, and they could not access any global networks.<p>3. A Bitbucket service account, which was used to access our source code management system<p>4. AWS environment that had no access to the global network and no customer or sensitive data.<p>Were these AWS access keys? Also, it looks like these keys did provide access to the AWS account. That means the access key didn’t require MFA.<p>The only production system the TA could access using the stolen credentials was our Atlassian environment.<p>Mitigations:<p>Blog: We decided a huge effort was needed to further harden our security protocols to prevent the threat actor from being able to get that foothold had we overlooked something from our log files.<p>What does hardening security protocol mean here? Is it techniques for D&R or something else<p>Blog: We undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials)<p>I believe this means forced resets on employees (Okta users), right?
Great write up.<p>> Over the next day, the threat actor viewed 120 code repositories (out of a total of 11,904 repositories<p>> They accessed 36 Jira tickets (out of a total of 2,059,357 tickets) and 202 wiki pages (out of a total of 14,099 pages).<p>Is it just me or 12K git repos and 2 million JIRA tickets sound like a crazy lot. 15K wiki pages is not that high though.<p>> Since the Smartsheet service account had administrative access to Atlassian Jira, the threat actor was able to install the Sliver Adversary Emulation Framework, which is a widely used tool and framework that red teams and attackers use to enable “C2” (command and control), connectivity gaining persistent and stealthy access to a computer on which it is installed. Sliver was installed using the ScriptRunner for Jira plugin.<p>> This allowed them continuous access to the Atlassian server, and they used this to attempt lateral movement. With this access the Threat Actor attempted to gain access to a non-production console server in our São Paulo, Brazil data center due to a non-enforced ACL.<p>Ouch. Full access to a server OS is always scary.