I found it interesting that the attacker looked for help on the attackee's own site. I guess it truly proves how good of a repository of information StackOverflow is.
Kudos to the team over there for being as transparent about what happened and where they were not following best practices - I am pretty sure most companies would not publicly admit this:<p><i>we had secrets sprinkled in source control, in plain text in build systems and available through settings screens in the application.</i>
> However, there is a route on dev that can show email content to CMs and they use this to obtain the magic link used to reset credentials<p>So many sites do this: allowing major changes to be effective immediately (like resetting credentials/password) by simply opening a "magic link" sent by email.<p>I think that this "immediately" is a major security antipattern.<p>I prefer it when such changes have a "cooldown" period of, say, 72 hours, during which the change is "ongoing" but not effective yet and during which the user can veto the change (say by either login on the site, where they'd then get a warning that a major configuration change is ongoing, and denying the change on the site or by opening another "magic link", sent by email, which allows to deny the change).<p>It's not a perfect solution but it stops so many of these oh-so-common attacks dead in their tracks.<p>Because there's a big difference between being able to read an email meant to someone (as happened here, on the server side) and being able to prevent a legit user from receiving emails while also being able to prevent that legit user from login onto a website with its correct credentials.
That was an interesting read. I'm left wondering "why" though. Anyone care to take a wild guess what they were after? That seems like quite a bit of work to be just doing it for no particular reason.
> A significant period of time is spent investigating TeamCity—the attacker is clearly not overly familiar with the product so they spend time looking up Q&A on Stack Overflow on how to use and configure it. This act of looking up things (visiting questions) across the Stack Exchange Network becomes a frequent occurrence and allows us to anticipate and understand the attacker’s methodology over the coming days.<p>Awesome writeup - this gave me a good laugh :-)
"However, there is a route on dev that can show email content to CMs and they use this to obtain the magic link used to reset credentials."<p>Zawinski's Law: "Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can."
> Our dev tier was configured to allow impersonation of all users for testing purposes, and the attacker eventually finds a URL that allows them to elevate their privilege level to that of a Community Manager (CM)<p>> After attempting to access some URLs, to which this level of access does not allow, they use account recovery to attempt to recover access to a developer’s account (a higher privilege level again) but are unable to intercept the email that was sent. However, there is a route on dev that can show email content to CMs and they use this to obtain the magic link used to reset credentials.<p>Many of these debugging tools are great for devs to test things quickly but I've always felt very weary of having these exist in an app without some strict access control with 2FA. Ideally you'd not have them in the app at all, maybe just on local dev.
As someone who doesn't work much with software teams, can someone fill in my gaps for understanding timeline.<p>I'm imagining after a security issue is identified, the steps taken are roughly in the below order and close-ish for the date. I guess my question is, why does it take 20 months from start to blog post?<p>-Contain the issue (1wk)
-Remove the threat (1wk)
-Build up remedies (a few months)
-Check and recheck what happened to make sure you're accurate when submitting final reports (a few months)
-Release a blog post (1month)<p>The timeline is a cool day by day instance, but I just don't understand the larger timeline.
Did they figure out who were behind these attacks? These seems to be quite sophisticated and quite long taking attacks to dig so deeply into SO system.
Thank you SO for being open and listing the best practices. It seems like even few security best practices makes it harder for hackers to get in to your system.<p>I have database connecting strings and password as ENV variables. But I still don't know what is the best practice. Lets say someone gets access to the server, they can still read the ENV vars, right? It definitely prevents from accidently checking in your code git repo. But still . Does anyone has good recommendation for storing credentials like database passwords in a way secured way.
Interesting that most of the mitigations are "move resource behind firewall." Kind of an indictment of the whole BeyondCorp idea - unless we really trust our 2FA to never have any access bypass issues like the initial access to the dev environment here. Speaking of that, I didn't see "fix bug allowing unauthenticated access to dev environment" listed as one of the mitigations, but maybe I glossed over it.
The report describes a security breech in 2019; the report was held back until now for legal reasons:<p>> Sunday May 5th<p>> ...a login request is crafted to our dev tier that is able to bypass the access controls limiting login to those users with an access key. The attacker is able to successfully log in to the development tier.<p>> Our dev tier was configured to allow impersonation of all users for testing purposes, and the attacker eventually finds a URL that allows them to elevate their privilege level to that of a Community Manager (CM). This level of access is a superset of the access available to site moderators.<p>EDIT: clarified that the report was held back
tldr;
1. Attacker found a stackoverflow dev environment requiring a login/password and access key to get in.<p>2. Attacker was able to login to the dev environment with their credentials from prod (stackoverflow.com) by a replay attack based on logging in to prod.<p>3. The dev environments allows viewing outgoing emails, including password reset magic links. The attacker triggered a reset password on a dev account, and changed the credentials. This gives them access to "site settings."<p>4. Settings listed TeamCity credentials. The attacker logged into TeamCity.<p>5. Attacker spends a day or so getting up to speed with TeamCity, in part by reading StackOverflow questions.<p>6. Attacker browses the build server file system, which includes a plaintext SSH key for GitHub.<p>7. Attacker clones all the repos
8. Attacker alters build system to execute an SQL migration that escalates him to a super-moderator on production (Saturday May 11th).<p>9. Community members make security report on Sunday May 12th, stackoverflow response found the TeamCity account was compromised and moved it offline.<p>10. Stackoverflow determines the full extent of the attack over the next few days.
> Fortunately, we have a database containing a log of all traffic to our publicly accessible properties<p><a href="https://stackoverflow.com/legal/privacy-policy" rel="nofollow">https://stackoverflow.com/legal/privacy-policy</a><p>GDPR anyone?