Heroku Security Notification

This is a much bigger problem than Heroku.<p>There are countless SaaS applications asking for full-repo access to Github (all the source code, with <i>write</i> access).<p>- Productboard<p>- Bugsnag<p>- Sentry<p>- Skylight<p>- Percy<p>- CodeTree<p>- Databox<p>There are heaps of others, these are just some on top the of my mind. A ticking supply chain attack waiting to happen, since these companies make themselves into alluring hacking targets.<p>Most of them need access only to issues (a few need read access to code or recent commits, almost none need write).<p>Solution:<p>- Let customers give granular access (only issues, only read to source code, etc) when the integration is setup. This is possible with Github&#x27;s APIs.<p>- Try to use push instead of pull where possible, i.e. provide a CLI tool to use with Github actions or use Github&#x27;s webhooks.

&gt; Additionally, we recommend disconnecting Heroku from your GitHub repositories.<p>I have respect for the Heroku&#x2F;Salesforce Security team for willing to ask users to perform this action. Many companies would be too worried about losing customers or having users not reconnect it afterwards.<p>My thoughts are with the team working on responding to this incident on Easter Friday.

We&#x27;re a small org with a github connected to heroku. All of our repos were cloned between April 8 and April 15 with the majority of them having no activity for several years. The audit logs don&#x27;t show this, you can only see this information in the traffic graphs (&#x2F;graphs&#x2F;traffic). If you&#x27;re seeing cloning of repos that you haven&#x27;t touched in a while, you&#x27;ve likely been compromised.

Ughh here I was at 1:30AM after a hard week, checking the news one last time before going to bed, and see this. I hate this dang industry and regret ever becoming a tech lead. I know the Heroku engineers have it worse and all, but just venting.

I&#x27;m not a security expert, but if you&#x27;re reading this and wondering what to do, a good start could be to just assume your repo was accessed, and so to run a tool like gitleaks against your repo. If it detects anything sensitive, I&#x27;d see about revoking&#x2F;deleting those secrets right away.<p>In general, it&#x27;s good practice not to check anything sensitive into source code for precisely this reason (if your code is compromised you don&#x27;t want your secrets to be as well). So it&#x27;d also be good practice to add something like gitleaks into your CI&#x2F;CD pipeline for the future.

I do remember hooking up Heroku to Github for auto-deployments and thinking to myself something along the lines of, &quot;why does Heroku need ALL of this access?&quot;<p>It&#x27;d be great if Github could allow read&#x2F;write permission grants on a per-repo basis. Maybe they do already!.. in which case I&#x27;d much rather have and setup that granular detail than have a token that goes across all my public&#x2F;private repos...<p>Edit: I do see in my Github&#x27;s integration page that the Heroku connection was used within the past week... but it doesn&#x27;t show how exactly it was used. Until Github can provide specific details, is it safe to assume that all repos, public and private, could have been cloned?

Disappointed that the GitHub security log doesn’t show access for personal accounts. Would be rather nice if they temporarily made that available for a short period of time so we can see if any of our repositories have been cloned&#x2F;downloaded.

FYI looks like the stolen Heroku OAuth tokens are already being used:<p><a href="https:&#x2F;&#x2F;github.blog&#x2F;2022-04-15-security-alert-stolen-oauth-user-tokens&#x2F;" rel="nofollow">https:&#x2F;&#x2F;github.blog&#x2F;2022-04-15-security-alert-stolen-oauth-u...</a>

Interesting I had an OLD project associated with a Gmail email address that would send emails in some cases.<p>Yesterday I got a notification that someone tried logging into that Gmail account. The password was hard coded in the code…

The attacker got _write_ access to all of these repositories as well? That&#x27;s extremely worrying. I hope github or someone will be able to track down if any code changes were made.

If I want to revoke all Heroku&#x27;s access to Github, is it the &quot;Heroku Dashboard&quot; I&#x27;m looking for under &quot;Authrorized Oauth apps&quot;? I revoked that one, but not sure if that is everything.

This is a chance to reiterate best practices:<p>Credentials and other secrets, like API keys, should never be hard-coded in the source code repo. Use some sort of secrets management or configuration for that kind of stuff.

Related HN article: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=31046791" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=31046791</a>

This is as good a time as any to ask about Render.<p>I&#x27;ve been eyeing it recently and I&#x27;m thinking about launching my next project with it. Does anyone have any takeaways from using Render vs Heroku?

See also GitHub&#x27;s post about this: <a href="https:&#x2F;&#x2F;github.blog&#x2F;2022-04-15-security-alert-stolen-oauth-user-tokens&#x2F;" rel="nofollow">https:&#x2F;&#x2F;github.blog&#x2F;2022-04-15-security-alert-stolen-oauth-u...</a><p>Travis-CI was also compromised here and that may actually affect more people than the Heroku side of this.

I have always wondered if putting confidential info in even private git repo was a good idea, although it seems to be a common practice? I feel like that question has been answered, for me anyway.

Instead of connecting to a Github repo, you can use <a href="https:&#x2F;&#x2F;github.com&#x2F;heroku&#x2F;heroku-builds" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;heroku&#x2F;heroku-builds</a>. It allows to create a build locally and then deploy it to Heroku. From what I&#x27;ve read so far, this approach has not been impacted and it should still be possible to do deployments like that.

I might be naive but how does a threat actor get access to encrypted bunch of keys.

I just started a Discord server: <a href="https:&#x2F;&#x2F;discord.gg&#x2F;K9ecetqn" rel="nofollow">https:&#x2F;&#x2F;discord.gg&#x2F;K9ecetqn</a> Please join if you&#x27;re impacted by this incident, or interested in these topics in general.<p>I&#x27;d like to discuss mitigations around this and similar incidents with other HN:ers:<p>- Knowledge sharing: resources, how-tos, tips - Discussing prevention, mitigation, etc - Moral support and venting<p>If there&#x27;s already such a forum (I assume there is), please send me an invite :)

The only thing in my security log over at GitHub that I&#x27;m not familiar with is a handful of &quot;repo.change_merge_setting...Blocked a merge setting on the ${my_repo} repository&quot; entries. Googling that provides little information. Can anyone explain what that means? Should I be worried?

Heroku users using GitHub should start rotating any secrets stored in their repos (people still do), if the OAuth has been compromised it means your repositories could have been cloned. Non-enterprise GitHub users have no detailed audit history to see if their repos were cloned&#x2F;zipped.

Does anyone know what to look for in the github audit logs, exactly?

To me, this reads like someone got read access to all the source code on Heroku, so if you have any credentials hardcoded in there, now&#x27;s the time to change them, too.<p>I wonder if the hackers were kids who got bored around Easter holiday - meaning Heroku&#x27;s security is shit - or if Heroku deliberately waited to announce this during Easter holiday to minimize the attention it gets - meaning they are as deceitful as all proper megacorps.<p>I haven&#x27;t been able to trust their status page to accurately reflect what works and what doesn&#x27;t for a long time. The only reliable signal is when their status page goes offline ;)

Heroku Support is sh*t. Thanks. I can&#x27;t access my account now.

Ah yes, the Friday night security incident drop.