We just deployed a workaround that brought our site back up.<p>1. Find your DNS Target in heroku. It should end with .herokudns.com<p>2. Lookup the historical DNS record to get the IP addresses. You can find historical DNS records here: <a href="https://securitytrails.com/dns-trails" rel="nofollow">https://securitytrails.com/dns-trails</a><p>3. Replace your CNAME record in your DNS provider with A records that point to the IP addresses you just found.<p>Your site should come back up shortly. We plan to revert back to CNAME records once Heroku gets their DNS issues sorted.
Looks like just DNS for the CNAME is broken.<p><pre><code> $ dig @1.1.1.1 stark-wisteria-rnbgkawldfk6gq7m8308ytts.herokudns.com A
...
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; OPT=15: 00 09 6e 6f 20 53 45 50 20 6d 61 74 63 68 69 6e 67 20 74 68 65 20 44 53 20 66 6f 75 6e 64 20 66 6f 72 20 68 65 72 6f 6b 75 64 6e 73 2e 63 6f 6d 2e ("..no SEP matching the DS found for herokudns.com.")
;; QUESTION SECTION:
;stark-wisteria-rnbgkawldfk6gq7m8308ytts.herokudns.com. IN A
</code></pre>
I wonder if there is any way to get out an IP address of the Heroku router we were assigned to that we can use in place of the CNAME.<p>Might be in the logs somewhere, or in Cloudflare somewhere?
Haven't made any updates, or changes in the last few days, and still won't show my site. So don't think it has to do with new apps or updates.
It seems that NS1 (Heroku's provider) was having problems, but according to them it has been fixed: <a href="https://ns1status.com/#!/incident/365716" rel="nofollow">https://ns1status.com/#!/incident/365716</a>
This does not appear to be a total outage. I cannot reach any of our sites, and Pingdom also reports we are down, however, I can see normal looking traffic reaching our servers (via heroku logs --tail). In addition, members of our team are reporting via Slack that some can reach our Heroku-hosted sites, others cannot. It seems to be ISP-related. Two people within 1 block of each other on different ISPs see different results.<p>We proxy some services through Cloudflare to gain IPv6 support, and all of those are down, which suggests the Cloudflare -> Heroku network route is broken.
Looks like Heroku uses NS1 as the upstream DNS provider. you can find the information like this<p><pre><code> dig NS @1.1.1.1 test.herokuapp.com -> fail
dig @1.1.1.1 test.herokuapp.com -> fail
dig NS @dns1.p03.nsone.net test.herokuapp.com -> works
dig @dns1.p03.nsone.net test.herokuapp.com -> works
</code></pre>
So my conclusion is that NS1 is having issue responding DNS queries from other DNS servers. Interestingly, there is no public information on heroku being dependent on NS1 or any current outages from NS1 status page.
Seeing issues with hackerweb.app and substack.com too.<p>They don't share upstream DNS (and I'm not sure heroku's homepage has the same DNS provider as customer domains). NS matches SOA for each of these domains.<p><pre><code> SOA heroku.com. 1h00m00s "dns1.p04.nsone.net." "hostmaster.nsone.net."
SOA hackerweb.app. 1h00m00s "olga.ns.cloudflare.com." "dns.cloudflare.com."
SOA substack.com. 1h00m00s "ali.ns.cloudflare.com." "dns.cloudflare.com."</code></pre>
Wow. Imagine being so out of touch that you have to have another entity run your DNS services.<p>Email I get, because there has been a hard push for decades to force everyone on to big providers, but DNS can literally be run by anyone, anywhere.<p>Did the primary servers push bad data, making the secondary / tertiary ones break, too? If not, why not extend the cache lifetime and run off of them until the primary are fixed?<p>Sigh. This is rather ridiculous, and is rather embarrassing for Heroku.
I am now seeing one of my services coming back online. For this service I have not replaced the CNAME. Anyone else seeing some service restoration as well?
Why does Heroku have so many uptime issues? Seems to be happening every few months. Last week there was downtime, now again...<p>Is Salesforce committed to Heroku?
Yes, cant even access <a href="https://dashboard-next.heroku.com/" rel="nofollow">https://dashboard-next.heroku.com/</a> so the problem seems broader than what they describe on their status page which seems to imply only issues related to updating DNS settings.
It worked for us too. Thanks.<p>Note: If you used to have a CNAME record for yourdomain.com to www.yourdomain.com, then you have to add two A records per each IP (one whose name is yourdomain.com pointing to the IP, and another whose name is www pointing to the same IP)
Weirdly, dig +trace works fine, but public resolvers like Google and Cloudflare refuse to return the DNS records. This has to be a DNSSEC issue, right? paging tptacek :p