Heroku has been running a second copy of my scheduler instance

Heroku has been an absolute dumpster fire. Every time their name comes up I can't help but mention that they deleted my account for inactivity, but have still been charging my credit card. No easy way to cancel because... they deleted my account.

This is why I use the git commit short hash as part of the identification of the service (eg: container name). Allows for quick check of which is the running code version and identify problems like these.

Kinda refreshing - within the ever growing genre of stories about Heroku&#x27;s downfall into junk status - to find such an article with a higher level of detail and analysis of the origins of the problem.<p>There&#x27;s one thing that somewhat surprised me though: this line:<p>&gt; &quot;In 2018 Heroku was pretty much the only option for a newbie like me to get started with web-development&quot;<p>Was it? Really? I mean, I&#x27;m not saying the author did anything wrong in choosing Heroku at the time (after all, who could have known back then what a dumpster fire Heroku would turn out to be), and the commercial success Heroku achieved before going down the gutter shows that it must have had some good reasons going for it, at least on paper before the users are confronted with problems that make them judge the situation differently…<p>… but I don&#x27;t really get where that &quot;pretty much the only option&quot; vibe would come from. Maybe It&#x27;s just me having completely different selection criteria, but from my perspective, there were lots of other options to &quot;get started with web-development&quot; and Heroku was just one among many. I don&#x27;t see at all by what measure Heroku would have seemed like &quot;pretty much the only option&quot;?

In case it&#x27;s not clear from the article, Nixpacks build OCI (essentially, Docker) images, so they are perfectly vendor neutral as far as I know. The ending ambiguously implies that Nixpacks may be undesirable vs using Docker, but actually it&#x27;s just another way to build an image. Fly.io supports Nixpacks out of the box too, I believe, which makes it even easier to use them with Fly (although, you can also just push an OCI image directly to the fly.io registry and deploy with that, which I have been experimenting with in spare time as an option for using Nix itself to build an OCI image for Fly. I&#x27;m sure there will be an even nicer way some day, but it works for now...)<p>I&#x27;m sure the author knew this, but I wanted to iterate it in case anyone misunderstood.

Everyone can relate to the sad demise of Heroku at this time. Sad, because they could&#x27;ve dominated this space which still has a lot of issues.<p>Yet-another-infrastructure-group nailed it in their graphic about &quot;what your infrastructure says about you?&quot;... &quot;Heroku =&gt; &quot;You are either negotiating the pricing or deplatforming&quot; <a href="https:&#x2F;&#x2F;www.yaig.dev&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.yaig.dev&#x2F;</a>

Not sure if you have done this already but: If you rotate your database, redis and email provider passwords&#x2F;tokens, the zombie process might still be running but will not have any effects any more.

Probably related to <a href="https:&#x2F;&#x2F;status.heroku.com&#x2F;incidents&#x2F;2531" rel="nofollow">https:&#x2F;&#x2F;status.heroku.com&#x2F;incidents&#x2F;2531</a> (&quot;On April 6, 2023, between 21:20 UTC and 22:30 UTC, our Common Runtime customers were unable to provision dynos in the US and EU regions. This issue impacted starting&#x2F;scaling dynos in existing apps, one-off dynos, builds, and release phases.&quot;)

Similar thing happened to me. Took a long time to realize there was a zombie Heroku instance running jobs.<p>That horror when you delete code but it&#x27;s still running, you&#x27;re just so confused...<p>I&#x27;ve forgotten how I resolved it, but man it was a pain to diagnose.

&gt;&gt; So is it Celery then? This awesome library has its fair share of complaints about duplicated task execution — most are due to misconfiguration. I scanned dozens of Stack Overflow threads and GitHub issues in Celery’s repository. Very few were relevant to my case, and none of the fixes were.<p>I was so annoyed by the pain of configuring Celery I stopped my project and wrote my own job queuing system.<p>Job and message queues are ridiculously over complicated - including the one I wrote. For trivial usage, job queuing should be zero config.

&gt; In 2018 Heroku was pretty much the only option for a newbie like me to get started with web-development<p>Free PHP webhosts have been around for 15+ years

When heroku first came out they introduced so many wonderful new things that blew the development community away. Many people including myself were inspired by their work. They did serverless before serverless even existed as a marketing concept.<p>But I guess having salesforce buying them out pretty much stalled their progress, and over time leading them to where they are today.<p>Anyway this is why we&#x27;re building something to continue to innovate in this space, there is a lot to do. <a href="https:&#x2F;&#x2F;instellar.app" rel="nofollow">https:&#x2F;&#x2F;instellar.app</a>

If Heroku is running some zombie instance of your app, what if you were to just kill the instance you control and let Heroku run the zombie instance? When your service goes fully down, you&#x27;ll know they&#x27;ve killed the zombie instance, so you can then run a clean copy.

It’s such a pity that Heroku has fallen so much. But I definitely wouldn’t run anything I cared about on Heroku these days. I ended up with my apps on NorthFlank which I’ve been very happy with. Railway also looks like a good choice, especially for those looking for a free plan.

An option might be to fix your message IDs, e.g. something like 20230413-userid@yourapp. Let the recipient&#x27;s email server de-duplicate if you are not sure you can do better than at-least-once delivery on your side.

Friendly reminder that if you want Heroku-like deployments for simple projects you can go and roll your own with <a href="https:&#x2F;&#x2F;github.com&#x2F;piku">https:&#x2F;&#x2F;github.com&#x2F;piku</a> :)

Author and me have so much in common. From journalism to software engineering and our formative career shift years were spent on home made music apps. Now I’m wondering just how many of us are out there.

In 2017, I worked at a company that built a websocket based app and ran it on Heroku. We had this exact same problem and all 5 of the engineers at the time spent all day until we realized that we had socket connections still connected to some &quot;ghost dyno&quot;.<p>I can&#x27;t remember exactly what we did but I think we just kept deploying over and over again until the old one finally went away.

I guess any &quot;easy&quot; way to solve this right now is to put some authentication on the celery communication bus? I wonder if flower is being used.<p>Once auth is required for the celery processes, the old zombie wont have access. And if this occurs again OP could revoke access etc.

Great read, thanks!<p>I haven&#x27;t heavily used Celery anymore for the past few years, but I remember trying to setup a HA Celery Beat schedule process with Kubernetes in 2019 and hitting this exact same issue! There was an alternative scheduler[1] storing its state in Redis instead of a local file, after a few fixes it was working pretty well: multiple beat processes running concurrently and not firing twice a scheduled task.<p>Don&#x27;t know if that still works though.<p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;mixkorshun&#x2F;celery-beatx">https:&#x2F;&#x2F;github.com&#x2F;mixkorshun&#x2F;celery-beatx</a>

Clearly the author still hasn&#x27;t learned his lesson: his &quot;no more vendor lock&quot; solution is another vendor(s) - which is always able to vendor-lock him, again.<p>It is truly mind blowing on how many people keep falling for one of the oldest trick in IT : vendor lock-in.<p>===<p>No more vendor lock In 2018 Heroku was pretty much the only option for a newbie like me to get started with web-development. The landscape is very different now with providers like Render, Railway, and more

<a href="https:&#x2F;&#x2F;caprover.com" rel="nofollow">https:&#x2F;&#x2F;caprover.com</a> FTW - just host everything yourself. you can host multiple services without any problem on a 5$ VPS nowadays, especially if you using just a simple SSR webapp, some lightweight backend and sqlite db. as soon as your hardware is not keeping up with your demand, you make enough money to scale vertically.

Even though it won&#x27;t help you now, instead of celery-once I highly recommend using the celery-redbeat scheduler which uses a distributed lock to ensure that only one scheduler instance is active. It&#x27;s plug&amp;play and compatible with all the traditional task scheduling methods which beat supports.

Heroku really helped me get started. I then stated experimenting with running docker compose locally and finally was able to run my own kubernetes. You don&#x27;t need Heroku as you progress. It takes time. Be patient.

I vote for moving to Render!<p>My god I wish I could short Heroku stock.

The article doesn&#x27;t mention billing, I hope you weren&#x27;t billed twice? :o

Maybe Dokku us an option to self host and maintain Heroku code

I just hope you won&#x27;t be billed twice too

There&#x27;s been a lot of complaints about Heroku recently, and I think there will be a lot more.<p>I believe that Heroku under the Salesforce&#x27;s management is a different product that a lot of developers are used to.<p>It&#x27;s now in a Cash cow stage: <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Cash_cow" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Cash_cow</a>. Salesforce is trying to extract as much revenue as possible from the company they bought. And guess what - the real margins doesn&#x27;t come from SMBs and individuals - they come from larger companies with large contracts.<p>Heroku is no longer building a product for ordinary developer&#x27;s side-project, or for SMBs or early-stage startups. They are building a product for larger accounts paying 25k+&#x2F;month.<p>Customers that are expecting the old Heroku experience just need to look somewhere else. There are a lot of alternatives (such as render.com and many others). But the truth is, none of them is as good as Heroku. Luckily, these PaaS alternatives are getting better as we speak.<p>There are also different alternatives, such as <a href="https:&#x2F;&#x2F;stacktape.com" rel="nofollow">https:&#x2F;&#x2F;stacktape.com</a> (disclaimer: I&#x27;m a founder). For example, Stacktape offers PaaS-like experience directly within your own AWS account. It offers developer-friendly experience, and allows developers to deploy their apps entirely on their own, while at the same time offering the full power and flexibility of AWS.