I've done a setup similar to this in the past and really enjoy it. If you ever find yourself writing logic in your Resque/Sidekiq jobs that has to do with manually managing a data structure in Redis with regard to your jobs, your app is screaming for a language with better tools for modeling complex workflows.<p>A lot of people are hitting on the performance improvements (25 Sidekiq threads/workers versus 10k-100k+ Elixir/Erlang processes) but when I find myself doing this it's actually because my background process flows are really complex. The drastic increase in concurrency is just the cherry on top.<p>Background work is asynchronous by nature and often times you have dependencies between jobs, groups/batches jobs and also need robust error handling/recovery. This is where Elixir/Erlang really shines and using Actors & OTP to model these complex flows is the huge win in my experience. Sidekiq Pro offers support for batches which covers some of this, but ultimately Elixir/Erlang processes are just a huge improvement in modeling such systems.<p>A concrete example is sending off 10 jobs. When all 10 are done, move on to the next step. Each of these 10 jobs can also send data to the waiting step which it can then use as arguments. If any job should fail N times, cancel and rollback all of the jobs and try again. After the next step, split those 10 jobs into two groups of 5 and do those 5 sequentially. So on and so forth. Try doing this in Ruby. You'll likely have to manage your own data structures in Redis. In Elixir you just use GenServers and sometimes ets tables. It's all handled in the language itself, rather than external data stores.<p>Sidekiq is probably suitable for 90% of Ruby apps doing basic fire and forget and batching with Sidekiq Pro. But if your domain has seriously complicated background processing, use a tool more suited for that.
Best thing I ever learned - probably 15 years ago at this point - is just use the right tool for the job.<p>We're a mobile game company, and we use C, C#, Objective C, Java, Python, Go, Erlang, JavaScript, Lua, and more where they make sense.<p>I love language talk and debates (and hate the language hate) here on HN. But if you're only using a single language in production, you're likely not that big, or more likely have a lot to gain by reworking certain bits with other technologies where they make sense.
Why are you adding an extra level of abstraction in between Rails and Sidekiq? Why not just have Rails push to redis queue and elixir process finishes the job of sending email or doing what it has to do. Why have Sidekiq run at all? It seems adding multiple layers as such will make it harder to understand, debug and maintain in future.<p>EDIT: Also your example of polling on Redis queue using elixir is very inefficient and makes your article not so credible in my eyes or eyes of other good software architects. Redis is meant for pub sub and not polling.
We've done similar things at work (using a Redis queue and a Go co-process), but I actually really like the idea of using a fairly common and well-liked delayed job interface in Rails and swapping out the worker underneath. Cool idea.<p>One minor suggestion for the author (post doesn't have a comment section). `:timer.sleep(10)` is explained as "sleep for 10 seconds", but sleep takes millis as an argument (<a href="http://www.erlang.org/doc/man/timer.html#sleep-1" rel="nofollow">http://www.erlang.org/doc/man/timer.html#sleep-1</a>).
Excuse my poor understanding of Ruby's GIL, but is the need for an Elixir worker because Ruby is by default single-threaded, and therefore actual default concurrent Sidekiq workers are not possible? Or this a problem scaled up to the degree that we're talking ~thousands of jobs per second that Ruby's limitations start to show?
Polyglot web IDE that allows you to mix Python, Javascript, R, Ruby, Scala, Java: <a href="http://BeakerNotebook.com" rel="nofollow">http://BeakerNotebook.com</a> work with them all in the same page. As a corollary, you can work with Python2 and Python3 in the same notebook, and share data.<p>I totally agree that each language has its strength and each facet of your problem may best be solved in a different language.
Yeah, be polyglot by speaking American English and British English! I mean, Ruby and Elixir!<p><a href="http://satwcomic.com/multi-language/moby-is-a-dick" rel="nofollow">http://satwcomic.com/multi-language/moby-is-a-dick</a>