Sequential and parallel execution of long-running shell commands

I am planning to use this for my homemade system-wide update scripts -- that updates via the OS package manager, Golang tools, Rust tools, OCaml tools, NPM-managed tools, editor configs (NeoVim headless updating of plugins as well), and others.<p>Seems like it has everything I need, the most crucial ones being separate queues &#x2F; worker pools (f.ex. I only want the tasks that might involve source compilation to be executed one after another on a separate queue, never in parallel).<p>Anybody knows a good alternative to `multitail`? Still haven&#x27;t found any other tool that can track the output of several commands at the same time on a screen that&#x27;s automatically split -- and subsequently the finished tasks get removed from view until only the last one remains, taking the entire screen, and then it says &quot;Done, press Q to quit&quot;.

What exactly is the is doing that you can’t do with pure bash? You can set up chains of commands in serial or in parallel, use job arrays, establish dependencies or other run conditions, hop into a backgrounded run, etc.

Pueue dumps the state of the queue to the disk as JSON every time the state changes, so when you have a lot of queued jobs this results in considerable disk io. I actually changed it to compress the state file via zstd which helped quite a bit but then eventually just moved on to running NATS [1] locally.<p>[1] <a href="https:&#x2F;&#x2F;nats.io&#x2F;" rel="nofollow">https:&#x2F;&#x2F;nats.io&#x2F;</a>

I usually just need to run one big interactive job on a server, in which case I just do<p>tmux my-command Ctrl-b d (detach) logout, go to bed, sleep, whatever log back into the server, tmux attach<p>Would there be an advantage to using this over that?

If you see following error logs,<p>$ pueue Error: Couldn&#x27;t find a configuration file. Did you start the daemon yet?<p>Run `pueued -d` first. I think this prompt should be printed on the screen the first time it runs, or automatically executed.

&quot;pueue follow &lt;task_id&gt;&quot; lets you see stdout or stderror of the specified task.<p>If one enqueues a single chain of tasks (no parallel tasks), is there a way to monitor stdout or stderror for the chain, without having to issue the follow command for each task at the time the task starts to run? This would provide better observability of what is running, as in a shell script with the tasks sequentially listed.

&gt; Pueue is considered feature-complete<p>Oh that is sweet sweet music to my ears!

This looks quite helpful actually. Thanks for sharing.

A similar tool I highly recommend: <a href="https:&#x2F;&#x2F;github.com&#x2F;justanhduc&#x2F;task-spooler">https:&#x2F;&#x2F;github.com&#x2F;justanhduc&#x2F;task-spooler</a><p>At first I thought it would just be a one-off tool I used for one of my projects, not until I discovered later that it has everything I need and became my daily driver ever since.

I&#x27;ve wanted to write something like this for years! For me it&#x27;s always long running rsync commands I want to chain.<p>I still might write one as it would be a fun way to play around with some low level code, but when I actually want to get things done I&#x27;ll be checking this out.

Under &quot;Features&quot;:<p>&gt;Pause&#x2F;resume tasks, when you need some processing power right NOW!<p>How is the pause and resume done?

Actually pretty sweet. `ps | grep` can be tricky to navigate sometimes. I know I&#x27;ve certainly grabbed the wrong process a few times.<p>I&#x27;m not exactly sure what advantage this has over managing `screen` sessions tho. Maybe it&#x27;s cleaner from a process tree perspective?

This looks awesome and the README seems to undersell it. Would it be possible for you to put code examples all over the readme so we can see what you’re talking about without needing to dig into the codebase?

“Pueue is not designed to be a programmable (scriptable) task scheduler&#x2F;executor.”<p>Any alternatives similar to Pueue with capabilities of above?

Ah, I wish this was available when I was using gnu parallel.

The title is unclear, it should include the tool name

I do all this with systemd user services.