Mpire: A Python package for easier and faster multiprocessing

I&#x27;ve spent a lot of time writing and debugging multiprocessing code, so a few thoughts, besides the general idea that this looks good and I&#x27;m excited to try it:<p>- automatic restarting of workers after N task is very nice, I have had to hack that into places before because of (unresolveable) memory leaks in application code<p>- is there a way to attach a debugger to one of the workers? That would be really useful, though I appreciate the automatic reporting of the failing args (also hack that in all the time)<p>- often, the reason a whole set of jobs is not making any progress is because of thundering herd on reading files (god forbid over NFS). It would be lovely to detect that using lsof or something similar<p>- it would also be extremely convenient to have an option that handles a Python MemoryError and scales down the parallelism in that case; this is quite difficult but would help a lot since I often have to run a &quot;test job&quot; to see how much parallelism I can actually use<p>- I didn&#x27;t see the library use threadpoolctl anywhere; would it be possible to make that part of the interface so we can limit thread parallelism from OpenMP&#x2F;BLAS&#x2F;MKL when multiprocessing? This also often causes core thrashing<p>Sorry for all the asks, and feel free to push back to keep the interface clean. I will give the library a try regardless.

Why does everyone compare against `multiprocessing` when `concurrent.futures` (<a href="https:&#x2F;&#x2F;docs.python.org&#x2F;3&#x2F;library&#x2F;concurrent.futures.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;docs.python.org&#x2F;3&#x2F;library&#x2F;concurrent.futures.html</a>) has been a part of the standard library for 11 years. It&#x27;s a much improved API and the are _almost_ no reasons to use `multiprocessing` any more.

The particular pain point of multiprocessing in python for me has been the limitations of the serializer. To that end, multiprocess, the replacement by the dill team, has been useful as a drop in replacement, but I&#x27;m still looking for better alternatives. This seems to support dill as an optional serializer so I&#x27;ll take a look!

Multiprocessing is great as a first pass parallelization but I&#x27;ve found that debugging it to be very hard, especially for junior employees.<p>It seems much easier to follow when you can push everything to horizontally scaled single processes for languages like Python.

Another good library for concurrency and parallel tasks is futureproof:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;yeraydiazdiaz&#x2F;futureproof">https:&#x2F;&#x2F;github.com&#x2F;yeraydiazdiaz&#x2F;futureproof</a><p>&gt; concurrent.futures is amazing, but it&#x27;s got some sharp edges that have bit me many times in the past.<p>&gt; Futureproof is a thin wrapper around it addressing some of these problems and adding some usability features.

I often use lox for this sort of thing. It can use threads or processes, and has a very ergonomic api.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;BrianPugh&#x2F;lox">https:&#x2F;&#x2F;github.com&#x2F;BrianPugh&#x2F;lox</a>

Some potential issues about Python multiprocessing <a href="https:&#x2F;&#x2F;blog.mapotofu.org&#x2F;blogs&#x2F;python-multiprocessing&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;blog.mapotofu.org&#x2F;blogs&#x2F;python-multiprocessing&#x2F;</a>. COW is quite tricky. BTW, most of the related official Python docs doesn’t mention the usage under ‘spawn’.

I&#x27;ve written a very tiny multiprocessing pipeline in Python. It&#x27;s documented.<p>I&#x27;ve actually never made use of it but at the time I got a bit obsessed and wanted to write it. It does seem to work as expected.<p>Is highly hackable as it is only a single file and a couple of classes.<p>Maybe is useful to someone, here&#x27;s the link: <a href="https:&#x2F;&#x2F;github.com&#x2F;lliendo&#x2F;SimplePipeline">https:&#x2F;&#x2F;github.com&#x2F;lliendo&#x2F;SimplePipeline</a>

Very cool.<p>Except I&#x27;m a bit concerned that it might have <i>too many</i> features. E.g. rendering of progress bars and such. This should really be in a separate package and not referenced from this package.<p>The multiprocessing module might not be great, but at least the maintainers have always been careful about feature creep.

How is this different from ray.io?

Why has Python never added something like We workers&#x2F;isolates? That seems like the obvious thing to do but they only have multiprocess hacks.

i see that all the benchmarks have processpoolexecutor either equal to or outperforming multiprocessing and i do not find this to be the case for about 90% of my cases.<p>also a niche question, is this able to overcome the inability to pickle a function within another function to multiprocess it?<p>i&#x27;m still excited to try this as i haven&#x27;t heard of it and good multiprocessing is hard to come by.

Why is this faster than the stdlib? What does it do to achieve better performance?

Would anyone be in a position to comment on how this compares to Dask?

Always dreamed of multiprocessing with tqdm, this is great

Ah darn, was hoping for some MPI Python interface.