The multiprocessing module in Python

My only gripe is:<p>&gt; Since a new instance of Python VM is running the code, there is no GIL and you get parallelism running on multiple cores.<p>Has the potential to be misinterpreted by new Python devs. As I understand it, each <i>process</i> gets its own interpreter with its own GIL. So its not that there is no GIL, just that there are now 3 GILs each managing separate execution spaces.<p>Also, at a higher level of discussion, this is just a rehash of multi-<i>processing</i> vs. multi-<i>threading</i> so perhaps it would be more illuminating to start from there and move down to the library examples.<p>The casual reference to multi-<i>threading</i> in the quote from the POSIX fork definition also adds to potential reader confusion.

There is a nice hack for multiprocessing module to use for IO-bound code without having to fork processes:<p><pre><code> from multiprocessing.dummy import Pool as ThreadPool pool = ThreadPool(16) res = pool.map(one_arg_fun, [arg1, arg2, ...]) </code></pre> A 3 line IO parallelism speedup trick. Used it fetch stuff from multiple servers recently.

Some good alternatives, for when you just want quick and easy parallelization:<p>Joblib[0]: &#x27;Embarrassingly parallel for loops&#x27;. Basically just write a generator and get multicore processing on it. Pretty straightforward.<p>Multiprocess[1]: This is a fork of the multiprocessing module. The biggest benefit to me (the one time I used it) is that it&#x27;s easier to create shared data structures for multiprocessing (which I couldn&#x27;t do with multiprocessing.Pool). Last week I had to do a really long graph calculation that only needed to return a result for a very, very small fraction of the arguments. Joblib stored all of the null results as &#x27;None&#x27;, which tore through my RAM after billions of relatively fast calculations before crashing. But using a shared dictionary and multiprocess.Pool and imap_unordered, I was able to use mutlicore processing, adding items to the dict only if the right conditions were met, and discarding the &#x27;None&#x27;s. RAM use was very minimal.<p>[0]: <a href="https:&#x2F;&#x2F;pythonhosted.org&#x2F;joblib&#x2F;index.html" rel="nofollow">https:&#x2F;&#x2F;pythonhosted.org&#x2F;joblib&#x2F;index.html</a> [1]: <a href="https:&#x2F;&#x2F;pypi.python.org&#x2F;pypi&#x2F;multiprocess" rel="nofollow">https:&#x2F;&#x2F;pypi.python.org&#x2F;pypi&#x2F;multiprocess</a>

I used this module for a fractal rendering program, years ago. The APIS are (were?) a little annoying - last I checked, the relevant code had to be picklable. But it used all my CPUs and scaled just fine.

I had a really rough time finding practical&#x2F;real world examples of this stuff. After digging around and piecing together a bunch of tips&#x2F;guides&#x2F;pointers, I managed to get what I wanted working.<p>If you&#x27;re looking for some real world examples of using multiprocessing (v3), check here: <a href="https:&#x2F;&#x2F;github.com&#x2F;dpgailey&#x2F;asteria&#x2F;blob&#x2F;master&#x2F;asteria-v3&#x2F;client.py" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;dpgailey&#x2F;asteria&#x2F;blob&#x2F;master&#x2F;asteria-v3&#x2F;c...</a>

Nice job hijacking my browser back button...

If you need multi-core for performance, first you should rewrite in a compiled language instead of Python. Java&#x2F;C&#x2F;C++&#x2F;Go&#x2F;FORTRAN&#x2F;whatever will run 10 to 20 times faster before even worrying about running multi-core.