Parallel Python

There are much more active projects that do the same thing as this.<p>- IPython parallel (<a href="https:&#x2F;&#x2F;ipyparallel.readthedocs.io&#x2F;en&#x2F;latest&#x2F;" rel="nofollow">https:&#x2F;&#x2F;ipyparallel.readthedocs.io&#x2F;en&#x2F;latest&#x2F;</a> ) is pretty much the same idea but under active development, supports both local processes, small clusters, cloud computng, an HPC environments (SGE style supercomputing schedulers).<p>- Joblib is a great tool for common embarrassingly parallel problems - run on many cores with a one liner (<a href="https:&#x2F;&#x2F;pythonhosted.org&#x2F;joblib&#x2F;index.html" rel="nofollow">https:&#x2F;&#x2F;pythonhosted.org&#x2F;joblib&#x2F;index.html</a>)<p>- Dask provides a graph-based approach to lasily building up task graphs and finding an optimal way to run them in parallel (<a href="http:&#x2F;&#x2F;dask.pydata.org&#x2F;" rel="nofollow">http:&#x2F;&#x2F;dask.pydata.org&#x2F;</a>)

By my estimate, this projects seems to be old &#x2F; stalled, not in active development. This notion is based on the lack of Python 3 support and the fact that majority of the activity in the forums happened in years 2006 - 2010.<p>EDIT: I was wrong, there is some activity in a different branch of the forum [1]. Last release almost two years ago.<p>[1] <a href="http:&#x2F;&#x2F;www.parallelpython.com&#x2F;component&#x2F;option,com_smf&#x2F;Itemid,29&#x2F;board,2.0" rel="nofollow">http:&#x2F;&#x2F;www.parallelpython.com&#x2F;component&#x2F;option,com_smf&#x2F;Itemi...</a>

From the FAQ:<p>Q) What Python versions are supported?<p>A) PP was tested with Python 2.3 - 2.7 on a variety of platforms.

Most fellow python people know this already, but native compiled C&#x2F;C++ code generally runs about 100x faster than interpreted CPython code for any kind of number crunching or computation. So like sonium also mentioned, definitely consider the C route also if you&#x27;re at all familiar with it and you&#x27;re having performance problems. Threading python wouldn&#x27;t generally get you as far (perhaps 4-6x speedup on an 8 core machine). Python&#x27;s C integration is very well thought out and lets you use the best of both languages, as well as gain a deep understanding of what&#x27;s going on under the hood in your python programs.

I&#x27;m a daily user of Python but the parallelization efforts really make me wonder. When things in Python are slow, its most likely something with native Python code (in contrast to e.g. Numpy). Wouldn&#x27;t it then make more sense to rewrite this in native C code which gives a speedup of two orders of magnitude instead of getting into the mess of paralellization which scales, in the best case, with number of cores?

<i>sigh</i> Why do all of these Python reimplementation projects target only Python 2.x?

I guess this is supposed to cure the GIL curse of threading in Python? i.e. the fact that only one thread can be running in a n interpreter process.<p>There is still the multiprocessing module which can spawn processes, so you can effectively run code in parallel. It&#x27;s pain to manage though.<p>I think in the end I am grateful for no parallel threading in Python as it forces me to either do things so that they can be run in naive-parallel, or to use things that have the concurrency solved out of Python.

Wouldn&#x27;t it make a lot of sense to just use Pyspark with RDDs? Latency would be relatively high, but it&#x27;d also bypass the GIL while also being more modern.

It seems to be Python&#x27;s &quot;multiprogramming&quot; module with some extra features.<p>Python&#x27;s parallelism options are &quot;share everything (but with only one thread computing at a time)&quot;, or &quot;share nothing&quot; in a different process on the same machine. If you&#x27;re parallelizing to get extra crunch power, neither is efficient.

I wish someone could port Julia&#x27;s parallelization notation to python. That&#x27;s the only Language that struck me as &quot;wow&quot;.

Expected redirect to golang.com

Needs IPC - not interested due to comm. bottleneck. Anyone got experience with Jython? How drop-in is it?