I'd like to point out that the Python standard library offers an abstraction over threads and processes that simplifies the kind of concurrent work described in the article: <a href="https://docs.python.org/dev/library/concurrent.futures.html" rel="nofollow">https://docs.python.org/dev/library/concurrent.futures.html</a><p>You can write the threaded example as:<p><pre><code> import concurrent.futures
import itertools
import random
def generate_random(count):
return [random.random() for _ in range(count)]
if __name__ == "__main__":
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
executor.submit(generate_random, 10000000)
executor.submit(generate_random, 10000000)
# I guess we don't care about the results...
</code></pre>
Changing this to use multiple processes instead of multiple threads is just a matter of s/ThreadPoolExecutor/ProcessPoolExecutor.<p>You can also write this more idiomatically (and collect the combined results) as:<p><pre><code> if __name__ == "__main__":
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
out_list = list(
executor.map(lambda _: random.random(), range(20000000)))
</code></pre>
In this example case, this will be quite a bit slower because the work item (in this case generating a single random number) is trivial compared to the overhead of maintaining a work queue of 200000000 items - but in a more typical case where the work takes more than a millisecond then it is better to let the executor manage the division of labour.
This example is not too realistic and just narrows it down to the case where a job can be divided into isolated tasks with no shared data/state.<p>Often times threads need to update shared dict/list etc... With multiprocessing this cannot be done. You can use a Queue for this but it's horribly inefficient.<p>Generally speaking if you need performance and Python is not meeting the requirements then you are better off using another language.
For the every day when I want to make embarrassingly parallel operations in Python go fast I find joblib to be a pretty good solution. It doesn't work for everything, but it's quick and simple where it does work.<p><a href="https://pythonhosted.org/joblib/" rel="nofollow">https://pythonhosted.org/joblib/</a>
I've had good success using Celery to parallelize tasks/jobs in python.<p>www.celeryproject.org<p>Also, it has a very nice concept called canvas that allows you to chain/combine the data/results of different tasks together.<p>It also allows you to switch out different implementations of the communication infrastructure that Celery uses to communicate and dish-out tasks.
For python developers who dislike the continued existence of the GIL in a multicore world, and who feel that multiprocessing is a poor response given the existence proofs of IronPython and Jython as non-GIL interpreter implementations, please consider moving to Julia.<p>Julia addresses nearly all the problems I've found with Python over the years, including poor performance, poor threading support on multicore machines, integration with C libraries, etc. I was a big adherent of Python but as machines got more capable, the ongoing resistence to solving the GIL problem (which IronPython demonstrated can be done with reasonable impact on serial performance) I could not continue using the language except for legacy applications.
Have you seen there is an error in the code for the threading part?<p>the right way if you want to use a thread is
thread = threading.Thread(target=CALLABLE, args=ARGS)<p>and not<p>thread = threading.Thread(target=CALLABLE(ARGS))
For the example task we could use the multiprocessing Pool and (the undocumented) ThreadPool.<p>This implements the worker pool logic already so we don't have to.