NumPy developer here. First, I agree that the "the GIL is not an issue" is an annoying meme. It can be an issue, and when it is, it is annoying as it complicates some architectures. I don't think those architectures are as common as people usually think.<p>A few remarks: I would qualify the GIL as a tradeoff rather than a mistake. You loose CPU-bound parallelism with threading, but you gain easy to write C extensions (well, relatively speaking). I think this point is critical to the existence of something like NumPy (which is unrivalled in general PL AFAIK).<p>If you need to share a lot of data (a big numpy array), then you can use mmap arrays, there is no serializing/desarializing cost, and that's efficient. See for example some presentations by O. Grisel from scikits-learn (<a href="https://www.youtube.com/user/EnthoughtMedia/" rel="nofollow">https://www.youtube.com/user/EnthoughtMedia/</a> -- disclaimer, I work for Enthought, but all those video happened at scipy 2013).<p>IMO, the only significant niche where this is an issue is reaching peak performance on a single machine, but python is not really appropriate anyway -- that's where you need fortran/C/C++, and even scala/java/haskell are quite unheard off there.
All of these problems can be addressed using inter-process shared memory. Shared memory support is built into multiprocessing [1].<p>Now I agree it might not be convenient, but that's a matter of libraries. This post would be much more constructive if it was speculation on what such a library could look like, instead of pretending that concurrency "doesn't work" in single-threaded Unix processes.<p>1. <a href="http://docs.python.org/2/library/multiprocessing.html#module-multiprocessing.sharedctypes" rel="nofollow">http://docs.python.org/2/library/multiprocessing.html#module...</a>
Nice post, Chris, and it's definitely a problem that many people are confused about (as the comments in this thread show). People who think the GIL is a problem either have no idea what they're talking about, or they really know what they're talking about. People who don't think the GIL is a problem either have no idea what they're talking about, or they really know what they're talking about. :-p<p>One of the lesser-appreciated facts about the GIL is that <i>it is an implementation detail of CPython</i>. That is, it is entirely possible to implement a Python that does not have process-level globals and C statics. There is no <i>structural</i> reason why a C or C++ implementation of Python <i>needs</i> to have a GIL. It's just a legacy of the implementation that has stayed around for a long, long time.<p>For instance, Trent Nelson has done some work to show that you can move to thread-local storage for the free lists and whatnot, and get nice multicore scaling, even with the existing CPython codebase[1]. There are still concurrency primitives that the language would need to offer at the C API level to manage the merger of these thread-local objects, but it's a whole lot better than only being able to use a single core in modern days.<p>Fortunately I mostly get to work in the scientific field with (mostly) data parallel problems.<p>[1] <a href="https://speakerdeck.com/trent/parallelizing-the-python-interpreter-an-alternate-approach-to-async" rel="nofollow">https://speakerdeck.com/trent/parallelizing-the-python-inter...</a>
<p><pre><code> But fundamentally, the GIL prevents Python from being used as a systems language.
</code></pre>
This is only true when severely restricting the definition of a systems language. The vast majority of command line utilities you'll find in a typical operating system are often primarily I/O-bound or do not have critical performance requirements.<p>As such, I'd only be willing to agree with the author's statement for a very specific subset of systems programming.<p>For example, I've worked on a packaging system written in Python for the last five years or so. The package system is primarily I/O-bound the vast majority of the time (waiting on disks or network), and almost all significant performance wins (some as much as 80-90%) have come from algorithmic improvements rather than rewriting portions in C (very little is written in C).<p>As one of my colleagues is fond of saying (paraphrasing), "doing less work is the best way to go faster".<p>It also ignores the fact that depending on the problem space involved, there may be readily available solutions that provide excellent performance that don't involve threading (e.g. the multiprocessing module, shared memory, etc.).
This feels to me more like a critique of fork() and the unixy process-oriented parallelism model than a critique of Python. Of course, the author mentions this as a caveat, but it makes me wonder if the blame is being laid where it should be (and also whether in some scenarios like this, you should really just build applications the way applications are normally built for a platform, no matter how much you dislike it)
He has a valid point. There are certain types of workloads which don't scale well on a single machine when writing pure python code. If your workloads happens to need to get more cpu performance out of a single machine in pure python and isn't easily to parallelize using processes, python's not the best choice. Personally I feel like go is good for this use case because I consider writing my own object lifecycle code a pita. Others will only use multithreaded c code, something I'm not eager to touch.<p>I do however feel the need to make a couple clarifications:<p>If you fork a process, you don't necessarily duplicate memory. Yay for COW. Fork twice and you've got fanout with very little extra memory. Threading works fantastically for I/O. C libraries can release the GIL during processing. So if you have to do something computationally expensive, you can let another python thread run.<p>I rarely feel restricted by python, and when I do I usually find that other aspects make up for it. I find it makes a decent systems programming language (although not as good as C). The os library and the ctypes library give you the capability to do most (all?) system level tasks.<p>I tend to prefer go over python when library support isn't an issue, but that's more due to it's type system and channel primitives. As someone who's pushed python to it's limits as a systems programming language I'm pretty comfortable dropping down to C isn't <i>often</i> called for, and writing in java/scala isn't ever required.
This is just my opinion, but it's served me well in the 8 or so years I've done Python development:<p>Why are you doing compute intensive work in Python? Python is not well optimized for doing compute intensive work. It's made some strides over the years, being able to do limited bytecode optimization and multiprocessing, but it's not a high performance computation language.<p>However, it is a fantastic glue language. Write your compute intensive portions in C, and use Python to glue the portions of C together. You can even do your thread generation in Python, release the GIL during your C execution, and you don't even have to worry about the complicated process of spinning up threads in your C.<p>In other words, use the language like it was designed to be used.
Timely and related talk given today at EuroPython conference about concurrency in Python and why (and when) GIL doesn't matter: <a href="http://www.youtube.com/watch?v=b9vTUZYmtiE" rel="nofollow">http://www.youtube.com/watch?v=b9vTUZYmtiE</a>
> Memory duplication has a relatively simple solution, namely using external cache such as redis. But the thundering herds problem remains. At time t=0, each process receives a request for f(new input). Each process looks in the cache, finds it empty, and begins computing f(new input). As a result every single process is blocked.<p>this is incorrect. The processes coordinate on a lock held in redis itself.<p>This solution is available right now using the Redis backend in dogpile.cache (of which I am the author): <a href="https://dogpilecache.readthedocs.org/en/latest/usage.html" rel="nofollow">https://dogpilecache.readthedocs.org/en/latest/usage.html</a>
no mention of gevent, celery, etc. As someone who runs thousands of concurrent tasks in a mix of process/gevent (one UNIX process for each 100 greenlets across 48 cores on two boxes), I find the OP's toliling rather misguided.
<i>The standard Python workaround to the GIL is multiprocessing. Multiprocessing is basically a library which spins up a distributed system running locally - it forks your process, and runs workers in the forks. The parent process then communicates with the child processes via unix pipes, TCP, or some such method, allowing multiple cores to be used.</i><p>Multiprocessing is <i>the way</i> to do parallelism. Deviating from that should be an exception -- for example, shared memory maps could be used to transfer select data objects instantaneously between the processes instead of serializing/deserializing over a pipe, and only those while still retaining separate process images. Threads were practically invented as a compensation for systems with heavy process image overhead.<p>I think Python is very Unix in this regard. And that's not a bad thing per se. Unix and Linux can do multiprocessing very efficiently.
How can the GIL, which is restricted to one process, hinder concurrency? It can only impact 1 single form of concurrency, threading.<p>Why use threads?<p>Noone does parallel compute on CPUs these days, not since GPGPUs rocked up almost 5 years ago (and we often use python as the host language, thanks pyCuda!)<p>Parallel IO then? Well, except that async IO is often far more resource efficient (at the cost of complexity though).<p>Threading is dead(-ish) because its hard to write, hard to test and expensive to get right.<p>Concurrency in python is very much alive though.
> The implementation would also likely be considerably more complicated than the 160 linues of code that the Spray Cache uses.<p>Not likely, using Twisted deferreds and a sane cache-wrapper with herd awareness -- you probably want this regardless of long-running cachables.<p>Of course, Python 3.2 also has a futures [thread or process] builtin, if that's your thing.<p>10-20 lines of code.
I don't get how having multiple copies of a quote hanging around in memory is a big problem. It's probably less than 100 bytes total (symbol + side + price) and probably significantly smaller than the size of the state associated with the "statistics process."
Regardless of other arguments, it basically means that threads in python become only a way of thinking about a problem rather than a way yo utilise hardware fully. It's a shame.
The author appears to not be aware of where the big leagues are. Additionally, as is becoming a recurring theme here, he doesn't know about PyPy nor Twisted. This is continually disappointing.