While this is all nice, I have yet to find a package system that is as straight forward to use as NPM.<p>Don't get me wrong, I love Python and I'm not so much a fan of NodeJS, as Python is so much better designed than JavaScript, even in time of ES6. But their NPM is really well designed. NPM alone gives NodeJS a competitve advantage over many other popular languages including Ruby (Gem) and Python (PIP).<p>For example, why does Python packaging need such a complicated bootstrapping process? Install ensure_pip, then the real pip, and on top of that virtualenv. NodeJS just ships with NPM. Done.<p>Any why do we need to create and manage a "virtualenv"? Why do you have to go through extra hassle just to avoid global installs? NPM just creates a local sub directory node_modules and automatically installs and removes from that. Global installs happen only if you really want it to (npm -g). But you usually don't.<p>The only central place at NPM is the package cache so it doesn't download things twice if you have two separate projects with similar dependencies. In the Python/PIP world, everything is central and even today you still have to accept extra hassle to get a sane, local setup.
Python packaging is great.<p>What's a bother is - as that article mentions - end-user deployments.<p>Now, I sort of have that down to a tee by using cx_freeze and Inno Setup, such that I can distribute my software to my end-users and all they have to do is click on a Setup.exe and my applications install just like any other Windows program.<p>I can do a similar thing on the Mac, but it's still a pain.<p>The main trouble with packaging Python applications up, is an apparent assumption (at least, that's the impression I have gotten over years of figuring out how to distribute my applications), that Python programs should all be open source - so for example, distutils assumes that the author wants his code to be readable by everyone who wishes to.<p>I wanted to be able to produce full applications and somehow compile and package them as in for e example, a C/C++ application.<p>The closest I can get to that currently, is cx_freeze, which isn't compilation, merely turning the Python code into bytecode.<p>Nuitka is showing lots of promise, but as yet it still cannot compile my applications into something usable (my really big Qt4 GUI applications either won't compile or there are lots of problems cropping up even if they do manage to run).<p>Here's hoping end-user deployment gets much easier than it is now, someday!
In the past 12 hours, I started a new Python project. I can't agree with the sentiment - the most I can say is that Python packaging isn't terrible now.<p>My intention was to create a Python 3 package using best-practices, a nice division of labour by process to avoid threading problems, etc.<p>So far I've run into the following issues regarding packaging:<p>Getting started - there's a glut of documentation, much of it contradictory, much of it out-of-date. Simply google "Python packaging" and you'll have to pick between 4 sources of info on the first page, all of which look official.<p>The tools - this one was a new one for me; I normally use virtualenvs for development, but vaguely remembered it being bundled in Python 3. Turns out it's not virtualenv but "pyvenv". Beyond the confusion of the name change, there's also a tool called "pyenv" muddying the waters.<p>The specifics - I started with <a href="https://packaging.python.org/" rel="nofollow">https://packaging.python.org/</a> and was struck by how ambiguous the instructions were. Doing something as simple as declaring your package's version has 7 different options: <a href="https://packaging.python.org/single_source_version/" rel="nofollow">https://packaging.python.org/single_source_version/</a>. In the end, I got the most help from reading the guide's sample project... I only have to wonder about the licensing implications of that.<p>The infrastructure - because I was creating a project with multiple executables, I needed a way to start one from the other. Despite knowing that setuptools is responsible for generating the executables, I quickly found that there's just no way to find those executable paths programmatically.<p>Library choice - I needed an inotify library. I searched for "inotify", "notify" and "watch" on Pypi but had little way of distinguishing the results - the "weight" column is useless as compared to a rating system or even download count. Thankfully, I remembered about awesome-python.com which bridges the gap.<p>All told, it took me several hours to bootstrap the project, not even including things like testing. I'd say Python packaging has a way to go before I'd call it "good".
Is it?<p>On Debian there is Debian Python, and Debian pip. Then there is easy_install to install own pip, and that's different. virtualenv comes on top of all. Of course, there is Debian's Python packages, too.<p>On MacOS X with Macports, there is Apple's Python and Macports' Pythons, all with different pips. Of course, there is Macports' Python packages, too, for different Pythons.
No matter what small tooling changes come about, the Python packaging ecosystem is fundamentally flawed, and will always be more complicated and less useful than others such as Maven (yes, Maven).<p>The primary reason is that setup.py, etc., have you download/manage one set of dependencies for <i>each project</i> you're working on rather than simply housing all dependencies in a common folder, separated by version, as Maven does. It's pointlessly redundant. But the real pain comes when you start to install things. Since in Python land, dependencies for all tools are typically stored in the same folder, you very easily and quickly hit version conflicts since the dependencies are not really segregated by version. This leads to brutal bugs that creep on you at runtime when you realize, too late, that some tool you installed has dependencies that conflict with another.<p>What's worse, complex Python projects like OpenStack may have numerous dependencies which all in turn have differing and conflicting dependency versions of their own. Trying to keep them all in sync is a ridiculous exercise in time wasting and pointlessness, and problems caused by version mismatches are common.<p>The brilliant solution (sarcasm here), of course, is to create virtual environments with separate copies of every dependency for every tool we run. Again, pointlessly redundant and a pain to manage.<p>Is there a better way? Obviously. Use a single common repo with dependencies structurally segregated by version (ie: different folders), as is the way of doing things in Java-land. I know that Java-land stuff is bad and we're all supposed to hate it. But I've always failed to understand why developers in other language ecosystems continue to create tools in a vacuum, as if roadmaps for solving these problems don't already exist elsewhere. Look beyond your bubble. Hopefully one day the Python folks will catch on.
<p><pre><code> The real takeaway here though, is that although it’s still not
perfect, other languages are no longer doing appreciably
better.
</code></pre>
Not sure how accurate is this.<p>At least not true in the context of OCaml/OPAM.<p><pre><code> As always, I’m sure none of this applies to Rust and Cargo is
basically perfect, but that doesn’t matter, because nobody
reading this is actually using Rust.
</code></pre>
So it seems that those using rust don't read HN:<p><a href="https://github.com/servo/servo" rel="nofollow">https://github.com/servo/servo</a><p><a href="https://github.com/dropbox/rust-brotli" rel="nofollow">https://github.com/dropbox/rust-brotli</a><p>--<p>Bottom line: Bash everyone else arbitrarily and claim oneself to be the best!
> python setup.py sdist bdist_wheel<p>Wheels suck.<p>By using the bdist mechanism they make it impossible to build a single cross-platform, cross-Python package that just works. Wheels can't run 2to3 at installation so guess what? You have to make separate packages for Python 2 and 3.<p>There should be an easy way to have a pure Python package that "just works" on Python 2 and 3, 32-bit and 64-bit platforms without having to build separate packages. Right now that can only be done with the third-party Setuptools using sdist and a traditional setup.py. Distutils is still a half implemented packaging system. I wouldn't say that Python packaging is a solved thing.
No discussion of python packaging without mention of conda is complete. It's as good as "apt-get" for Python (and not just for Python, but ... mostly for Python).<p><a href="http://conda.pydata.org/miniconda.html" rel="nofollow">http://conda.pydata.org/miniconda.html</a>
For a long time, PyPi intermittently reported extremely exaggerated download statistics and sometimes never even reported any statistics. Then they got bored of trying to fix it and removed it altogether, saying they want to concentrate on warehouse. I believe currently the only way to get any kind of statistics is to perform a query on Google's Big. Any attempt to talk about this issue results in the thread being locked and offended replies, and while I understand what it's like from the developer's point of view, the current state of things is completely unreasonable.
It may well be. But one problem is all the references to all the various old ways that a first time deployer has to wade through. It took me quite a while to figure out what's current and what should be ignored, as well as what's old but still needs to be done, on a small project that I did to learn exactly this.
Personal perspective of someone who's only used python 4 years and mostly in an linux ops capacity is that the packaging has always been pleasant to use and the few times I've had to install a node/ruby app as an end user have been comparatively horrible.
This is the first time I've heard of ensurepip. It looks cool, but I'll wait until it becomes common knowledge before declaring that "Python packaging is good now", but I definitely agree that it's improving. I'll have to update my StackOverflow answer now to mention this. <a href="http://stackoverflow.com/a/14753678/247696" rel="nofollow">http://stackoverflow.com/a/14753678/247696</a>
"From 2008 to 2012 or so, Python packaging was a total mess."<p>Some things have improved, but fundamentally there really is not much difference in creating a package now vs. then.<p>A certain xkcd comes to mind.
> <i>There should be tools that help you write and update a setup.py. Or a setup.python.json or something, so you don’t actually need to write code just to ship some metadata.</i><p>There are some out there, e.g. OpenStack uses <a href="https://pypi.python.org/pypi/pbr" rel="nofollow">https://pypi.python.org/pypi/pbr</a> - it was pretty painful around the 1.0 transition when no two libraries seemed to be using compatible versions, but it hasn't caused me any trouble in quite a while now.<p>I'm more concerned by the lack of a (non-greedy) dependency solver in pip.
While I agree with the fact that it isn't terrible as it once was; I personally can't say pip is really starting to look any better until the following is solved:<p><a href="https://github.com/pypa/pip/issues/988" rel="nofollow">https://github.com/pypa/pip/issues/988</a><p>''Pip needs a dependency resolver''<p>Without such a thing you really don't quite have a way to ensure what you are requesting to be installed actually is a valid set of things.
NPM did it right. PIP got it wrong. Requiring the same version of a library per application is really a bad idea.<p>It creates a lot of work to align your dependencies.
My main peeve is that distutils2 died, and with it, the idea of putting package metadata (author, version, dependencies) in `setup.cfg` instead of `setup.py`.<p>It's bad enough that I have a boilerplate `setup.py` that just does `import ConfigParser` and blasts the entire `[setup.py]` section into `setuptools.setup()` as `<i></i>kwargs`. I copy and paste this boilerplate into every new project that I start.<p>Oh and while we're ranting about Python packaging, let me rant about tox, which is so-very-useful but also so-very-dangerous: IF YOU (SLIGHTLY) MISCONFIGURE YOUR TOX.INI FILE, IT WILL DELETE YOUR ENTIRE PROJECT DIRECTORY! [1]<p>I suppose I am getting too old and cranky for this stuff. :)<p>[1] <a href="https://bitbucket.org/hpk42/tox/issues/352/tox-deleted-all-of-my-code" rel="nofollow">https://bitbucket.org/hpk42/tox/issues/352/tox-deleted-all-o...</a>
The article say that with any recent python you can get started with a one liner:<p><pre><code> Python Core started distributing the ensurepip
module along with both Python 2.7 and 3.3 ...
$ python -m ensurepip --user
</code></pre>
Tried here with python-2.7.11-3 and just got a traceback error.<p>Turns out according to one of the links given by the
article[1] that this feature was actually not backported
to these older versions of Python, apparently to encourage
people to upgrade to new Python. This is unfortunate.<p>[1] <a href="https://www.python.org/dev/peps/pep-0453/#including-ensurepip-in-python-2-7-and-3-3" rel="nofollow">https://www.python.org/dev/peps/pep-0453/#including-ensurepi...</a>
<p><pre><code> $ python -m ensurepip --user
$ python -m pip install --user --upgrade pip
$ python -m pip install --user --upgrade virtualenv
</code></pre>
> ...<p>> From here on out, now the world is your oyster; you can pip install to your heart’s content, and you probably won’t even need to compile any C for most packages. These instructions don’t depend on Python version, either: as long as it’s up-to-date, the same steps work on Python 2, Python 3,<p>Nit: I suppose installing virtualenv works in 2 and 3, but hasn't venv in 3 made that unnecessary?
The last three (cross-platform, Python 2 and Python 3) packages I published to PyPI have been Ctypes/Cython wrappers around Rust binaries. Yes, I'm a sample size of 1, but whatever, never let reality get in the way of your rhetorical blogpost, I suppose.
A while back, I wrote some of my thoughts on Python packaging. I got excited seeing this title, but reading the article, I remain unconvinced. The whole thing is still way too confusing, and it's still not nearly as intuitive (or teachable) as "npm install ___".
I don't know what this article is talking about. What is wrong with apt-get install python-something to install the python package called something? Or if it's not in the repositories, you checkout the library to a subdirectory of your project and call `import something` just like you would when apt-getting it.<p>So honest question, what's this easy_install stuff? Apt has been around since the nineties, just like python, and didn't change much as far as I know.