People always get up in arms about this, but as someone who has used Python as her daily driver for years it's really... never been this serious of an issue for me?<p>I have used virtualenv/venv and pip to install dependencies for years and years, since I was a teen hacking around with Python. Packaging files with setup.py doesn't really seem that hard. I've published a few packages on pypi for my own personal use and it's not been too frustrating.<p>A lot of the issues people have with Python packaging seem like they can get replaced with a couple shell aliases. Dependency hell with too many dependencies becomes unruly in any package manager I've tried.<p>Is the "silent majority" just productive with the status quo and getting work done with Python behind the scenes? Why is my experience apparently so atypical?
pip-tools is almost never mentioned because it's boring but great. I always default to it.<p><a href="https://github.com/jazzband/pip-tools" rel="nofollow">https://github.com/jazzband/pip-tools</a>
Every attempt to solve this problem in Python seems to eventually end up in a pretty terrible place. Pipenv got off to a great start but got slower and slower to the point that it was more painful to use than not. Poetry (which is still my preferred option) started off with something seemingly beautifully thought through, and very fast too. But after only a few version updates, it seems to be hitting the same problems Pipenv did. On one project I was working on recently I managed to screw up the Poetry.lock file, so I ran `poetry lock` and it took 18 minutes. I still have high hopes for poetry, but I spend way more time trying to work around its shortcomings now (v1.0.5) than I did when it was at version 0.10.0 two years ago.
The missing ingredient to really, REALLY solve these problems once and for all is an authoritative decision to switch package formats and run the whole dependency resolution stack by the core python language contributor team.<p>I get backwards compatibility and open-source governance and bla-bla, but the reality is that this cannot be done by a third-party library author and needs to become part of the core stack, including proper support rather than just shipping a tool which covers 90% of cases. It's crazy that apart from venv and pip, nothing else comes with python and you're left on your own.<p>npm + the registry is part of node<p>apt-get + registry is part of a normal linux distro<p>budnler comes with ruby<p>This is a solved problem elsewhere. What we lack is a fully-supported, agreed upon, working DEFAULT choice, so people don't have to make their own choices. I don't know if not having that DEFAULT is a function of how the python community thinks or its diversity, but it's painful to watch. I've almost given up myself and seen many newcomers give up because of a trivial problem like this.
Dependency management can be pretty overwhelming for a lot of people entering Python. This is especially true in the data science realm, where many don't have a SWE background. Even after you have selected a tool, it can be easy to use it in a poor way. I have recently written a short article on how I use conda in a disciplined way to manage dependencies safely: <a href="https://haveagreatdata.com/posts/data-science-python-dependency-management/" rel="nofollow">https://haveagreatdata.com/posts/data-science-python-depende...</a>
> Pipenv or poetry?<p>If you used pipenv for a complex project with huge dependency tree, or used it for a long time, you definitely run into a blocker issue with it. That is the worst package manager of all, and probably the reason why Python has such a bad reputation in this area. It's because it's fundamentals are terrible.<p>Just go with Poetry. It's very stable, easy to use, has a superior dependency resolver and way faster than Pipenv.
If you're interested in the technical issues behind Python packaging, a recent Podcast.__init__ episode features three people working on improving Pip's dependency resolution algorithm. My use cases are simple enough that I've gotten by for years just using pip and venv with requirements.txt files, but it was still fascinating to listen to how package management is approached in more complex situations.<p>Dependency Management in Pip's Resolver: <a href="https://www.pythonpodcast.com/pip-resolver-dependency-management-episode-264/" rel="nofollow">https://www.pythonpodcast.com/pip-resolver-dependency-manage...</a>
I've spent far too many hours fighting with these tools in two completely different scenarios<p>* Developing and deploying production Python solutions<p>* Helping beginners run their first script<p>While it's great for beginners to use the same tools that are used in industry, I strongly believe that the problem nearly all of these tools face is that they can't decide whether they want to _manage_ complexity or _hide_ complexity.<p>You can't do both.<p>Some of them do a fairly good job at managing complexity. None of them do a good job of hiding it. The dream of getting Python to "just work" on any OS is close to impossible (online tools like repl.it are the closest I've found but introduce their own limtiations). I recently saw a place force their beginner students onto Conda in Docker because getting people started with Conda was too hard. If you're battling with the complexity of your current layer of abstraction, sometimes it's better to start removing abstraction rather than adding more.<p>That said, I'm also a happy user of `pip` and `virtualenv` and while I'm sure that many people can use the others for more specific needs, I think defaulting to them because they aim to be "simpler" is nearly always a mistake. I still teach beginners to install packages system wide without touching venv at first - it's enough to get you through your first 2-3 years of programming usually.
I've been working in python roles for some years now and I never understood why the python dependency tooling is so poor.<p>Pip feels like an outdated package manager, lacking essential functionality that package managers of other languages have implemented for years. For example, credential redacting in pip was only introduced in 2019, 8 years after its initial release!<p>Not to mention the global-first nature of pip (package is installed globally unless the user explicitly requests for a local installation). You can still install packages locally, but this only shows that pip was not built with environment reproducibility in mind. As a consequence, the need for additional environment tooling (like venv) arose, which increased the complexity of the local python setup.<p>Tools wrapped around pip are also under par. I cannot see why Pipenv is that resource intensive, leading to long and noisy builds (my machine gets close to exploding on a pipenv lock), with very fragile lock files. Debugging an unsuccessful locking in the CI of an enterprise project is a mystery that could take an entire week to solve.
Its javascript counter-part (npm) does the exact same thing, faster and with less CPU usage.<p>Trusting the OS community, I understand that there would be very good reasons for Pipenv to perform like this, but as the consumer of a package managing tool all I see is the same generation of file hashes I see on npm, but with npm doing it way more efficiently. I really see value in the principles that Pipenv is promoting, but to me the developer experience of using it is suboptimal.
Serious question: What is the difference between virtual environments and just having several Python installs like:<p><pre><code> /home/foo/a/usr/bin/python3
/home/foo/b/usr/bin/python2
</code></pre>
Python is so fast to compile and install that I just install as many throwaway Pythons as needed.<p>I do not recall any isolation issues between those installs, unlike with conda or venv, which are both subtly broken on occasion.<p>But I dislike opaque automation in general.
The trouble is these tools all do different things and aren't really comparable. I wouldn't even include Docker in this kind of thing as it doesn't really do anything on its own.<p>For me, there are two main choices today:<p>* An ensemble of single-purpose tools: pip, venv, pip-tools, setuptools, twine, tox,<p>* An all-in-one tool, for example Poetry, Pipenv or Anaconda (or Miniconda).<p>I prefer the former approach, but if I had to choose an all-in-one tool it would be poetry.
I think this is a good basic overview of the dependency management landscape. I have a few things to add.<p>One is that because Python has been around for so long, it's easy to find outdated or conflicting advice about how to manage Python packages.<p>I think it's important to stress that pyenv isn't strictly a dependency manager, too, and depending on your OS, isn't necessary. (Supported Python versions are in the AUR[0].)<p>A lot of pain from Python 2 -> 3 is that many operating systems were so slow to switch their default Python version to 3. Unless something has changed in the last month or so, Mac OS _still_ uses Python 2 as the default.<p>It's a shame to see Python take a beating for OS-level decisions.<p>[0] <a href="https://aur.archlinux.org/" rel="nofollow">https://aur.archlinux.org/</a>
> [Pipenv] loads packages from PyPI so it does not suffer from the same problem as Conda does.<p>False. Conda manages packages installed from PyPI. This is discussed under the Conda section, so I'm surprised the quoted line wound up in the article.
pipenv is terrible. poetry ain't there yet. seems author forgot to mention the problems with pipenv and poetry. virtualenv + pip will take you far. then to reproduce pipe to requirements.txt.
poetry etc are still using pip under the hood
I've alwyas used conda since I use the scipy stack. Can anyone clue me in if I can instead use pipenv and it will download all the requisite binaries etc. ?
Could someone summarize the issues with Pipenv (and by Extension Poetry). Been using them happily for the last few years, didn't know people disliked them.<p>With Pipenv, last year ownership switched from the Request's lib owner to the Pypa, so more or less an officially blessed solution.<p>The only downside on this thread that I could understand so far is that it might be slow to install dependencies on larger projects, can't think of anything else.
After hitting some weird PyInstaller bugs, I gave up and started compiling Python myself. One interpreter for every project. Shell scripts to set the paths. All libraries go directly in site-packages, not some other layer. A little more complicated at the outset, but this approach has yet to let me down. And compared to the nightmares I was trying to fix, building Python is dead easy.
Reading this makes me grateful that I can get away with just <i>apt-get</i>. I wonder how prevalent this is, since not every project needs specific or latest versions of the runtime and libraries, only a minimum. Some are just plumbing tools that stick to the stable core, and the Python 3 ecosystem has been mature for enough years that older distro packages are still useful and capable.
Personally I just try to avoid Python development because I hate feeling like I'm dealing with what should be a solved problem. Recently I had to work with an outdated Python Tensorflow framework and the only way we could get it to work correctly across different dev and deployment machines was with a fat Docker image that took hours of head scratching to build. It was miserable.
pip-tools should really be included here. That's the single-purpose tool that handles environment reproducibility, if you're going with the single-purpose-tool route of pyenv + pip + venv instead of the all-in-one route of poetry/pipenv.
You know how pathlib does for paths and files? Python needs something like that for distribution/versions/import hacking, eh?<p>Glyph (of Twisted fame, whence pathlib IIRC) pointed this out ages ago: model your domain [in Python] with objects.
Oof, that footnote:<p>> It’s 2020, but reliably compiling software from source, on different computer setups, is still an unsolved problem. There are no good ways to manage different versions of compilers, different versions of libraries needed to compile the main program etc.<p>I wonder how much stuff like this has to do with python's popularity. When I have opaque issues like "libaslkdjfasf.so is angry with you and/or out to lunch and/or not doing expected things," it's the most frustrating part of programming. I'd pay devops people infinite money to not have to deal with installation/setup issues anymore.
Everytime I read an article about all these tools I really can't help but think what would happened if Linus would have taken over the desktop. All the tools really largely seem to try to poorly replicate Linux package management and the fact that because of this devs now don't care anymore about api stability and not always building against the latest and greatest.<p>I admit a pyenv is nice for testing against different python versions if necessary. But on my Linux systems generally fine with just installing system packages and doing pip install --user for the odd package that is not in the repositories
I just use Pyenv and pip-tools. Create a requirements.in and Makefile targets to build the requirements.txt based on it. So far I haven't find an sceneario where that combination is detrimental.
This is a good overview of something that took me an annoyingly long time to learn. My personal preference is to keep things simple with pyenv, venv, and pip.<p>Tangentially related is the tool tox [1], which is often used to run a test suite inside of virtual environments created by venv, on multiple versions of Python managed by pyenv.<p>Now if only setuptools could work well without hackery...<p>[1]: <a href="https://tox.readthedocs.io/en/latest/" rel="nofollow">https://tox.readthedocs.io/en/latest/</a>
It's worth noting that on Linux it's slightly different because most of the popular libraries can be installed with the system package manager (no problem of dependency management, updates, ...), I rely on alternative solutions only when I want to use a version of a library different from the one shipped with the package manager (which is not that frequent with fast paced distros like Fedora) or when the library is not packaged.
For some of the problems that Node.js and JS at large have with a centralized package manager, I for one am very happy that it's not in the python situation. 100% of the packages I've tried to install in the last 3+ years are simply `npm install PKG`.
Python dependency management is much like Python 2 to 3, a mess. It's shocking to think pip and pipenv are so widely used and still such terrible tools.
I like this overview. However, it points to a fundamental problem with Python environment, going much against its own credo:<p>"There should be one - and preferably only one - obvious way to do it." - The Zen of Python; see also <a href="https://xkcd.com/1987/" rel="nofollow">https://xkcd.com/1987/</a>.<p>When it comes to the package, environment and dependency management, I think that ironically JavaScript environment is light years head, vide: <a href="https://p.migdal.pl/2020/03/02/types-tests-typescript.html" rel="nofollow">https://p.migdal.pl/2020/03/02/types-tests-typescript.html</a>
It's 2020, but the python community still has not converged to a small set of sane solutions.<p>It seems to me that Ruby, PHP, JS, and Rust communities have solved the problem.