One does not simply 'pip install'

The next Debian&#x2F;Ubuntu releases will no longer allow `pip install` outside of a venv: <a href="https:&#x2F;&#x2F;discuss.python.org&#x2F;t&#x2F;pep-668-marking-python-base-environments-as-externally-managed&#x2F;10302&#x2F;69" rel="nofollow">https:&#x2F;&#x2F;discuss.python.org&#x2F;t&#x2F;pep-668-marking-python-base-env...</a><p>You can still force it via `pip install --break-system-packages ...` if needed.

&gt; There’s no shortage of package management alternatives available for Python [...]<p>&gt; How someone is meant to pick between these as a new developer is a mystery.<p>This.<p>Every time I get booked to look at some Python project hours are usually wasted initially figuring out what dependency mgmt solution was used how. And with what &#x27;special sauce&#x27; the resp. developers deemed to be &#x27;the right way&#x27; (or some library required because ... it just does)<p>As the author wrote: it seems common to omit the dependency setup in the Readme for Python projects.<p>I can understand why one would not mention this &#x27;step&#x27; in a Rust or Node project but for Python it seems very much necessary.

I&#x27;m convinced that there are very few python libraries that Just Work if you follow their installation instructions. I&#x27;ve never found one that didn&#x27;t come with issues myself.<p>Complain about this to a Python dev and you&#x27;ll be &quot;Well actually&quot;ied to oblivion and each and every one will have their own opinion-as-fact on the best practice for managing these -- totally unaware how antithetical Python development has become from The Zen of Python.

Slightly tangential, but... one also does not simply `pip download` if one does not want execute code - <a href="https:&#x2F;&#x2F;github.com&#x2F;pypa&#x2F;pip&#x2F;issues&#x2F;1884">https:&#x2F;&#x2F;github.com&#x2F;pypa&#x2F;pip&#x2F;issues&#x2F;1884</a>.<p>I wanted to run guarddog on source packages. Only then build them locally and install. Turns out, `pip download` triggers code execution in fetched packages.<p>Somewhat surprising and in this day and age worth spreading awareness of.

A Makefile makes this trivial:<p><pre><code> # Makefile all: venv frozen test venv: python3 -m venv install venv frozen: [ -e frozen.txt ] || { echo &quot;ERROR: run &#x27;make update-frozen&#x27;&quot;; exit 1 ; } .&#x2F;venv&#x2F;bin&#x2F;pip install -r frozen.txt update-frozen: clean install-requirements freeze freeze: .&#x2F;venv&#x2F;bin&#x2F;pip freeze &gt; frozen.txt install-requirements: [ -e requirements.txt ] || { echo &quot;ERROR: make a requirements.txt file&quot;; exit 1 ; } .&#x2F;venv&#x2F;bin&#x2F;pip install -r requirements.txt test: .&#x2F;venv&#x2F;bin&#x2F;python3 run_tests.py clean: rm -rf venv </code></pre> Put your package names in requirements.txt and run `make update-frozen`. To reinstall everything from frozen state, `make clean frozen`. (And replace the first space with a tab; HN is stripping my tabs out)<p>I know Pythonistas like to use Python for <i>everything</i>, but there are other tools out there that will make your life much simpler.

The article talks about installing Python packages for development, but if you find yourself using `pip` to install Python <i>tools&#x2F;scripts</i> then you should use `pipx` - it will properly sandbox those tools so they don&#x27;t break (or be broken by) the system or other Pythons:<p><a href="https:&#x2F;&#x2F;pypa.github.io&#x2F;pipx&#x2F;" rel="nofollow">https:&#x2F;&#x2F;pypa.github.io&#x2F;pipx&#x2F;</a>

While I agree with the author to not do global pip installs for every new project, I also don’t want to see text in every git repo README explaining Python package managers.

I appreciate the concern for new developers, but I really don&#x27;t think it&#x27;s a good solution to have every project readme describe pip, poetry, pipenv, and whatever other new hotness there is in the package management world. There&#x27;s a reason that all the readmes describe pip installation: it&#x27;s the lowest common denominator, present with every standard python install, and along with virtualenv (also standard) it can do most of the requirements for package management.<p>I think to help new developers, we could encourage documentation to briefly point to the official PyPA documents on the variety of options available. It would be better to focus on making that more accessible, rather trying to throw the burden onto package maintainers to describe using their package with every new tool.<p><a href="https:&#x2F;&#x2F;packaging.python.org&#x2F;en&#x2F;latest&#x2F;key_projects&#x2F;" rel="nofollow">https:&#x2F;&#x2F;packaging.python.org&#x2F;en&#x2F;latest&#x2F;key_projects&#x2F;</a>

Omg this is so true! I installed a package globally, but then my interpreter was using another version of python, which doesn&#x27;t have the installed package. It took me an hour to find out about this. What a waste of time.

Pyenv virtualenv is my personal fav for this issue

It&#x27;s also interesting how things like AWS Lambdas, Graviton, etc, are exposing all the shortcomings of the various pip install, venv, poetry, etc, approaches.<p>It&#x27;s not impossible to figure it out, but you end up spending a lot of time to come up with something that works locally, within containers, inside a CI&#x2F;CD system, and then deployed out across things like Lambdas, or non x64 machines.<p>Then, after it&#x27;s all working, upgrading the Python version, or an extension that has C code, etc, repeats some of the hard bits.

When I switched to Arch Linux, I learned that pip has a --user option to install Python packages in the home dir of the current user. This is essential to not interfere with the system install from the system package manager. I had really trouble with that in former times.<p>Furthermore, as I now be used to bleeding edge packages, I update at least once a week all the outdated Python packages of my &gt;450 installed ones. When some packages get downgraded because of requirements, I ask: Do I need the package that caused the downgrade more often or with more of the packages in the main environment, or is this true for one or some of the downgraded packages?<p>According to the answer, I put the &#x27;problematic&#x27; package(s) in a new or existing venv, and update the downgraded ones in the main environment, if necessary.<p>This work cannot be done by a package manager!<p>Costs me &lt;10 minutes every week to keep the main environment up to date, a bit more if I want that for some or all venvs.

&gt; You might expect if I were to pip uninstall requests that I get back to a clean system, right?<p>Why would i expect that? If one day I install A and another day I install B, which depends on A, I wouldn’t expect to lose A of I were to uninstall B.

This post points out one of my struggles with python.<p>I am not a python developer but I use python heavily for some tooling. So all I need to do is to “distribute” my tools to other servers in a replicable and consistent matter, isolated from global packages.<p>Can you please help me understand two points?<p>1. If I use venv+pip to install some python app, do I have to “activate” that specific virtual environment before executing that tool or can I just simply call it by its path on the file system?<p>2. Are there any official guide rails for making venv-wrapped app accessible to other users on a server? Or just as simple as placing links to &#x2F;usr&#x2F;local&#x2F;bin&#x2F; for example?

Also, it&#x27;s 2023 - please do not suggest approaches that do not use package hash based pinning. For example, use pip-tools.<p>Hopefully, in 2024, we will be able to say same thing about signing via sigstore ecosystem.

&gt; Lets say you use the same package again, but theres been a new release with some additional features. When you upgrade your global Python to use it, you now need to ensure every project you’ve done now works with it. That’s annoying and unlikely to happen, what you’ll be left with is a broken build.<p>Wait, what?<p>Don&#x27;t python packages generally use `semver` versioning, and ensure that upgrades in the same major version are backwards-compatible?<p>And that different major versions are co-installable?

I saw this Twitter thread the other day (<a href="https:&#x2F;&#x2F;twitter.com&#x2F;fchollet&#x2F;status&#x2F;1617704787235176449?s=46&amp;t=75AlDtvEJGqALnTvCn43qg" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;fchollet&#x2F;status&#x2F;1617704787235176449?s=46...</a>) about similar problems, and some comments suggest using Docker. I couldn’t find any guides or ways to do this for a Python project; anyone here know more or has done this before?

poetry breaks once a while for me, so I am not using it these days.<p>pipenv used to be my first choice but it became inactive, seems it is actively under development again?<p>a few weeks ago there is a recommendation for PDM but I have not really used it.<p>For now I am using the pip+venv approach.<p>By the way, you better do: `python -m pip install` instead of `pip install`, don&#x27;t remember why anymore but I did read somewhere that explained the difference and I agreed on then to prefer &#x27;python -m pip install&#x27;

I use Python for research. If I need some package, I simply want the latest version; pip install is usually fine.<p>If something depends explicitly on the fixed (old) version, that&#x27;s when problems happen and I grudgingly remember how to use pyenv. But I like to use the most recent versions and most recent Python, and I like packages that share this bleeding edge approach.

Article conflates global installation into the system python with global installation in general. Not everything is a project dependency. If you want, say, ipython, available everywhere, global installation is appropriate. You can get this without clobbering my system python by simply not using the system python for my projects.

I&#x27;ve been a happy user of pipenv for several years (at work, in production) and still recommend it. You lock the versions you want independently of the requirements.txt so you can update just the packages you want without worrying about sub-dependencies. 10&#x2F;10 recommend.

Not to mention dependencies that compile C modules so you also need a compiler, headers etc

npm certainly has a number of problems (at the end the article compares pip to npm) -- but after reading this article I didn&#x27;t realize pip was so problematic. I also didn&#x27;t realize it installed things globally.<p>So the solution is?

Is there a canonical example of how python projects should manage dependencies and sandboxing such that other developers can just clone, install, and get to work?

Personally I usually found pip + venv (or conda) less painful than NPM, Nuget or whatever voodoo and goat sacrifices you have to do for C++.

Poetry is pretty much the way to go.

Well written article

Obligatory XKCD: <a href="https:&#x2F;&#x2F;xkcd.com&#x2F;1987&#x2F;" rel="nofollow">https:&#x2F;&#x2F;xkcd.com&#x2F;1987&#x2F;</a>

Just use Docker.

I&#x27;m not sure if I get the point of this article. So basically the author has learnt that there are a different ways of managing packages in Python? I&#x27;m aware that this might be a problem in Python, but let&#x27;s be serious guys, you only need to spend 5 mins to learn about venv&#x2F;conda and you will never face any problem in a basic Python project. You don&#x27;t have to write an article about that.