Underappreciated challenges with Python packaging

Pillow has most of the issues that are listed in the article. (oddly enough for a graphics library, the GPU part is the only part that I don&#x27;t think we&#x27;ve stumbled over at one point or another.)<p>From a quality of life issue -- having the sdist install behind an opt-in flag by default for our package would be great. Unless you&#x27;re a developer with a lot of -dev packages for imaging libraries already on your system, you&#x27;re not going to be able to build from source. And even if the error that pops up is along the lines of<p><pre><code> The headers or library files could not be found for {str(err)}, a required dependency when compiling Pillow from source. Please see the install instructions at: https:&#x2F;&#x2F;pillow.readthedocs.io&#x2F;en&#x2F;latest&#x2F;installation.html </code></pre> We still get people posting issues about pillow failing to install.<p>Build farms would be nice. We&#x27;ve burned tons of time on it between travis and GH actions and @cgohlke single handedly making all of the windows builds for the entire scientific python community.<p>Ultimately, something like the debian packaging system is probably the best open source model for this. (though the splitting of the python standard library so that virtual envs aren&#x27;t in the base install is a pita). Unstable gets a reasonably current set of packages, and crucially all of the underlying library dependencies are compiled together. It&#x27;s also not _that_ hard to rebuild individual packages from source, in an automated fashion. (This may be what Conda is doing, but I&#x27;ve never looked in detail at their system)

I&#x27;ve been planning on packaging a python package recently, and the internet is annoyingly full of guides which are, I think, out of date. They at least suggest quite different things.<p>I just have a single python file, meant to be treated as an executable (no package at present). There are a whole bunch of tests, but that&#x27;s obviously separate. Any suggestions on modern best practices welcome!

This is an incredible example of organizing information well and making a case to a wide audience. It&#x27;s difficult enough to shave all the yaks necessary to get a high-level view of all issues related to a problem, and to express all those problems in good writing is an additional tough challenge. These folks have done an amazing job at both.<p>Shoutout to Material for MkDocs enabling the swanky theme and Markdown extensions. <a href="https:&#x2F;&#x2F;squidfunk.github.io&#x2F;mkdocs-material&#x2F;" rel="nofollow">https:&#x2F;&#x2F;squidfunk.github.io&#x2F;mkdocs-material&#x2F;</a>

With modern tooling packaging pure python code to be used by other python developers is a relatively painless process.<p>The main problem with python packaging is that it&#x27;s often C&#x2F;C++ packaging in disguise, among multiple OSes and CPU architectures, and that&#x27;s far from being solved. Building such python wheel is essentially like building a &quot;portable&quot; (aka one you don&#x27;t need to properly install into the system) linux&#x2F;windows&#x2F;macos application. That comes with a variety of caveats and requires some specialized knowledge one wouldn&#x27;t pick up playing around with just python alone.

Is there any consensus on how to deal with packaging and environments in Python by now? Can you suggest me some tutorial for that?<p>I&#x27;ve been out of the loop for a long time, and would like to get an update on how things are in Python in 2023, but I&#x27;m not sure if there even is a consensus — what I can find by googling seems to be several kinda competing approaches. This seems surprising, because most &quot;modern&quot; languages seem to have a very well defined set of practices to deal with all of that stuff. Some languages already come with their built-in stuff (Go, Rust), others simply have well-known solutions (like, technically there still exist PEAR and PECL for PHP, but everyone just knows how to use composer, which solves both packaging and dependency-management problems, and it&#x27;s also pretty clear what problems it doesn&#x27;t solve).<p>For Python there seems to be like a dozen of tools and I&#x27;m not sure which are outdated and not used by anyone, which are useless fancy wrappers (not used by anyone) and what is the actual go-to tool (if there is any) for all common cases. Dependency-management, version locking, shipping an executable, environment separation for some local scripts, should I even ever use pip install globally, etc.

Python joke:<p>Sdist is only one letter away from sadist.

This is exactly why I avoid pip et al whenever I can and just use Nix.

Obligatory Xkcd comic.<p><a href="https:&#x2F;&#x2F;pypi.org&#x2F;project&#x2F;xkcd2347&#x2F;" rel="nofollow">https:&#x2F;&#x2F;pypi.org&#x2F;project&#x2F;xkcd2347&#x2F;</a>

Python packaging’s greatest challenge is 10 competing tools and standards.

Python packaging is a solved problem:<p><a href="https:&#x2F;&#x2F;python-poetry.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;python-poetry.org&#x2F;</a>