TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Vanilla Python Packaging

3 pointsby ldubinets9 months ago

1 comment

zahlman9 months ago
&gt; My build step is then just python3 -m build. This populates the dist&#x2F; directory with everything needed to install and run the project on a server. &gt; ... &gt; Just python, pip, rsync, and systemd are enough to build and deploy a production ready python service.<p>It should be noted here that, while it&#x27;s a small, semi-standard tool under the PyPA umbrella, `build` is not part of the standard library. It must be installed separately, and thus should have been listed as one of the tools used. (On the other hand, I don&#x27;t really see what any of this has to do with systemd; it can equally well be made to work on Windows or on non-systemd Linux distros.)<p>&gt; First of all, I have nothing against Poetry, uv, pdm, and others. I just prefer not to learn more tools, more configurations, and use only standardized built-in tools wherever possible. So as a result, I use setuptools only.<p>`setuptools` has the advantage of preferential treatment within the ecosystem (e.g., at least transitionally, a `[build-system]` table like the one shown could be omitted completely, as Setuptools would be assumed by default; and virtual environments used to bootstrap Setuptools as well as Pip by default.<p>However, with current versions of Pip (and Python, thus venv from the standard library), as well as with `build`, the default behaviour is to create an &quot;isolated environment&quot; for building (even, AFAIK, if the project is already containerized - how would the tools know, after all?). This entails creating a new venv and installing Setuptools therein - using Pip to find the latest version (subject to any version constraints specified in pyproject.toml) on PyPI. In theory, packages that Pip downloads from PyPI are cached; but even when Setuptools hasn&#x27;t released a new version since the last build, any number of things could defeat that caching scheme.<p>This is important because, while Setuptools <i>works</i> and it succeeds in making the simple cases simple (including with the new pyproject.toml-powered approach), it&#x27;s a <i>massive</i> boat-anchor for most projects that use it. The wheel size approaches 1MB for a possibly-transient download used just to &quot;build&quot; your project - which in most cases doesn&#x27;t even involve a compilation step (actually telling Setuptools to run compilers requires a separate setup.py file, even with pyproject.toml) but just moves some files around and creates a zip archive - and which in most cases is itself quite a bit smaller. It got to this size, effectively, by starting its life as a decades-old attempt at a full project management toolchain, and then collapsing under the weight of deprecations and backwards-compatibility hacks (including, now, the need to vendor `distutils` since that was removed from the standard library in 3.12).<p>Notably, originally setup.py was intended to be run directly chosen by the user, but now Setuptools does magic to spawn a new process (IIRC), run it with very specific arguments and collect the results somehow. Meanwhile, even the most conservative attempts to remove old deprecated functionality (such as the ability to run `setup.py test` manually, even though it has nothing to do with the modern Setuptools workflow and everyone uses Pytest and invokes <i>that</i> directly nowadays) have caused huge breakages in the ecosystem and embarrassing rollbacks (see <a href="https:&#x2F;&#x2F;github.com&#x2F;pypa&#x2F;setuptools&#x2F;issues&#x2F;4519">https:&#x2F;&#x2F;github.com&#x2F;pypa&#x2F;setuptools&#x2F;issues&#x2F;4519</a> as a starting point), <i>even for pure-Python projects where that functionality was grossly unnecessary and wasn&#x27;t actually being used</i> (notably: Requests - <a href="https:&#x2F;&#x2F;github.com&#x2F;psf&#x2F;requests&#x2F;issues&#x2F;6775">https:&#x2F;&#x2F;github.com&#x2F;psf&#x2F;requests&#x2F;issues&#x2F;6775</a> - and Celery - <a href="https:&#x2F;&#x2F;github.com&#x2F;celery&#x2F;celery&#x2F;issues&#x2F;9157">https:&#x2F;&#x2F;github.com&#x2F;celery&#x2F;celery&#x2F;issues&#x2F;9157</a> ).<p>Setuptools is one of the most popular packages on PyPI. I estimate, based on the available public data, that it accounts for about 1% of <i>all PyPI download bandwidth</i> - petabytes per year, worth perhaps a dev&#x27;s entire salary at market rates. I suspect that a very high fraction of these downloads should be unnecessary if people were properly educated about how this works - and even where Setuptools is used, it could be much smaller while providing the same non-deprecated functionality - and most users aren&#x27;t using most of that functionality anyway.<p>I have much more to say about this, as part of a broader discussion of Python wheel sizes. I&#x27;m hoping I can get to that blogpost within the next month or so.<p>In the mean time: I have been working on a build backend called `bbbb` that attempts to be minimal. It&#x27;s not at all production-ready yet, but you can get a basic sense of what I&#x27;m going for: <a href="https:&#x2F;&#x2F;github.com&#x2F;zahlman&#x2F;bbbb&#x2F;">https:&#x2F;&#x2F;github.com&#x2F;zahlman&#x2F;bbbb&#x2F;</a> . It&#x27;s at least powerful enough to package itself, and should work for any project that&#x27;s only Python code aside from its dependencies, if you use src layout and are careful about file placement. If you need something that&#x27;s properly tested and such, now, I recommend Flit. It doesn&#x27;t, by design, provide a way to deal with compiled extensions (whereas I will eventually), and it&#x27;s definitely not as small as it could be (it vendors `tomli` and includes its tests in the wheel, which you&#x27;ll never have a reason to run yourself) - but it&#x27;s still much smaller than Setuptools.