My User Experience Porting Off Setup.py

Packages have always been Python&#x27;s Achilles heel. Python&#x27;s philosophy for <i>code</i> is, &quot;there should be one, and preferably only one, obvious way to do it.&quot; But for <i>packaging systems</i>, their philosophy is more like Perl: &quot;there&#x27;s more than one way to do it (and all of them have their own pitfalls).&quot;<p>I don&#x27;t understand why the Python leadership hasn&#x27;t shown stronger... leadership... in which tools they recommend.

A friend of mine had a critique of videogames that has stuck with me. He called them &quot;antiknowledge&quot;. Your brain&#x27;s learning faculties are repurposed into &quot;learning&quot; something that isn&#x27;t actually a <i>thing</i>. You get really good at some artificial skinner box treadmill system ... and then they change the meta and you have to keep up with the new stats and strategies and so on. Yet after sinking in hundreds of hours, you don&#x27;t come out the other end with any tangible real-life skill, that would be useful outside of the game.<p>That&#x27;s what Python packaging feels like. At least videogames are fun.

I just wanna say I really hate the new poetry+toml crap, it&#x27;s just more complicated and yet another set of crap I gotta deal with arbitrarily by arbitrary projects.<p>Why not just improve upon pip? I don&#x27;t know, just have pip use toml and give it different flags or auto detect things? Was a whole new ecosystem of tools needed?<p>I look at decisions like this and I know a Python 4.0 is on the way, just like the 2-&gt;3 jump because why not right?<p>Any language that updates it&#x27;s syntax or toolset in a way that is impossible to have backwards compatibility is an irresponsible toy language, as awesome and powerful as it may be, the developers are still toying and tinkering with it with no regard to real world impact of their decisions.<p>Why can&#x27;t python3.12 have a flag like --std=2.7.18 for example, like gcc? If the devs don&#x27;t have enough resources, I would live to see a donate page I can help out with.<p>We are at a point where to deploy python you can&#x27;t use python but you need shell scripts to figure out versions, venvs, pyev, pipx, poetry, etc... and reliably fail of course and every time you have to troubleshoot the problem. This is a failure in software engineering, new grads should be taught of python and similar languages and the lack of planning and organization and resulting cascading chaos as examples of how not to design the user experience of any piece of software.<p>Sorry if I exaggerated a bit anywhere, it&#x27;s difficult to pretend all the frustrations and crying out &quot;why???&quot; when using python don&#x27;t exist. But at the same time, it is still my #1 go to language for most use cases because the language itself is just fabulous!

About &quot;eager deprecations&quot; let me give you another absolute gem:<p><pre><code> ******************************************************************************** The license_file parameter is deprecated, use license_files instead. By 2023-Oct-30, you need to update your project and remove deprecated calls or your builds will no longer be supported. ******************************************************************************** </code></pre> Yes, please go ahead and break people&#x27;s builds at an arbitrary date because the technical challenges of supporting both `license_file` and `license_files` are insurmountable.

Thank you Gregory for writing this post. There have been a bunch of announcements about &quot;setup.py has been deprecated&quot;, but few have clearly outlined how to move away from setup.py, and more importantly, fewer have outlined what a struggle it is to move away from setup.py.<p>I was sad to see setuptools officially deprecated, because it looks like another way in which Python packaging is being red-taped away for a non-expert. If someone like the OP (who has 10+ years programming Python) had to do so much for what appears to be a zstd CFFI&#x2F;Rust wrapper, where does that leave the rest of us?<p>Here&#x27;s a python package of mine that uses setup.py: <a href="https:&#x2F;&#x2F;github.com&#x2F;ahgamut&#x2F;cliquematch&#x2F;blob&#x2F;master&#x2F;setup.py">https:&#x2F;&#x2F;github.com&#x2F;ahgamut&#x2F;cliquematch&#x2F;blob&#x2F;master&#x2F;setup.py</a> which I have not upgraded to the new tool(s) yet. I think I will need to upgrade it soon. If anyone has suggestions for a tool that will _fully replace_ setup.py, I would like to see tutorials with the following examples:<p>1. How would I build a package that has pure-Python files and data files? With setuptools I would use maybe MANIFEST.in or package_dir.<p>2. How would I build a package that has a CPython extension accessed via cffi? (this post points to the answer)<p>3. How would I build a package that has a CPython extension _without_ cffi, that just wraps some small C code I wrote with CPython&#x27;s API? What about an extension that uses PyBind11? What about an extension that uses Rust?<p>4. How would I build a package that requires a &quot;system&quot; package like libblas-dev? Can something like numpy be built optimally without ever writing setup.py? What would a config for that look like? Last I remember numpy used their own patch of distutils to build, I wonder what it is now.

I think the documentation (or lack of documentation) can be very illuminating in the focus and though process of those behind a project. That there is concise &quot;This is how you did it before and this is how it is done now&quot; shows that there seemingly haven&#x27;t been any though process in building that bridge. And in general it seems like the process of packaging projects that is not going to be published on pypi, but are going to be used internally is a dark spot as well.<p>Having the mess of different blog posts and documentation sources saying different stuff is far from ideal though. If you can&#x27;t sum up the process in a clear concise way you are far from done.

As someone who just yesterday also went through the exercise of (finally) understanding how to package with pyproject.toml, I empathize with the author and agree with a few painpoints he mentioned. Namely the confusion caused when opening some pages in <a href="https:&#x2F;&#x2F;packaging.python.org&#x2F;en&#x2F;latest&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;packaging.python.org&#x2F;en&#x2F;latest&#x2F;</a> and seeing references to a number of soon to be deprecated tools and approaches to packaging, as though they still have a place in the horizon. It&#x27;s especially frustrating because the website is versioned, so you would expect a deliberate use of deprecation warnings and clear recommendations to migrate to new approaches and tools. I opened that website with the understanding that pyproject.toml is the future and setup.py is out. Instead, I still saw pages where the two are treated as though they will coexist for a while.<p>Having said that, the author also sounds like he&#x27;s ranting a bit. He seems to insist in finding specifically how to work the way setup.py used to, but without setup.py, instead of just learning how to use pyproject.toml. While learning the new way of doing something, how it replaces the old way is usually self-evident. The (official) tutorial he eventually lands on (<a href="https:&#x2F;&#x2F;packaging.python.org&#x2F;en&#x2F;latest&#x2F;tutorials&#x2F;packaging-projects&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;packaging.python.org&#x2F;en&#x2F;latest&#x2F;tutorials&#x2F;packaging-p...</a>) is actually a pretty good primer. Without previously knowing what <i>hatchling</i>, <i>build</i>, <i>twine</i>, or even <i>pyproject.toml</i> was, I was able to quickly understand their purpose. From clicking a few other links on the side bar, I understood that packaging is done with tools that present a frontend and interact with a backend. Sometimes a tooling set provides both. Hatch seems to be the frontend of one such project, while Hatchling is the backend.

I&#x27;ve had a similarly frustrating time trying to understand and wrangle the pyproject.toml builder system, (egg-layer? wheel-roller? cheese-monger?)<p>One thing the author might want to try is writing their own &quot;build-backend&quot;. You can specify your own script (even use setup.py) and that will be the target of python -m build or pip wheel or presumably whatever build-frontend you use.<p><pre><code> # pyproject.toml [build-system] requires = [&quot;setuptools&quot;] build-backend = &quot;setup&quot; # import setup.py as the build-module backend-path = [&quot;.&quot;] </code></pre> Then in setup.py you should write two functions:<p><pre><code> def build_sdist(sdist_directory, config_settings): ... def build_wheel(wheel_directory, config_settings, metadata_directory): ... </code></pre> Where config_settings is a dictionary of the command line &quot;--config-settings&quot; options passed to the builder. (sys.argv does not have access to the actual invocation, I suppose to ensure frontend standardization)<p>example:<p><pre><code> $ python -m build --config-setting=foo=bar --config-setting=can-spam # will call &gt;&gt;&gt; build_sdist(&quot;the&#x2F;dist&#x2F;dir&quot;, {&quot;foo&quot;: &quot;bar&quot;, &quot;can&quot;: &quot;spam&quot;}) </code></pre> Of course, you can extend the default setuptools build meta so you only have to do the pre-compilation or whatever your custom build step requires:<p><pre><code> from setuptools.build_meta import build_sdist as setuptools_build_sdist def build_sdist(sdist_directory, config_settings): # ... code-gen and copy files to source ... # this will call setup.py::setup, to make things extra confusing return setuptools_build_sdist(sdist_directory, config_settings) </code></pre> I had to create a temporary MANIFEST.in file to make sure that the setuptools build_sdist saw the generated files. Maybe there&#x27;s a better way? I think the wheel &quot;just&quot; packages whatever the sdist produces, though that might be more difficult if you&#x27;re compiling .so files or whatnot.<p>Still overall pretty fiddly&#x2F;under-documented and a shame there seems to be a push for <i>more</i> dependencies rather than encouraging users to build their own solutions.<p>More info in PEP 517: <a href="https:&#x2F;&#x2F;peps.python.org&#x2F;pep-0517&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;peps.python.org&#x2F;pep-0517&#x2F;</a>

<p><pre><code> I open https:&#x2F;&#x2F;blog.ganssle.io&#x2F;articles&#x2F;2021&#x2F;10&#x2F;setup-py-deprecated.html in my browser and see a 4,000+ word blog post. Oof. Do I really want&#x2F;need to read this? </code></pre> <i>*proceeds to write an 8,000 word blog post about it*</i><p>-----<p>Kidding aside, good content and a great reminder to think before blindly upgrading—at least until the kinks and details are worked through

I’m quite surprised to see that setup.py has been deprecated for years. I set out to setup my first package earlier this year and after spending way too much time trying to figure out what is recommended and why I actually settled on setup.py<p>I knew it was “the old” way, but didn’t realize it was abandoned.<p>Getting your code packaged seems way harder than developing your code.

&gt; No offence meant to the Poetry project here but I don&#x27;t perceive my project as needing whatever features Poetry provides: I&#x27;m just trying to publish a simple library package<p>I don&#x27;t understand... Poetry solves this too.

I&#x27;ve decided (a few years ago) that if i&#x27;ll ever have to upgrade the packaging for my stuff, i&#x27;m going to do it with nix (and only that - sorry other os users, you&#x27;ll have to install nix - actually: not sorry!). I was in (distro) packaging way too deep and decided that my limited time in this world doesn&#x27;t allow for that kind of crap, anymore.<p>The blatant mess that ensued in the meantime (i.e. last few years) proves me right, imho.

It is as if someone has taken the time to describe my typical day. I&#x27;m not a package maintainer or anything, just a guy trying to keep multiple CI&#x2F;CD systems working.<p>I did not get halfway through this before I could start to feel the hairs of my neck start to stand on end. The numerous blind alleys. The promising leads that aren&#x27;t. The official documentation that contains links to other documents that contradict the first and so I have to try and also piece together some temporal state to make any sense of any of it. Some days it seems I am actually some kind of an information archeologist, piecing together the detritus of overlapping civilizations.

Its more funny to migrate the old setup.py process to the pyproject.toml and an rpm specfile. with an setup.py you can say something like &quot;python3 setup.py install -root=$builddir&quot;

Thank you for sharing this journey! Very helpful.

from* ffs

Ouch. I stopped reading 15% of the way through because I noticed that I was only 15% of the way through and I felt like it should be the end.<p>It seems like with a tenth of the effort of this blog rant the author could have written a flowchart or best practices.<p>Luckily for 99% of Python users, we only need to install libraries and not package them...