TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

The problem with packaging in Python

44 pointsby jimsojimover 9 years ago

16 comments

syllogismover 9 years ago
Noooooooo<p>This is a really terrible suggestion.<p>The last thing we need is <i>another</i> way to do things, that only takes care of, say, 70% of the functionality.<p>All that will happen is, someone now has to look through the simple interface, and decide oh wait, I need one feature this doesn&#x27;t provide. Now your &quot;solution&quot; has made the problem worse, not better.<p>If you&#x27;re going to provide this sort of simplified interface, you need to make damn sure a <i>super majority</i> of users never need to look past it. Otherwise you&#x27;ve done much more harm than good, by providing yet another competing alternative.<p>The real problem with packaging in Python is that users are asked to write this program, setup.py, that by itself should not be a very difficult program to write. Its tasks simply are not that complicated.<p>What makes it complicated is that they&#x27;re then told that their program should consist of a single function call --- a call to this monstrously complicated setup() function, with a confusion of conflicting arguments.<p>This is a stupid way to write a program! This design decision is at the heart of the whole problem. The interface to setuptools, distribute, distutils etc is fucked and always will be fucked, because there&#x27;s no way to provide a good interface to the functionality via a single function call.<p>That&#x27;s why every semi-complicated setup script ends up having to monkey patch the internals of setuptools or distutils, to say, replace some of the Extension class machinery, or try to correct a compiler flag. It&#x27;s because the design is fundamentally a failure. The whole idea does not work.<p>Things would be much easier if we had direct, simple control. You can provide a default fall-through, but it should be easy to route the &quot;build_ext&quot;, &quot;install&quot;, &quot;sdist&quot; etc commands to your own function.<p>Just tell us what our program needs to do and provide us a nice library of building blocks to do it, and we will write this program. Simple.<p>By far the majority of my tickets for my NLP library, spaCy, are related to packaging and installation. Why should putting some files onto a user&#x27;s computer and zipping up some files on my machine and calling out to a compiler and specifying some dependencies be harder than understanding natural language?
评论 #10411176 未加载
评论 #10411147 未加载
crdoconnorover 9 years ago
One thing that is direly needed in pip is a way to specify system package dependencies. The way packages fail right now if you&#x27;re missing something like libjpeg is cryptic and nasty. Users need to know that they need to &quot;apt-get install x&quot; or &quot;yum install y&quot; without googling for that cryptic error message.<p>Compared to that, I&#x27;d consider the overgrown setup.py interface or need for a MANIFEST.in file to be barely even a problem.<p>I tried writing a package that took care of specifying unix packages in a distro independent way (<a href="https:&#x2F;&#x2F;github.com&#x2F;unixpackage&#x2F;unixpackage" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;unixpackage&#x2F;unixpackage</a>), checked for installation and asked politely via sudo to install if the package wasn&#x27;t installed.<p>However, in pip, the &quot;ask politely&quot; approach became impossible after v1.5.6 because of this bug&#x2F;decision:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;pypa&#x2F;pip&#x2F;issues&#x2F;2732" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;pypa&#x2F;pip&#x2F;issues&#x2F;2732</a><p>(a sudo prompt came up but it was not possible to give an indication to the user of why it was there, so I gave up on the idea)
评论 #10411153 未加载
falsedanover 9 years ago
So Perl had the same sort of problem—everyone used the cpan client to install packages by wrapping the calls to<p><pre><code> perl Makefile.PL make make test make install </code></pre> And package authors would use helper modules to handle the Makefile.PL cruft[0][1]. It sucked.<p>rbjs wrote Dist::Zilla[2], a packaging abstraction tool: you would tell it what was in your package, and it would write the installer for you (and you could choose which of the Makefile.PL helpers to use, if you cared). The end user never sees any remnant of the tool.<p>The closest Python has right now is pbr[3].<p>[0] <a href="https:&#x2F;&#x2F;metacpan.org&#x2F;pod&#x2F;ExtUtils::MakeMaker" rel="nofollow">https:&#x2F;&#x2F;metacpan.org&#x2F;pod&#x2F;ExtUtils::MakeMaker</a> [1] <a href="https:&#x2F;&#x2F;metacpan.org&#x2F;pod&#x2F;Module::Install" rel="nofollow">https:&#x2F;&#x2F;metacpan.org&#x2F;pod&#x2F;Module::Install</a> [2] <a href="https:&#x2F;&#x2F;metacpan.org&#x2F;pod&#x2F;Dist::Zilla" rel="nofollow">https:&#x2F;&#x2F;metacpan.org&#x2F;pod&#x2F;Dist::Zilla</a> [3] <a href="http:&#x2F;&#x2F;docs.openstack.org&#x2F;developer&#x2F;pbr&#x2F;" rel="nofollow">http:&#x2F;&#x2F;docs.openstack.org&#x2F;developer&#x2F;pbr&#x2F;</a>
yen223over 9 years ago
It&#x27;s funny that a lot of Python&#x27;s design hinges around having &quot;one right way to do it&quot;, which is a strength of the language, and yet there&#x27;s a million different ways to do packaging.<p>Python really needs one, and only one, true way to do packaging, but I think it&#x27;s too late for that to happen now.
评论 #10411125 未加载
评论 #10411101 未加载
评论 #10410902 未加载
slavik81over 9 years ago
I actually just built my first python package two weeks ago. It was pure-python and I found it fairly straightforward, starting with the bare-bones LPTHW [1] template, and adding things from the Sharing Your Labor of Love [2] guide.<p>It basically &#x27;just worked&#x27; across Python 2, Python 3, Linux, OSX and Windows. There&#x27;s a lot of outdated examples out there, and it might be more troublesome if I had a more complicated project, but... well, it was ok for me.<p>[1] <a href="http:&#x2F;&#x2F;learnpythonthehardway.org&#x2F;book&#x2F;ex46.html" rel="nofollow">http:&#x2F;&#x2F;learnpythonthehardway.org&#x2F;book&#x2F;ex46.html</a> [2] <a href="https:&#x2F;&#x2F;hynek.me&#x2F;articles&#x2F;sharing-your-labor-of-love-pypi-quick-and-dirty&#x2F;" rel="nofollow">https:&#x2F;&#x2F;hynek.me&#x2F;articles&#x2F;sharing-your-labor-of-love-pypi-qu...</a>
colandermanover 9 years ago
As an outsider to Python&#x2F;Perl&#x2F;Ruby&#x2F;etc., I&#x27;ve never understood why these languages have each their own packaging system, rather than simply relying on the host OS&#x27;s packaging system. After all, it serves C, C++, and Bash well; and pip packages generally are repackaged as Red Hat&#x2F;Debian&#x2F;etc. packages anyway.<p>Making .rpms and .debs is not all that hard, though I admit there are a few too many package formats (and that still leaves non-Linux users without a clean solution). Beside this, what benefits do users of these language-specific packaging systems enjoy that I am missing?
评论 #10411430 未加载
评论 #10411371 未加载
评论 #10411355 未加载
jaywunderover 9 years ago
My biggest takeaway from the article is that you &quot;want something simple&quot;, and most people would agree with you. What you don&#x27;t talk about, or dodge along the lines of is saying &quot;X is how we should do it&quot;.<p>Personally I like node&#x27;s way of using package.json and cargo&#x27;s Cargo.toml. If someone made that for python that&#x27;d be awesome.<p>The other thing node has specifically is the &quot;node_modules&quot; folder, which recursively has all the dependencies and their node_modules folders. While it might not be super space efficient it&#x27;s really stupid simple.
评论 #10410656 未加载
awinter-pyover 9 years ago
I want to see an &#x27;algebra&#x27; approach to packaging -- strongly-typed packaging modeled as types and operators. Package and import systems present a lot of surprising behaviors and edge cases in part because they&#x27;re not well-defined enough to be well-documented.<p>Also, packaging systems need to get better at integrating with VCS versioning. The elm packaging system goes beyond this to automatically bump versions when your api changes <a href="https:&#x2F;&#x2F;github.com&#x2F;elm-lang&#x2F;elm-package#version-rules" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;elm-lang&#x2F;elm-package#version-rules</a>, whoa cool. But at minimum, make a note of git&#x2F;hg id.<p>The &#x27;just use npm&#x27; approach is interesting too <a href="http:&#x2F;&#x2F;dominictarr.com&#x2F;post&#x2F;25516279897&#x2F;why-you-should-never-write-a-package-manager" rel="nofollow">http:&#x2F;&#x2F;dominictarr.com&#x2F;post&#x2F;25516279897&#x2F;why-you-should-never...</a>. NPM&#x27;s approach to &#x27;dependencies of dependencies&#x27; has a lot of fans.<p>Finally: part of the problem here is that there&#x27;s no package manager for C. The author dances around this in saying &#x27;C extensions are hard, we should ignore them&#x27;. If we had a good C package manager a lot of work linux distros do in supporting C libraries would be simplified and builds of everything would be <i>so much</i> more portable.
评论 #10410803 未加载
评论 #10411133 未加载
评论 #10410761 未加载
objectifiedover 9 years ago
I think that for the majority of use cases when it comes to packaging and shipping Python programs, we really want to ship OS packages. We don&#x27;t want users (that includes sysadmins&#x2F;operations teams in this case) to deal with having pip available, understanding what virtualenv is, and so on. We just want them to be able to use apt, yum, or whatever they experience as most common on their system. Yet at the same time, we don&#x27;t want our application to take in regard ancient versions of OS packages to depend on.<p>I think in the end we want a self contained OS package, to offer a sane install method for any target OS. Why would users have to make a difference between e.g. a C program that they install through apt, or a Python program that all of a sudden requires a completely different way of installing?<p>Here are some interesting thoughts on it: <a href="https:&#x2F;&#x2F;hynek.me&#x2F;articles&#x2F;python-app-deployment-with-native-packages&#x2F;" rel="nofollow">https:&#x2F;&#x2F;hynek.me&#x2F;articles&#x2F;python-app-deployment-with-native-...</a><p>And here&#x27;s a tool that I wrote to be able to wrap up a virtualenv into an OS package, while specifying a precise list of OS dependencies. It&#x27;s not perfect by far, and admittedly a bit crude, but it already does the job well for some real life &quot;production&quot; stuff. It&#x27;s called vdist (virtualenv distribution).<p><a href="https:&#x2F;&#x2F;github.com&#x2F;objectified&#x2F;vdist" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;objectified&#x2F;vdist</a><p>Documentation is here:<p><a href="https:&#x2F;&#x2F;vdist.readthedocs.org&#x2F;en&#x2F;latest&#x2F;" rel="nofollow">https:&#x2F;&#x2F;vdist.readthedocs.org&#x2F;en&#x2F;latest&#x2F;</a>
评论 #10412261 未加载
dikaiosuneover 9 years ago
Having used the combination of Gradle and Capsule in Java-land, I desperately want to equivalent for Python. A tool that can declarative state dependencies and bundle them into a single file (akin to a static binary) which has a single entry point and all dependencies included. For libraries, the JAR model works well too.
评论 #10410877 未加载
评论 #10411163 未加载
评论 #10410817 未加载
Othersover 9 years ago
<a href="https:&#x2F;&#x2F;xkcd.com&#x2F;927&#x2F;" rel="nofollow">https:&#x2F;&#x2F;xkcd.com&#x2F;927&#x2F;</a>
mkhpalmover 9 years ago
I often skip all that and use native packages with pyinstall, dh_python, and dh_virtualenv (from spotify)<p>For me its much cleaner and controlled than distutils plus the knowledge is largely transferable to other stacks. I wrap our node, java, and ruby applications using the same tooling as well as any services we depend on. (kafka, kibana, spark, etc) After that I can pass off making images, containers, installing, managing stuff, whatever onto anybody in IT or operations that knows linux.<p>Obviously that wouldn&#x27;t work well for everybody but it sure beats trying to manage 20 different ways to do everything in my case.
pjtrover 9 years ago
I&#x27;d prefer docs. And please: PEPs are NOT docs!<p>In the last couple of years setuptools changed &#x2F; broke so many things that previously worked without documenting anything. Even trying to figure out what the relevant docs would be, and who maintains them is hard. As Nick Coghlan mentions in the comments: Even the people maintaining this stuff have no idea what it all does.<p>And still, the correct (but incredibly boring and frustrating) solution is for someone to sit down and figure this stuff out and document it. Definitely not me though. :)
moonbugover 9 years ago
Use Anaconda and be done with it.
评论 #10410688 未加载
harlowjaover 9 years ago
Openstack uses PBR (no not the beer):<p>&quot;&quot;&quot;Python Build Reasonableness&quot;&quot;&quot;<p><a href="http:&#x2F;&#x2F;docs.openstack.org&#x2F;developer&#x2F;pbr&#x2F;" rel="nofollow">http:&#x2F;&#x2F;docs.openstack.org&#x2F;developer&#x2F;pbr&#x2F;</a><p>It is pretty nice and helps a ton with packaging sanity...
sigsergvover 9 years ago
We don&#x27;t need to get rid of setuptools&#x2F;distutils&#x2F;etc. We need something like debhelper but for python, set of tools that generates and maintains proper setup.py and relatives.