The Python Standard Library - Where Modules Go To Die

I'm not really a Python programmer, but a hacker who occasionally has cause to pick up Python scripts and do stuff with them. Perhaps I've been unlucky, but every time I've done this, it's turned into a profoundly frustrating exercise. There have always been dependencies outside the standard library, and those have had dependencies -- which more often than not, are incompatible with whatever version of Python my environment is set up for. I've frequently run across scripts with dependencies that somehow only execute in <i>mutually incompatible versions of Python</i>, which always makes for an exceedingly aggravating day of programming.<p>As much as people love to bash PHP -- and I agree that it's pretty awful as a <i>language</i> -- its standard library is so comprehensive, backwards-compatible, and superbly-documented that I have <i>never</i> had a comparably aggravating experience with it. The same is true of Javascript: a language with warts, but whenever I try something, it Just Works.<p>Like I say, perhaps I've just been unlucky, but my distinct impression of Python has been that it's a beautiful language surrounded by a particularly problematic ecosystem of incompatible libraries and sparse documentation. I suspect that the Python community would benefit from paying less attention to the purity of the language, and a lot more attention to the quality of everything surrounding it.

The fact that the standard library is well-maintained, carefully debugged, and backward-compatible is a far stronger indicator of Python's awesomeness than the existence of shiny, new libraries. Hackers naturally gravitate toward high-visibility projects with brave horizons and bold scopes. By contrast, it is incredibly hard to find the motivation to update, for the umpteenth time, a warty API -- and that's precisely the reason why contributions of the latter sort are the truer test of the vitality of a language's ecosystem.

Agreed with the OP. The following is a shameless plug:<p>Python's ConfigParser module is a pain to use. It provides no validation, only supports a limited number of types of data you can retrieve, etc. Similarly, getopt vs optparse vs argparse is a mess. getopt is universal: not only is it going to be in all versions of Python, but it is also the same library available in virtually every other language. The problem with it is that it is not declarative, so you will typically see a giant if/elif statement that goes with it. argparse/optparse are better, but aren't universal even between versions of Python, though argparse has been backported and is available via pypi.<p>To unify all this into one convenient module, I ended up writing <a href="http://ipartola.github.com/groper/" rel="nofollow">http://ipartola.github.com/groper/</a>. groper lets you specify your parameters declaratively, and if you specify defaults, use them right away without having to create/modify a config file. It automatically figures out the priority of arguments: cmd &#62; config &#62; defaults. It also has some niceties such as the ability to automatically generate usage strings, give the user intelligent error messages, generate sample config files, etc.

The permalink for the article is <a href="http://www.leancrew.com/all-this/2012/04/where-modules-go-to-die/" rel="nofollow">http://www.leancrew.com/all-this/2012/04/where-modules-go-to...</a> (the posted link is actually the home page of the blog).

I've heard core Python developers tell people not to worry about getting a module into the stdlib. The problem being that once the module is there it won't be able to change much. APIs have the exact same problem. If you change it, you're changing other peoples' software. Tight coupling.<p>Is it a terrible way to write software? Maybe... but perhaps that's a different discussion.<p>I think the requests library is amazing. It has a much more simple API than urllib/urllib2. Does it need to replace those modules in the stdlib? I hope not!<p>There are only three reasons I would write a module/package that depended solely on stdlib:<p><pre><code> 1. The module/package would be distributed primarily through package management systems. 2. The installation of my module needs to avoid depending on anything else outside of a base python installation. 3. The module or package will need to be supported for a long time and will likely not be updated frequently. </code></pre> The first case is because you can't control what versions of third-party libraries the package manager will make available. Some might run your setuptools script while others may not. It's just easier to live with the cruft/warts of the stdlib and be sure that they'll always be there.<p>The second case covers a very unique situation. Modules and libraries written with this constraint are typically targeting one of two different kinds of developers. The first are the beginners who may not know about development environments and versioning. The other are experienced developers who want a minimalist script for their little one-off utility. Both should require zero dependency installation if possible.<p>The final case is harder to define up front. If you're writing something that you expect to run for a long time and receive little maintenance (ie: cron scripts, tools, etc) then you don't want to deal with API updates breaking your code. Fire and forget is what a long-term stable API gets you.

&#62; An overstatement, certainly, but with more than a germ a truth. Once a library is enshrined in the standard set, it can’t change radically because too many programs rely on it—and its bugs, idiosyncrasies, and complications—remaining stable.<p>That's a problem inherent in the standardization process, though - it's all but contradictory to have something be both 'standard' and 'continuously improving'.<p>Once something enters the standard, does anyone propose a better way of removing cruft without constantly deprecating everything, rendering the concept of a 'standard' somewhat meaningless?

The thing with python standard library is that it is crapily documented imho. I often can't make heads or tails of it, while I have a much simpler time with any other language (you name it: Java, Ruby, PHP, C, Scala, Lisp, ...).

Yes, for a language as widely deployed and used as Python, retaining backwards compatibility and stability is more important than adding new and shiny tools to the stdlib at a faster pace. Users rely on the fact that a module in stdlib will remain there and will remain stable for a long time. More modules means more maintainers, and Python is an open-source project developed by volunteers. It's that simple.<p>I'm not sure what the solution this article proposes is. The tradeoff between "coolness" and "stability" is inherently difficult, and I'm sure Python is not the only language "suffering" from it.<p>After all, it's quite easy to install a new Python module, and not much harder to distribute it with your application (for web apps it's even easier), so what is the problem?

It's funny, I've had the opposite problem. I was trying to write an IRC bot in Python, noted there didn't seem to be a standard library module for the IRC protocol, and so found myself looking at this:<p><a href="http://pypi.python.org/pypi?%3Aaction=search&#38;term=IRC" rel="nofollow">http://pypi.python.org/pypi?%3Aaction=search&#38;term=IRC</a><p>That's 400+ results - at least 20 of which are actually IRC protocol modules. There's no way of telling how mature each one actually is 'til you download it. It turned out the first three I tried were undocumented, buggy, incomplete, or otherwise no good.<p>So I gave up on PyPi and hacked it as an xchat plugin instead.<p>----------------<p>Perhaps the way forward would be styling your package repo after, say, addons.mozilla.org -- add just enough community functionality (as in ratings/reviews/"times downloaded" counters/etc) to allow the occasional gems to rise to the top of the muck. Once one solution for a given problem has been established as the best (well, most popular), that'll get more eyeballs on its internals as well, and it'll only increase its lead until it's de facto standard -- but the possibility is still there for a newcomer to dethrone it if it's genuinely better. And meanwhile, both can exist side-by-side without causing ugly compatibility issues.

This happens in every language. I dont think it's that big of a deal.<p>In all honesty, you could continue to maintain a package outside of stdlib, and just require a newer version (which gets installed via the standard packaging tools). This type of behavior isn't well defined in Python, but it's not unrealistic to think it could happen.

This has kind of happened in Ruby, too.<p>Fortunately, Ruby gems are super easy to install and the standard library got some much-needed spring cleaning in 1.9.<p>Python could use the same. There have been many times where I've wanted to do some simple task that would be made easier with an external library (like Requests) but I'm not going to bother dealing with the Python module install pain for a one-off task.

Every language's standard library needs a "current best practices" concept, even if it's just a well-maintained document and not something structural like a special namespace.<p>I think the Python "decorator" concept goes a long way toward cleaning up code. Basically you can add a decorator to a routine that you've deprecated so that it will complain if it's actually used (you can even include advice on what would be a good replacement call).<p>As far as cleaning up what's installed as standard, it's not really practical to remove anything (the fact that it stays is one of the attractive things about Python in old code bases). What you can do though is define a preferred namespace, e.g. "preferred"; this would physically contain only those libraries that are recommended, and perhaps even forked copies of modules that only contain the <i>functions</i> that should be used. This gives programs the option to explicitly import from "preferred" and request purity over long-term stability.

Permalink to this post: <a href="http://www.leancrew.com/all-this/2012/04/where-modules-go-to-die/" rel="nofollow">http://www.leancrew.com/all-this/2012/04/where-modules-go-to...</a><p>(The current article link goes to the front page of the author's blog.)