“Python's batteries are leaking”

Note that similar issues were raised with Ruby stdlib, which is being addressed in part with “Gemification” of stdlib, so that all of stdlib (targeted for 3.0, though it&#x27;s been going on since 2.4)[0] is being moved out to externally-updatable packages that are included by default (default and bundled gems), so that it is still “batteries included” but the batteries are at least replaceable.<p>Amber&#x27;s suggestion seems to be in the same direction (though perhaps not as extreme.)<p>[0] <a href="https:&#x2F;&#x2F;www.slideshare.net&#x2F;mobile&#x2F;hsbt&#x2F;gemification-for-ruby-2530" rel="nofollow">https:&#x2F;&#x2F;www.slideshare.net&#x2F;mobile&#x2F;hsbt&#x2F;gemification-for-ruby...</a>

If your project has any third-party dependencies, and so (nowadays) you&#x27;re going to set up requirements.txt and virtualenv and whatever anyway, I can see that you&#x27;re going to think things like &quot;this XML parser in the standard library is just getting in the way; I can get a better one from PyPi&quot;.<p>But I think a lot of the value of a large standard library is that it makes it possible to write more programs without needing that first third-party dependency.<p>This is particularly good if you&#x27;re using Python as a piece of glue inside something that isn&#x27;t principally a Python project. It&#x27;s easy to imagine a Python script doing a little bit of code generation in the build system of some larger project that wants to parse an XML file.

She seems to be advocating that Python do pretty much what Perl has ended up doing, which is &quot;we have some batteries, but we haven&#x27;t been adding new ones for a decade or more&quot;.<p>The reasons are similar, it&#x27;s a constant drag on core compiler development to need to support various batteries included that most core contributors aren&#x27;t going to care about, so it&#x27;s easier to tell people &quot;use CPAN&quot;.<p>There was even talk of &quot;distros&quot; for the interpreter. Where the core bits would be similar to what Linux is, and all the batteries would be provide as collections of add-on packages.<p>Strangely enough these efforts seem to stop at OS distributors. They really seem to like to install just the one &quot;compiler&quot;, and wouldn&#x27;t stand for a project like Perl or Python telling them &quot;we mean for you to distribute the core compiler plus these 100 packages, because that&#x27;s what forms our &#x27;language&#x27;&quot;. &quot;Strangely&quot; because you&#x27;d think they&#x27;d be the best positioned to make easy work of packaging up such a thing, and it shouldn&#x27;t in principle make a difference if you need to install 100 RPMs &#x2F; APTs by default.

The Python standard library has been a huge help for me. Evaluating which third party packages to trust and handling updates is a hassle. (Would love a solution for this. Does anyone have a curated version of PyPI?) I’m surprised that people want to slim it down other than for performance on a more constrained system.<p>As an aside, why doesn’t the Python standard library extend&#x2F;replace features with code from successful packages like Requests? Tried it and it didn’t work? Too much bloat? Already got too much on the to-do list?

Hawk Owl is a _fantastic_ dev. She was the main force behind the twisted 2-&gt;3 transition. But because she is, she is missing the point of batteries included.<p>Asyncio is in the stdlib so that we have an official lib and API. The main benefit is that most people now, when looking for async, are not wondering about twisted or gevent or tornado. Most just go asyncio. Most dev efforts go to asyncio. It&#x27;s the end of the great async war. Is it perfect ? No. And I don&#x27;t care. It&#x27;s one thing less to worry about. For those who know what they are doing, you can still choose and pip install twisted, but most people don&#x27;t, and that&#x27;s solved. Before that, just choosing the lib was a nighmare, as basically it&#x27;s a definitive call. Out it on pypi, even with a &quot;stdlib&quot; tag, we go back to the 200X era. And it was not fun.<p>And the goal for having things like xml&#x2F;sqlite&#x2F;ssl without installing anything makes python very useful in a load of situations where you can&#x27;t install stuff. Sometime you are offline. Sometime you are in a restricted env. Sometime you are not on your machine. Sometime your security protocol is hell. Don&#x27;t assume people use Python as we do, from our comfortable dev laptop driven by the knowledge of our craft. Python is used in banks, by scientists, in schools, by kids, by poor people in the third world, by geographers and pentesters. The python user base is incredibly diverse, it&#x27;s why it&#x27;s so popular: it fits a lot of use cases.<p>So I see the benefit of having a side version of official modules we can pip install that can move faster. I see the benefit of cleaning the stdlib of old stuff, like the wave module, Template or @static.<p>But I&#x27;m glad I don&#x27;t have anything to install to generate a uuid or unzip stuff. I&#x27;m glad I don&#x27;t have to worry about twisted anymore (depiste that I did write a book on the topic !).<p>Also, pip install is NOT simple when you learn the language. I have to spend some time in the classroom, even with adult professionals, to explain the various subtleties of site-packages, import path, py -x on windows, python-pip on linux, -m, virtualenv, header files, etc. before my students become autonomous with it. Without a teachers, this turn into months of bad practices and frustrations.<p>You&#x27;d have to fix that first, way, way before moving stuff to pypi. I do think it should be high priority actually: it affects way more than pip.

It&#x27;s funny to me that they&#x27;re making a point that PyPI is better than core, because actually I think PyPI has created a rather crap ecosystem. The non-hierarchial organization of packages, the lack of curation, lack of inheriting past functionality and extending it as more standard functionality, etc has resulted in a confusing sprawl of packages with duplicate, incompatible, buggy functionality. It&#x27;s a bit like Linux internals; it&#x27;s grown haggard over time, isn&#x27;t organized well, is badly documented, and so it&#x27;s difficult to pick it up and use it without stumbling over a decade or more of stale documentation and obsolete software.<p>Perl has a <i>much</i> better set of modules that extend standard functionality, which considering how much flack Perl gets for being hard to read, is rather funny. Rather than every new feature being its own independent project, most of the useful modules inherit a parent and follow the same convention, leading to <i>very</i> simple and easy to use extensions. And Perl Core isn&#x27;t all that great, but it does have <i>some</i> batteries included, and everything else is extended easily and in a more standard manner by CPAN.

I agree with Amber’s point that more stuff should be moved from the standard library to PyPI. I made my first pull request to CPython during the development sprints this year, and it’s honestly not the best experience. Everything is built from scratch in CI after every commit, even a documentation change. There’s nowhere near enough CI builds and pipelines for everything Python supports. Pull requests are outstanding for several months, and there’s at least a thousand PRs open when I checked this morning.<p>I’m not sure if Python’s ideal solution is to reduce stdlib and have endorsed packages in PyPI, but it would be an improvement over the current process.

The story of Python 2 to Python 3 migration, in a nutshell:<p>&gt; Van Rossum argued instead that if the Twisted team wants the ecosystem to evolve, they should stop supporting older Python versions and force users to upgrade. Brown acknowledged this point, but said half of Twisted users are still on Python 2 and it is difficult to abandon them. The debate at this point became personal for Van Rossum, and he left angrily.

When I first used python like 20 years ago I was blown away by how much functionality was blown in, and it can be annoying using languages where even the most basic functionality involves downloading 50 packages from the internet, but on the other hand the standard library does seem to be a mess now.

&gt; She thinks that some bugs in the standard library will never be fixed.<p>This is actually an interesting paradox to be in, and one that Linus Torvalds recently commented on. His focus, like Guido’s, is the user and even fixing a bug can break the user.<p><a href="https:&#x2F;&#x2F;lkml.org&#x2F;lkml&#x2F;2018&#x2F;8&#x2F;3&#x2F;621" rel="nofollow">https:&#x2F;&#x2F;lkml.org&#x2F;lkml&#x2F;2018&#x2F;8&#x2F;3&#x2F;621</a>

Guido is a good dude, through-and-through, despite his perhaps bad behavior here.<p>Amber is nothing short of an open source hero, having brought Twisted, one of the best open source projects in the world, to new heights. Her insights are as important as anyone in the python community, and after six consecutive PyCons sprinting at the Twisted table (including literally in a chair with Amber to my left and Glyph to my right earlier this month), I consider Amber&#x27;s voice to be one of the truest and clearest among the leadership of the language into the future.<p>Amber and Guido are both beautiful human beings.<p>In the dispute that is the topic of this blog post, Amber is basically totally right. Moreover, the distinction has less to do with any kind of nagging python 2 holdover than this article suggests. The standard lib&#x27;s role as a place where code goes to die is a view that is widely held and accurate for many cases.<p>The following question went unanswered during the Steering Council Q&amp;A:<p>&quot;Every feature request has a constituency of people who want it. Is there a constituency for conservatism and minimalism?&quot;<p>...and that&#x27;s really what this whole thing is about.

It took me an hour to create a program that tails some logs and alerts when it doesn&#x27;t receive any logs for a given amount of time. For this task I did not even need to leave the asyncio module. It lets you create subprocesses and execute call_later on the event loop in order to simulate a heartbeat while reading the output of tail at the same time.<p>Did asyncio module feel bloated? It certainly did. It seems like every module from subprocess to networking to io is crammed into it.<p>On the other hand, did it get the job done without resorting to any packages or threading? Yep, and that is pretty powerful and rare.

&gt; six is non-optional for writing code for Python 2 and 3<p>I maintain a Python 2 &amp; 3 compatible project that has no external dependencies.

&gt; Brown called out the XML parser and tkinter in particular for making the standard library larger and harder to build, burdening all programmers for the sake of a few<p>Tkinter needs to go...There is very little reason except for the legacy ones, why it needs to be there still...

I clicked on this link think &quot;uh oh standard ill informed rant post&quot;<p>However brown has solid good points.<p>The brilliant selling point of python is the massive standard lib. If the quality of the libraries fall, then python&#x27;s use as a tool drops dramatically.<p>One of Node&#x27;s massive failures is that is has no standard lib.

&gt; The debate at this point became personal for Van Rossum, and he left angrily.<p>Guido should stop acting like a child. Listening to people, hearing them out - even when it’s uncomfortable - is the mark of a good leader.<p>I tell new PMs “this is the best job in the world 90% of the time but the other 10% is eating shit with a smile”

Isn&#x27;t part of the issue mixing general purpose code that doesn&#x27;t change very often (the kind that belongs in a standard library) with code that changes all the time (the kind that belongs on PyPI)?<p>I remember one of Go&#x27;s core devs voicing some of the same concerns regarding the SMTP-library [0] a while back.<p><a href="https:&#x2F;&#x2F;golang.org&#x2F;pkg&#x2F;net&#x2F;smtp&#x2F;" rel="nofollow">https:&#x2F;&#x2F;golang.org&#x2F;pkg&#x2F;net&#x2F;smtp&#x2F;</a>

I may be terribly wrong about this, but I would think that in general if someone makes an improvement to something you make then you would want to integrate that person and their improvement in some way. Treat like any other update: mention it in the version notes and warn about compatibility of code using the previous version. I know that renovation can suck but it&#x27;s something that we need to be doing. A comprehensive stdlib means that once you&#x27;ve got it, you don&#x27;t need to worry about being able to access packages along the way - how are you going to download a package if you can&#x27;t connect to the repository? How much can you trust a third-party dev vs the core team?<p>If a package is really niche, it may not make sense to put in the integration work. But for a package that is used by a significant majority in a general application - why would you want to keep it separate if it is so much better?<p>I am ignoring human interaction here - there are probably of dozens of answers to that question if you count personal motivations.

I&#x27;ve wondered about standard libraries for a while now. What happens if you discover a security vulnerability in your stdlib? Presumably you&#x27;d have to bump the language version, deploy it out and beg users to upgrade. Except, users don&#x27;t upgrade stuff. While if you version the standard library, every new project will get the newer version of the standard library. Sure, there&#x27;s space tradeoffs, though you could offer a manual linking option. But at the very least, the amount of new projects with the vulnerability will be next to nil.<p>And what if the standard library just gets dated? Take Node for instance. The fs module has a whole bunch of outdated callback based functions. Sure, you can wrap them in promisify, but it sucks that we have these outdated functions stuck around forever.<p>There&#x27;s definitely tradeoffs with package&#x2F;dependency multiplication, but I don&#x27;t think standard libraries are as clear cut as people make them out to be.

IMO, Golang does the best job of maintaining a high quality standard library. I disagree that modules should be moved into the external package ecosystem. However, Go isn&#x27;t preinstalled on most systems like Python.<p>I have to develop enterprise software that runs across a wide range of platforms, and being able to take advantage of the fact that Python is pre-installed on all of these systems with its standard library is a godsend.

Some of her concerns do make sense, but the using, “a lot of our users stil use python 2.x” as a justification, in 2019, is ridiculous. Those same users had years to adopt&#x2F;change the code base.

I think it is important to notice here that a lot of the tension here is also Twisted vs. asyncio.<p>I&#x27;ve never liked asyncio, while Twisted felt natural to me. So I would agree that an inferior solution has been pushed heavily in the stdlib and also to the syntax level.<p>Moving the entire stdlib to PyPI is of course entirely foolish and would destroy Python.

I have mixed feelings about this because I&#x27;ve seen both parts of the same situation.<p>In certain situations, I&#x27;ve been working with a python interpreter on a RHEL machine where pip was not installed (and I was not allowed to install it as well as make other modification: the machine was owned by the client and I had to work with what I had available).<p>- having some basic functionality in the core libraries was a godsend because I could work with that, even though it was not &quot;ergonomic&quot;<p>- not being ergonomic, it was a &quot;poor experience&quot; (and certainly not optimized or anything nice to see).

&gt; Standard Library Modules Crowd Out Innovation<p>This heading is the essential problem in innovation writ large: some giant can ignore you and squash you without any effort at all, without even considering your existence.

There&#x27;s an important tradeoff going on between library code that is (in theory) trusted and library code that is less trusted but has other advantages like solving a problem better or being a solution to problems that the most trust code can&#x27;t solve.<p>Right now the equilibrium in this tug-of-war is that a certain set of functionality comes by default in the python standard library and everything else is just a package that you can install.<p>Obviously from a dev point of view it&#x27;s a hassle to have to decide which of two or more packages for X is best, the pythonic way would be that there should be one and only one package for X. Of course at the cutting edge there have to be competing packages because there needs to be room for innovation.<p>But obviously not everything can be in the python standard library.<p>Not really sure what the solution is, but maybe there should be tiers of packages, with &quot;Tier 1&quot; being standard library, &quot;Tier 2&quot; having some kind of official stamp that it has been security audited to a certain standard, that Python has some control over who gets to modify it and why etc. Then maybe &quot;Tier 3&quot; could cover everything else, i.e. any bob random can go make a package on PyPi and it&#x27;s Tier 3.<p>In addition, the process of going from Tier 3 to Tier 2 would give people a chance to winnow libraries down to one way of doing each thing at the Tier 2 level.<p>This might not be realistic but it&#x27;s what my gut is telling me. C&amp;C welcome.

Wow this is not the conduct I expect from a language creator. I don’t care if you’re Albert Einstein. Humility and being able to take criticism is far more admirable to me.

&gt; Brown went further adding that because few Python core developers are also major library maintainers, library authors’ complaints are devalued or ignored.<p>PHP has a similar issue to this. The people writing C were not using the language. The best example is PDO. A lot of C was written, but it was essentially abandonware because the PHP users could not make any changes without getting the C maintainers to both agree and have the time.

A lot of this is side effects of the Python 2 vs 3 schism imo. If it weren&#x27;t for that situation, practically everyone would on be 3.x, and supporting older versions wouldn&#x27;t be important, so package maintainership wouldn&#x27;t be as difficult.<p>Put another way, the whole Python universe from my point of view has become a cautionary tale about breaking changes. Given Python&#x27;s popularity, this might be an unpopular opinion, but I have yet to find someone who loves Python who still loves it as much when they discover other newer languages (I&#x27;m sure you exist, I just haven&#x27;t met you!).<p>Python is having its time in the sun really because it is a default install for most unix distributions, so even people stuck in government labs can use it because 2.7 is already installed. Even apt depends on it via the debian software-properties package, so it isn&#x27;t going anywhere any time soon.<p>The real question is how many people would use Python if it was as little known as, say, Elixir.

stdlib acts as the foundation upon which a lot of the 3rd party stuff us built - it&#x27;s a feature to have it move slowly and not break often.

What&#x27;s the point of saying that a standard library feature was not added soon enough? Nobody can go back in time and add it earlier. I can complain to death about features missing from C++11 or I can start using C++14.

I think the best model i have seen for a lean stdlib has been that of Golang. You have the standard lib and then you have the packages under Golang.org&#x2F;x&#x2F; which are experimental packages that sometimes end up being merged into the core language. The stuff which is not in the stdlib (TOML, yaml etc.) have been supported very well by community packages.

I got lost on the &quot;we have to back port to python 2.7&quot; argument.<p>Force the upgrade already. Code still on 2.7 if still useful can be upgraded.

Good points, some things on the standard library are just painful<p>My &quot;favourite&quot; library quirk. socket.fromfd is only available on &quot;Unix&quot; on Python 2.X, that was fixed in Python 3.X<p>The worse offender being the logging library. It&#x27;s the least pythonic thing in the whole std library (ok maybe ABC is worse, but oh well)

Most of this post seems to revolve around the idea of py2 having an outdated standard library.<p><a href="https:&#x2F;&#x2F;pythonclock.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;pythonclock.org&#x2F;</a> has Py2 reaching EOL in 7 months. Realistically I&#x27;d say the time passed years ago.<p>py3 is over ten years old now. We&#x27;re not talking about some new unstable piece of kit, I&#x27;d imagine that a large percentage, perhaps even 50%, of the HN audience started their career after the transition had already started.

Why, time and time again, does Guido seem incapable of reasonable debate, or ideas that challenge his own? It&#x27;s completely rude to interrupt a presenter with &#x27;what is your point?&#x27;<p>Years ago I was in contact with the author of Nuitka, who was very excited to share his work thus far. During his presentation, Guido kept huffing and making snide comments under his breath. All because he disagrees with the premise behind Nuitka.<p>I like Python, and can appreciate his work and contributions. That said, I can&#x27;t help think the community can become less toxic without him as BDFL.

At this point my MVP version of python is Anaconda-latest.

It seems weird to me that they specifically call out only Guido&#x27;s behavior. I feel like there was probably a lot of context lost in &quot;he left angrily&quot; and its a bit unfair bordering on disingenuous.

Its never too late to rewrite python

I like python, its a nice simple language that you can use to pump out a proof of concept real quick with little hassle, but for full on production I avoid it. But I think this is a larger trend in programming, in my opinion the majority of programmers are super lazy. Everyone is in a mad dash to get the cool new thing out so they just slap a bunch of dependencies on it and damn the consequences of developer debt down the road. More people would rather roll with an MVP as the final product than build something from scratch that&#x27;s more robust, resilient, and efficient. Then after a while you have this huge mess of old broken code that can&#x27;t be fixed anymore and just needs to be redone from scratch.<p>Sure you are going to need to redo code anyways from scratch eventually. But, programmers like I mentioned, which is surprisingly a huge chunk, make problems worse for themselves throughout the lifetime of the code by being short sighted and stamping their approval on code their too lazy to rewrite because their boss doesn&#x27;t know any better.

Why is it so difficult to admit Node.js did the package thing right, by keeping a local folder just for the app, isolation from other apps with zero effort?

Its not only that python stdlib libraries are getting old and sometimes appear to be unmaintained, some of the more recent additions and changes are lacking.

I was rather shocked to find that python didn’t have a full-featured crypto library included in its standard lib. The alternatives all ended up being unmaintained or maintained by small groups (which makes trust in the soundness difficult). I tapped our security team, who were in disbelief, but ultimately they gave up to and I wrote the software in go instead.

I had to code in Python recently and had PTSD over all the syntactically significant whitespace. I have no idea how I ever found productivity in the language coming back to it now with fresher eyes. Refactoring, editing, and writing new code feels like such a drag.