TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Freezing Python’s Dependency Hell

200 点作者 ammaristotle将近 7 年前

30 条评论

abhishekjha将近 7 年前
What&#x27;s wrong with pipenv? I am genuinely curious.<p>On local :<p><pre><code> mkdir my_project_directory cd my_project_directory export PIPENV_VENV_IN_PROJECT=1 (To make the virtual environment folder determininstic(.venv&#x2F;) otherwise you will get a hash based directory(my_project_directory-some-hash-value) which might not be suitable for automatic deployments in applications like docker. I don&#x27;t know why this is not default.) pipenv --python 3.6 (or any particular version number) pipenv install numpy scipy pandas matplotlib requests pipenv graph (Gives me a dependency graph) git add . git commit -a -S -m &quot;init&quot; git push </code></pre> On remote :<p><pre><code> git clone url&#x2F;my_project_directory cd my_project_directory export PIPENV_VENV_IN_PROJECT=1 pipenv install pipenv shell pipenv graph </code></pre> Is this workflow not enough? I have recently started using pipenv after a lot of struggle. The only issue I have is, Pycharm doesn&#x27;t allow native pipenv initialisation. I always end up creating an environment manually and then importing the project. Pycharm does detect the environment though.
评论 #17608780 未加载
评论 #17608591 未加载
评论 #17612172 未加载
评论 #17609844 未加载
评论 #17609868 未加载
评论 #17608563 未加载
评论 #17608477 未加载
评论 #17609102 未加载
评论 #17609902 未加载
评论 #17608651 未加载
评论 #17609362 未加载
评论 #17616887 未加载
评论 #17610243 未加载
bjpbakker将近 7 年前
I feel that all of these language specific solutions still only solve halve the problem. Your code depends on a lot more than _just_ the python libraries. And often this is exactly what makes projects break on different systems.<p>Let me make another suggestion: nixpkgs [0] it helps to define exactly that fixed set of dependencies. Not just on published version number, but on the actual source code _and_ all it&#x27;s dependencies.<p>[0] - <a href="https:&#x2F;&#x2F;nixos.org&#x2F;nixpkgs&#x2F;" rel="nofollow">https:&#x2F;&#x2F;nixos.org&#x2F;nixpkgs&#x2F;</a>
评论 #17611119 未加载
ris将近 7 年前
Here we go again. The source of the problems in in toy package managers (and I include all language package managers here) is not just the package managers themselves, it&#x27;s the &quot;version soup&quot; philosophy they present to the user. Not daring to risk displeasing the user, they will take orders akin to &quot;I&#x27;d like version 1.2.3 of package a, version 31.4.1q of package b, version 0.271 of package c, version 141 of package d...&quot;, barely giving a thought to inter-version dependencies of the result.<p>Unfortunately, software <i>does not work this way</i>. You cannot just ask for an arbitrary combination of versions and rely on it to work. Conflicts and diamond dependencies lurk everywhere.<p><i>Sensible</i> package systems (see specifically Nix &amp; nixpkgs) have realized this and follow a &quot;distribution&quot; model where they periodically settle upon a collection of versions of packages which <i>generally</i> are known to work pretty well together (nixpkgs in particular tries to ensure packages&#x27; test suites pass in any environment they&#x27;re going to be installed in). A responsible package distribution will also take it upon themselves to maintain these versions with (often backported) security fixes so that it&#x27;s no worry sticking with a selection of versions for ~6 months.<p>However, I can&#x27;t say I&#x27;m particularly surprised that these systems tend to lose out in popularity to the seductively &quot;easy&quot; systems that try to promise the user the moon.
评论 #17614079 未加载
评论 #17613083 未加载
zedr将近 7 年前
Using a local virtual environment and then building a Docker image removes most of the headaches. I also bundle a Makefile with simple targets. See this as an example: <a href="https:&#x2F;&#x2F;github.com&#x2F;zedr&#x2F;cffi_test&#x2F;blob&#x2F;master&#x2F;Makefile" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;zedr&#x2F;cffi_test&#x2F;blob&#x2F;master&#x2F;Makefile</a> New projects are created from a template using Cookiecutter.<p>It isn&#x27;t really so bad in 2018, but I do have a lot of scars from the old days, most of them caused by zc.buildout.<p>The secret is using, as the article mentions, a custom virtual env for each instance of the project. I never found the need for stateful tooling like Virtualenvwrapper.
评论 #17608934 未加载
aequitas将近 7 年前
&quot;Pipfile looks promising for managing package dependencies, but is under active development. We may adopt this as an alternative if&#x2F;when it reaches maturity, but for the time being we use requirements.txt.&quot;<p>If I where given the choice between community supported&#x2F;in development Pipfile&#x2F;pipenv or the 3rd party supported yet-another-package-manager lore to get those best practices my money would be on Pipfile&#x2F;pipenv. I&#x27;ve been using it for many project now and besides some minor annoyances (eg: the maintainer&#x27;s love for color output that is not form follow function) it has been a great tool.
michaelmcmillan将近 7 年前
Never had a problem with dependencies in Python. Just keep it simple.<p>When starting a new project:<p><pre><code> virtualenv venv -p *path-to-python-version-you-want* .&#x2F;venv&#x2F;bin&#x2F;pip install *name-of-package* </code></pre> When running that project:<p><pre><code> .&#x2F;venv&#x2F;bin&#x2F;python *name-of-python-file* </code></pre> Many people don&#x27;t realize that the venv&#x2F;bin&#x2F; contains all the relevant binaries with the right library path&#x27;s out of the box.
评论 #17608423 未加载
评论 #17608791 未加载
pytyper2将近 7 年前
I&#x27;m not sure why the scientists don&#x27;t use VMs and simply save the virtual disk files? That would at the very least allow them to verify the settings at a later date. Fresh install reproducibility doesn&#x27;t seem necessary to verify experimental findings as long as the original vm is available to boot up.
评论 #17610223 未加载
评论 #17611305 未加载
nickjj将近 7 年前
1. Build Docker image out of requirements.txt<p>2. Develop application<p>3. Repeat 1-2 until ready to deploy<p>4. Run Docker image in production with same dependencies as development<p>5. ??<p>6. Profit!<p>As long as you don&#x27;t rebuild in between steps 3-4, you&#x27;ll have the same set of dependencies down to the exact patch level.
评论 #17608911 未加载
评论 #17608647 未加载
评论 #17610603 未加载
sandGorgon将近 7 年前
genuine question - is nobody using anaconda&#x2F;conda in production ? I have found the binary install experience in conda far more pleasant than in anything else.<p>Going forward, the trend is going to be pipenv+manylinux (<a href="https:&#x2F;&#x2F;github.com&#x2F;pypa&#x2F;manylinux" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;pypa&#x2F;manylinux</a>), but conda is super pleasant today
评论 #17608884 未加载
评论 #17610891 未加载
评论 #17609811 未加载
评论 #17609287 未加载
tamatsyk将近 7 年前
Interesting reading, I share some of the points in the post, however, one more dependency manager?<p>Mostly I&#x27;ve used plain `python -m venv venv` and it always worked well. A downside - you need to add a few bash scripts to automate typical workflow for your teammates.<p>Pipenv sounds great but there are some pitfalls as well. I&#x27;ve been going through this post recently and got a bit upset about Pipenv: <a href="https:&#x2F;&#x2F;chriswarrick.com&#x2F;blog&#x2F;2018&#x2F;07&#x2F;17&#x2F;pipenv-promises-a-lot-delivers-very-little&#x2F;" rel="nofollow">https:&#x2F;&#x2F;chriswarrick.com&#x2F;blog&#x2F;2018&#x2F;07&#x2F;17&#x2F;pipenv-promises-a-l...</a><p>Another point is that it does not work well with PyCharm and does not allow to put all dependencies into the project folder as I used to do with venv. (just like to keep everything in one folder to clean up it easily)<p>Are there any better practices to make life easier?
评论 #17610200 未加载
评论 #17612416 未加载
jessaustin将近 7 年前
I bitch a lot about npm, but then I remember that time when python&#x27;s package distribution drove me to learn a new language. I can&#x27;t help but notice that TFA and all the comments here are only talking about one end of this: managing your dev environment. Is there a similar work explaining how to distribute python packages in a straightforward manner? Is that article compatible with this one?
superbatfish将近 7 年前
The author&#x27;s justifications for using this home-grown tool over miniconda are weak at best, if not plain incorrect.<p>Conda really is the tool he wants; he just seems not to understand that.
评论 #17612397 未加载
评论 #17613048 未加载
rbanffy将近 7 年前
Version pinning is technical debt and a fool&#x27;s errand. New versions will always come out and your new development is confined to what once worked. You need to keep testing with current versions to see what will break when you upgrade and fix it as soon as possible so as to minimize the odds of a big breaking change.<p>It may keep your environment stable for some time, but that stability is an illusion because the whole world moves on. You may be able to still keep your Python 2.2 applications running on Centos 3 forever, but you shouldn&#x27;t want to do it.
评论 #17616000 未加载
xycco将近 7 年前
virtualenv + pip-tools<p><a href="https:&#x2F;&#x2F;github.com&#x2F;jazzband&#x2F;pip-tools" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;jazzband&#x2F;pip-tools</a>
评论 #17608787 未加载
alanfranzoni将近 7 年前
One things that comes to my mind is: when I was starting using Python, I was eager to mock Java people and their absurd approach (write everything in Java, specify a full classpath for all dependency, etc). I pointed out as it was easy and quick to program in Python rather than in Java.<p>I did not appreciate what the pros of a linear and well-defined (by the language) approach to the dependencies, and a clear API between the system libraries (java, javax) vs the user libraries, actually gives A LOT of value. Even though it&#x27;s more cumbersome to use.
syoc将近 7 年前
Why would you do this? Redirect chain:<p><pre><code> https:&#x2F;&#x2F;tech.instacart.com&#x2F;freezing-pythons-dependency-hell-in-2018-f1076d625241 https:&#x2F;&#x2F;medium.com&#x2F;m&#x2F;global-identity?redirectUrl=https%3A%2F%2Ftech.instacart.com%2Ffreezing-pythons-dependency-hell-in-2018-f1076d625241 https:&#x2F;&#x2F;tech.instacart.com&#x2F;freezing-pythons-dependency-hell-in-2018-f1076d625241?gi=85c0588ca374</code></pre>
评论 #17609286 未加载
Waterluvian将近 7 年前
I ran into a migraine last week: cleaning up requirements.txt<p>How do you determine which requirements are no longer needed when you remove one from your code? In node, your package.json lists only packages YOU installed. So removing them cleans up their dependencies. But in Python, adding one package with pip install might add a dozen entries, none indicating they&#x27;re dependencies of other packages.
评论 #17608623 未加载
评论 #17608568 未加载
评论 #17608539 未加载
评论 #17609317 未加载
textmode将近 7 年前
Naive question: Why does this url 302 redirect to medium.com and then medium.com forwards back to the same original url?<p>Is there some commercial advantage?<p>Why not just post the medium url<p><a href="https:&#x2F;&#x2F;medium.com&#x2F;p&#x2F;f1076d625241" rel="nofollow">https:&#x2F;&#x2F;medium.com&#x2F;p&#x2F;f1076d625241</a><p>This 302 redirects to tech.instacart.com
dorfsmay将近 7 年前
Anybody played with the brand new XAR from Facebook?<p><a href="https:&#x2F;&#x2F;code.fb.com&#x2F;data-infrastructure&#x2F;xars-a-more-efficient-open-source-system-for-self-contained-executables&#x2F;" rel="nofollow">https:&#x2F;&#x2F;code.fb.com&#x2F;data-infrastructure&#x2F;xars-a-more-efficien...</a>
评论 #17661225 未加载
评论 #17610404 未加载
BerislavLopac将近 7 年前
Since we&#x27;re sharing XKCD cartoons, here&#x27;s one that comes to mind: <a href="https:&#x2F;&#x2F;xkcd.com&#x2F;927&#x2F;" rel="nofollow">https:&#x2F;&#x2F;xkcd.com&#x2F;927&#x2F;</a><p>So not to disappoint, here&#x27;s another contestant: Poetry [0]<p>That said, in my experience it works best if don&#x27;t force any particular workflow on your developers, but maintain a solid and repeatable process for testing and deployment. People have different mental models of their development environments -- I personally use virtualfish (or virtualenvwrapper if I&#x27;m on Bash), while a colleague works with `python -m venv`; and we have played with pipenv, pyenv, anaconda and poetry in various cases.<p>As long as your requirements are clearly defined -- requirements.txt works perfectly well for applications, and setup.py for libraries [1] -- any method should be good enough to build a development environment. On the other hand, your integration, testing and deployment process should be universal, and fully automated if possible, and of course independent of any developer&#x27;s environment.<p>[0] <a href="https:&#x2F;&#x2F;github.com&#x2F;sdispater&#x2F;poetry" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;sdispater&#x2F;poetry</a><p>[1] <a href="https:&#x2F;&#x2F;caremad.io&#x2F;posts&#x2F;2013&#x2F;07&#x2F;setup-vs-requirement&#x2F;" rel="nofollow">https:&#x2F;&#x2F;caremad.io&#x2F;posts&#x2F;2013&#x2F;07&#x2F;setup-vs-requirement&#x2F;</a>
Animats将近 7 年前
<i>Use a fresh virtualenv for each project</i><p>As a form of version pinning, this locks in old versions and creates technical debt. A few years downstream, you&#x27;re locked into library modules no longer supported and years behind in bug fixes.
评论 #17614130 未加载
chocks将近 7 年前
We’ve recently went through this process at our company &amp; chose to use pipenv as the dependency management tool. As mentioned in the article, pipenv is under active development but takes care of many things that we had custom scripts before such as requirements hashs, in-built graph of dependencies, automatic retries of failed dependencies, automatic re-ordering of dependency installations etc. it also has a few quirks - we had to pick a version that had most commands working &amp; also pipenv install is painfully slow &amp; didn’t seem to have a caching strategy for already built virtualenvs.
Wheaties466将近 7 年前
Doesn&#x27;t using requirments.txt not account for (I forget the official name) Double Dependencies, you dependencies in requirements.txt might have a dependency whose version number may change over time.<p>This seems like something pip freeze could handle but doesn&#x27;t.
ausjke将近 7 年前
I started using pipenv and it seems everything just works fine, except that I can&#x27;t really install wxPython with pipenv, but I can live with that.
jrochkind1将近 7 年前
ruby practices based around bundler aren&#x27;t perfect, but they did solve _this_ level of problem ~7 years ago.<p>It remains a mystery to me why python seems to have won the popularity battle against ruby. They are very similar languages, but in all ways they differ ruby seems superior to me.
评论 #17611947 未加载
AstralStorm将近 7 年前
Yes, let&#x27;s add another incompatible tool to the list. &#x2F;s<p>Here&#x27;s to Python 4 actually fixing this mess.
Alir3z4将近 7 年前
Dependency hell in Python ? The only annoying part would be missing some library to build certain packages, like lxml, etc.<p>That&#x27;s all.<p>We Python developers are fortunate to have amazing tools such as pip, virtualenv, etc.
alanfranzoni将近 7 年前
So... current tools miss some functionality. Let&#x27;s invent a new one. Reminds me of another xkcd: <a href="https:&#x2F;&#x2F;xkcd.com&#x2F;927&#x2F;" rel="nofollow">https:&#x2F;&#x2F;xkcd.com&#x2F;927&#x2F;</a>
avip将近 7 年前
Less blogs, more Dockerfiles. That&#x27;s the solution.
kanox将近 7 年前
This needs to be posted again: <a href="https:&#x2F;&#x2F;xkcd.com&#x2F;927&#x2F;" rel="nofollow">https:&#x2F;&#x2F;xkcd.com&#x2F;927&#x2F;</a>