TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Show HN: Morgan – PyPI Mirror for Restricted/Offline Environments

77 点作者 idop超过 2 年前
Mirroring PyPI packages for environments&#x2F;networks that do not have access to the Internet is hard. It&#x27;s actually hard even in environments that do have access to the Internet. Most solutions out there either:<p>1. Depend on pip to download and cache package distributions. This means those downloads will probably only work in a similar environment (same Python interpreter, same libc), because of the nature of binary package distributions and the fact that packages have optional dependencies for different environments.<p>2. Depend on other PyPI packages, meaning installing the mirror in a restricted environment in itself is too difficult.<p>3. Cannot resolve dependencies of dependencies, meaning mirroring PyPI partially is extremely difficult, and PyPI is huge.<p>Morgan works differently. It creates a mirror based on a configuration file that defines target environments (using Python&#x27;s standard Environment Markers specification from PEP 345) and a list of package requirement strings (e.g. &quot;requests&gt;=2.24.0&quot;). It downloads all files relevant to the target environments from PyPI (both source and binary distributions), and recursively resolves and downloads their dependencies, again based on the target environments. It then extracts a single-file server to the mirror directory that works with Python 3.7+, has no outside dependencies, and implements the standard Simple API. This directory can be copied to the restricted network, through whatever security policies are in place, and deployed easily with a simple `python server.py` command.<p>I should note that Morgan can find dependencies from various metadata sources inside package distributions, including standard METADATA&#x2F;PKG-INFO&#x2F;pyproject.toml files, and non-standard files such as setuptools&#x27; requires.txt.<p>There&#x27;s more information in the Git repository. If this is interesting to you, I&#x27;ll be happy to receive your feedback.<p>Thanks!

8 条评论

mofeing超过 2 年前
hey,<p>We were running with the same problem (supercomputer with clusters of different architecture and no outgoing connections permitted) and so we created &quot;pypickup&quot; [1,2]. nice to see that we came with similar solutions! I have some questions:<p>1. is the directory of packages you create compatible with the PEP 503? (so I can use `--index-url file:&#x2F;&#x2F;PATH_TO_LOCAL_CACHE` flat with pip and it should work)<p>2. is there some filtering mechanism? e.g. we are not interested in non-release versions (&quot;dev&quot; versions, &quot;rc&quot; versions, &quot;post&quot; versions, ...)<p>3. I guess that the way morgan resolves dependencies is by manually parsing files like &quot;pyproject.toml&quot; or &quot;requirements.txt&quot; and it does not ask the build-system for the dependencies. if so...<p><pre><code> - does &quot;morgan&quot; detect build-dependencies? - which build-systems are compatible? - is &quot;morgan&quot; capable of detecting more complex dependency specifications? e.g. &quot;oldest-supported-numpy&quot; which is used by &quot;spicy&quot; has dependency strings like the following: numpy==1.19.2; python_version==&#x27;3.8&#x27; and platform_machine==&#x27;aarch64&#x27; and platform_python_implementation != &#x27;PyPy&#x27; </code></pre> kudos for the good work<p>[1] <a href="https:&#x2F;&#x2F;pypi.org&#x2F;project&#x2F;pypickup&#x2F;" rel="nofollow">https:&#x2F;&#x2F;pypi.org&#x2F;project&#x2F;pypickup&#x2F;</a> [2] <a href="https:&#x2F;&#x2F;github.com&#x2F;UB-Quantic&#x2F;pypickup" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;UB-Quantic&#x2F;pypickup</a>
评论 #32954581 未加载
Galanwe超过 2 年前
Maybe I&#x27;m confused about what this offers, but I have been running private pypi repositories for a decade now, and it never required more than running an HTTP server with directory listing.<p>As for doing partial mirroring of pypi with only what you are using, is that really a good idea anyway? it will break whenever you add or change any dependency.
评论 #32954612 未加载
评论 #32956103 未加载
评论 #32954296 未加载
hackish超过 2 年前
Thanks for posting this. I&#x27;m going to give setting up Morgan a shot when I&#x27;ve got some free cycles.<p>I&#x27;d hesitantly accepted the risk of serving a devpi server over vsock and into my (personal) restricted VLAN. I did so because using a shared folder meant I&#x27;d need have cached the module and any dependencies from my internet-connected VLAN first.<p>Combined with debmirror[0], vscodeoffline[1], and some nightly snatcher shell scripts, I think I have most of my needs covered.<p>[0] <a href="https:&#x2F;&#x2F;help.ubuntu.com&#x2F;community&#x2F;Debmirror" rel="nofollow">https:&#x2F;&#x2F;help.ubuntu.com&#x2F;community&#x2F;Debmirror</a><p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;LOLINTERNETZ&#x2F;vscodeoffline" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;LOLINTERNETZ&#x2F;vscodeoffline</a>
uranusjr超过 2 年前
This is pretty cool. I created simpleindex[1] a while ago to solve a different problem, but since the solution is essentially also running a custom index server, it has several overlapping functionalities to Morgan’s server script. I wonder if there’s a common pattern that can be extracted out…<p>BTW I also maintain resolvelib (mentioned in another comment), feel free to shoot any questions in the issue tracker or the PyPA Discord[2], or any other means. The documentation is a bit sparse and there are not many resources on dependency resolution in general, and there’s a few of us that help each other out on things.<p>[1]: <a href="https:&#x2F;&#x2F;github.com&#x2F;uranusjr&#x2F;simpleindex" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;uranusjr&#x2F;simpleindex</a> [2]: <a href="https:&#x2F;&#x2F;discord.com&#x2F;invite&#x2F;pypa" rel="nofollow">https:&#x2F;&#x2F;discord.com&#x2F;invite&#x2F;pypa</a>
jvolkman超过 2 年前
This looks similar to some Bazel rules I&#x27;m working on. I&#x27;m also using the approach of defining target environments up front [1], but the main difference is that I&#x27;m currently offloading the actual resolution process to Poetry or PDM, which both generate cross-platform lock files.<p>But Poetry and PDM don&#x27;t add build dependencies to lock files - which I need - so I&#x27;m thinking of building a custom resolver.<p>Did you consider using resolvelib [2], which is what underlies both pip and PDM?<p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;jvolkman&#x2F;rules_pycross&#x2F;blob&#x2F;main&#x2F;examples&#x2F;poetry&#x2F;BUILD.bazel#L58-L71" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;jvolkman&#x2F;rules_pycross&#x2F;blob&#x2F;main&#x2F;examples...</a><p>[2] <a href="https:&#x2F;&#x2F;github.com&#x2F;sarugaku&#x2F;resolvelib" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;sarugaku&#x2F;resolvelib</a>
评论 #32957714 未加载
评论 #32956195 未加载
skbly7超过 2 年前
Thanks for creating it and looking forward to try it out.<p>I have been looking for similar solution and the whitelist used to fail with other tools as they weren&#x27;t resolving the dependencies.
danrocks超过 2 年前
When I worked at Microsoft, one team created a big solution for an e-commerce customer using Kubernetes, Helm charts, etc. Beautiful.<p>Then I had to take it to run in mainland China.<p>Nope.
indrora超过 2 年前
Oh neat. Not only do I share a name with a project, it&#x27;s a project I was seriously thinking of starting.
评论 #32955452 未加载