TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

CocoaPods downloads max out five GitHub server CPUs

826 点作者 jergason大约 9 年前

43 条评论

onli大约 9 年前
Note how perfect that response from mhagger is. A clear, honest sounding assurance of what Github wants to deliver. A perfectly comprehensible description of what is the problem, and where it is coming from. And then suggestion how to fix it the project actually can work on, plus mentioning changes to git itself that Github is trying to make that would help. It not only shows great work going on behind the scenes (and if that is untrue, it at least gives me that impression, which is what counts), but also explains it in a great way.
评论 #11247083 未加载
评论 #11245891 未加载
评论 #11246194 未加载
评论 #11250022 未加载
评论 #11247709 未加载
评论 #11246451 未加载
Gratsby大约 9 年前
From CocoaPods.org:<p>&gt; CocoaPods is a dependency manager for Swift and Objective-C Cocoa projects. It has over ten thousand libraries and can help you scale your projects elegantly.<p>The developer response:<p>&gt; [As CocoaPods developers] Scaling and operating this repo is actually quite simple for us as CocoaPods developers whom do not want to take on the burden of having to maintain a cloud service around the clock (users in all time zones) or, frankly, at all. Trying to have a few devs do this, possibly in their spare-time, is a sure way to burn them out. And then there’s also the funding aspect to such a service.<p>--<p>So they want to be the go-to scaling solution, but they don&#x27;t want to have to spend any time thinking about how to scale anything. It should just happen. Other people have free scalable services, they should just hand over their resources.<p>Thank goodness Github thought about these kinds of cases from the beginning and instituted automatic rate limiting. Having an entire end user base use git to sync up a 16K+ directory tree is not a good idea in the first place. The developers should have long since been thinking about a more efficient solution.
评论 #11246215 未加载
评论 #11246823 未加载
评论 #11246043 未加载
评论 #11246036 未加载
评论 #11246554 未加载
评论 #11247497 未加载
评论 #11246029 未加载
pjc50大约 9 年前
This reply: <a href="https:&#x2F;&#x2F;github.com&#x2F;CocoaPods&#x2F;CocoaPods&#x2F;issues&#x2F;4989#issuecomment-193810378" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;CocoaPods&#x2F;CocoaPods&#x2F;issues&#x2F;4989#issuecomm...</a><p><i>&quot;Not having to develop a system that somehow syncs required data at all means we get to spend more time on the work that matters more to us, in this case. (i.e. funding of dev hours)&quot;</i><p>In other words, using github as a free unlimited CDN lets them be as inefficient as they like. Such as having 16k entries in a directory ( <a href="https:&#x2F;&#x2F;github.com&#x2F;CocoaPods&#x2F;Specs&#x2F;tree&#x2F;master&#x2F;Specs" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;CocoaPods&#x2F;Specs&#x2F;tree&#x2F;master&#x2F;Specs</a> ) which every user downloads.<p>Package management and sync seems to suffer really badly from NIH. Dpkg is over 20 years old and yum is over a decade old. What&#x27;s up with this particular wheel that people keep reinventing it seemingly without improvement?
评论 #11247343 未加载
评论 #11246869 未加载
评论 #11246547 未加载
评论 #11246087 未加载
评论 #11246708 未加载
indygreg2大约 9 年前
I help run Mozilla&#x27;s version control infrastructure and the problems described by the GitHub engineer have been known to me for years. Concerns over scaling Git servers are one of the reasons I am extremely reluctant to see Mozilla support a high volume Git server to support Firefox development.<p>Fortunately for us, Firefox is canonically hosted in Mercurial. So, I implemented support in Mercurial for transparently cloning from server-advertised pre-generated static files. For hg.mozilla.org, we&#x27;re serving &gt;1TB&#x2F;day from a CDN. Our server CPU load has fallen off a cliff, allowing us to scale hg.mozilla.org cheaply. Additionally, consumers around the globe now clone faster and more reliably since they are using a global CDN instead of hitting servers on the USA west coast!<p>If you have Mercurial 3.7 installed, `hg clone <a href="https:&#x2F;&#x2F;hg.mozilla.org&#x2F;mozilla-central`" rel="nofollow">https:&#x2F;&#x2F;hg.mozilla.org&#x2F;mozilla-central`</a> will automatically clone from a CDN and our servers will incur maybe 5s of CPU time to service that clone. Before, they were taking minutes of CPU time to repackage server data in an optimal format for the client (very similar to the repack operation that Git servers perform).<p>More technical details and instructions on deploying this are documented in Mercurial itself: <a href="https:&#x2F;&#x2F;selenic.com&#x2F;repo&#x2F;hg&#x2F;file&#x2F;9974b8236cac&#x2F;hgext&#x2F;clonebundles.py" rel="nofollow">https:&#x2F;&#x2F;selenic.com&#x2F;repo&#x2F;hg&#x2F;file&#x2F;9974b8236cac&#x2F;hgext&#x2F;clonebun...</a>. You can see a list of Mozilla&#x27;s advertised bundles at <a href="https:&#x2F;&#x2F;hg.cdn.mozilla.net&#x2F;" rel="nofollow">https:&#x2F;&#x2F;hg.cdn.mozilla.net&#x2F;</a> and what a manifest looks like on the server at <a href="https:&#x2F;&#x2F;hg.mozilla.org&#x2F;mozilla-central?cmd=clonebundles" rel="nofollow">https:&#x2F;&#x2F;hg.mozilla.org&#x2F;mozilla-central?cmd=clonebundles</a>.<p>A number of months ago I saw talk on the Git mailing list about implementing a similar feature (which would likely save GitHub in this scenario). But I don&#x27;t believe it has manifested into patches. Hopefully GitHub (or any large Git hosting provider) realizes the benefits of this feature and implements it.
评论 #11246704 未加载
评论 #11247360 未加载
jdcarter大约 9 年前
Wow, really impressive response from GitHub. The right amount of technical detail coupled with balanced tone--halfway between &quot;we support you&quot; and &quot;you make us crazy.&quot;<p>One correction to the post title: it&#x27;s not maxing five nodes, but five CPUs.
评论 #11245884 未加载
评论 #11246926 未加载
web007大约 9 年前
I keep coming back to point #4 - who ever thought that 16k objects in a single directory would be a good idea? Ever since FAT that&#x27;s been a bad idea, and while modern FSes will handle it without completely melting down it&#x27;s still going to cause long access operations on anything to do with it.<p>Even Finder or `ls` will have trouble with that, and anything with * is almost certainly going to fail. Is the use-case for this something that refers to each library directly, such that nobody ever lists or searches all 16k entries?
评论 #11247124 未加载
评论 #11247280 未加载
mikeash大约 9 年前
The criticism against CocoaPods here seems awfully harsh.<p>Think about it from their perspective. GitHub advertises a free service, and encourages using it. Partly it&#x27;s free because it&#x27;s a loss leader for their paid offerings, and partly it&#x27;s free because free usage is effectively advertising GitHub. CocoaPods builds builds their project on this free service, and everything is fine for years.<p>Then one day things start failing mysteriously. It looks like GitHub is down, except GitHub isn&#x27;t reporting any problems, and other repositories aren&#x27;t affected.<p>After lots of headscratching, GitHub gets in touch and says: you&#x27;re using a ton of resources, we&#x27;re rate limiting you, you&#x27;re using git wrong, and you shouldn&#x27;t even be using git.<p>That&#x27;s going to be a bit of a shock! Everything seemed fine, then suddenly it turns out you&#x27;ve been a major problem for a while, but nobody bothered to tell you. And now you&#x27;re in hair-on-fire mode because it&#x27;s reached the point where the rate-limiting is making things fail, and nobody told you about any of these problems before they reached a crisis point.<p>It strikes me as extremely unreasonable to expect a group to avoid abusing a free service when nobody tells them that it&#x27;s abuse, and as far as they know they&#x27;re using it in a way that&#x27;s accepted and encouraged. If somebody is doing something you don&#x27;t like and you want them to stop, you have to tell them, or nothing will happen!<p>I&#x27;m not blaming GitHub here either. I&#x27;m sure they didn&#x27;t make this a surprise on purpose, and they have a ton of other stuff going on. This looks like one of those things where nobody&#x27;s really to blame, it&#x27;s just an unfortunate thing that happened.<p>(And just to be clear, I don&#x27;t have much of a dog in this fight on either side. My only real exposure to CocoaPods is having people occasionally bug me to tag my open source repositories to make them easier to incorporate into CocoaPods. I use GitHub for various things like I imagine most of us do, but am not particularly attached to them.)
评论 #11248430 未加载
评论 #11248551 未加载
评论 #11248635 未加载
评论 #11248239 未加载
wpeterson大约 9 年前
It&#x27;s totally reasonable to host your code on github and to build a package manager that loads the content of a package from it&#x27;s github repo.<p>What seems insane is to use a single github repo as the universal directory of packages and their versions driving your package manager.<p>There&#x27;s a reason rubygems has their own servers and web services to support this use case for the central library registry, even if the source for gems are all individually projects hosted on github.
评论 #11246803 未加载
评论 #11250700 未加载
riscy大约 9 年前
&gt; Scaling and operating this repo is actually quite simple for us as CocoaPods developers whom do not want to take on the burden of having to maintain a cloud service around the clock (users in all time zones) or, frankly, at all.<p>The CocoaPods developers seem to be missing the entire point of git: it&#x27;s a _distributed_ revision control system.<p>Setup a post-recieve hook on Github to notify another server, that is setup with a basic installation of git, to pull from Github so as to mirror the master repo. Then, have your client program randomly choose one of these servers to pull from at the start of an operation. Simple load balancer to solve this problem.
评论 #11246600 未加载
spoiler大约 9 年前
I find it amusing how GitHub&#x27;s contact[1] form has (probably a recent addition):<p>&gt; GitHub Support is unable to help with issues specific to CocoaPods&#x2F;CocoaPods.<p>---<p>[1]: <a href="https:&#x2F;&#x2F;github.com&#x2F;contact" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;contact</a>
评论 #11246520 未加载
评论 #11246427 未加载
评论 #11246398 未加载
评论 #11247129 未加载
评论 #11246328 未加载
评论 #11246499 未加载
rmoriz大约 9 年前
CocoaPods (and Homebrew) mainly exist because of a lack of tooling in the typical Apple ecosystem. So I would blame Apple for not supporting the community with money or tooling. Letting GitHub with its limited amount of funding pay the bill isn&#x27;t a nice move. Apple dev relations should throw some money at GitHub so they can provide some dedicated resources or offer to pay the cost of other solutions (like a 3rd party CDN&#x2F;AWS&#x2F;Google Cloud&#x2F;…).
评论 #11247256 未加载
评论 #11247484 未加载
zymhan大约 9 年前
I&#x27;ve always found Github&#x27;s business model interesting. What if a massive open-source organization (e.g. Fedora, Apache) decided to use it for all of their development, integrating it with continuous builds and all the associated pulls. Of course this isn&#x27;t likely to happen for a number of reasons, but there are large open source projects that could put a significant load on their infrastructure if they chose to use Github as their main code versioning system.
评论 #11246001 未加载
评论 #11246660 未加载
评论 #11246753 未加载
评论 #11246041 未加载
评论 #11245969 未加载
iBotPeaches大约 9 年前
This bug report is a great step in the direction for GitHub. As of this comment there are 3 different GitHub staff members responding and providing feedback to the CocoaPods team. From the previous &quot;Dear GitHub&quot; messages and responses, this seems like perfect community involvement.
paradite大约 9 年前
I have been seeing this trend of GitHub getting &quot;abused&quot; for purposes other than hosting source code.<p>- My school uses GitHub to host and track our software engineering project (which still can be argued as OSS).<p>- People using GitHub issue system as a forum.<p>- Friends uploading pdfs to GitHub.<p>- Recently people posted on HN about using GitHub to generate a status page.<p>I think this is a really bad trend and people should stop doing that.
评论 #11247990 未加载
fpgaminer大约 9 年前
GitTorrent: <a href="http:&#x2F;&#x2F;blog.printf.net&#x2F;articles&#x2F;2015&#x2F;05&#x2F;29&#x2F;announcing-gittorrent-a-decentralized-github&#x2F;" rel="nofollow">http:&#x2F;&#x2F;blog.printf.net&#x2F;articles&#x2F;2015&#x2F;05&#x2F;29&#x2F;announcing-gittor...</a><p>Imagine a world where GitTorrent is fully developed, includes support for issue tracking, and has a nice GUI client that makes the experience on-par with browsing github.com.<p>I mention this not as an &quot;Everybody bail out of GitHub and run to GitTorrent!!!&quot; sort of statement, because I believe GitHub&#x27;s response here was excellent and confidence inspiring. But it&#x27;s an unnatural relationship for community supported, open source projects to host themselves on commercial platforms such as GitHub. GitHub primarily hosts them to promote its business. That&#x27;s not necessarily a bad thing, but it results impedance mismatches like demonstrated here.<p>That isn&#x27;t to say that a mature GitTorrent would replace GitHub. Rather, I envision GitHub becoming a supernode in the network, an identity provider, and a paid seed offering, all alongside their existing private repo business.<p>Honestly, once I scrape a few projects off my plate, I&#x27;m inclined to dive into GitTorrent, see where it&#x27;s at in development, and see if I can start contributing code. It just seems like such a cool and useful idea.
评论 #11251048 未加载
pavlov大约 9 年前
I&#x27;ve never really understood CocoaPods. Dragging a framework into Xcode was never much trouble, and the amount of 3rd party libraries in a OS X &#x2F; iOS project ought to be fairly small, so the gains are trivial.<p>The potential downsides seem much more annoying. Do you really want to have your dependencies on an overloaded central server somewhere?
评论 #11246425 未加载
jrochkind1大约 9 年前
What an unusually reasonable discussion. good on everyone.
sdegutis大约 9 年前
I love how this was like <i>the</i> perfect storm of things that could go wrong, and how it seems like mhagger is just amazed more than anything else.
评论 #11246301 未加载
评论 #11246091 未加载
评论 #11245879 未加载
ak217大约 9 年前
I love GitHub&#x27;s response, but I would urge the project more strongly to use modern CDN solutions. CDNs are dirt cheap and incredibly powerful nowadays, for the data sizes that we&#x27;re talking about here.
评论 #11246401 未加载
评论 #11247356 未加载
tjdetwiler大约 9 年前
Rust&#x27;s cargo does something similar, however it looks like they were much more conscious of git-scalability (ex: limiting the directories in a single level, only appending lines to files to make diffs small).<p><a href="https:&#x2F;&#x2F;github.com&#x2F;rust-lang&#x2F;crates.io-index" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;rust-lang&#x2F;crates.io-index</a>
评论 #11249204 未加载
iamleppert大约 9 年前
Amazing to me that people create inefficient systems like this and then complain when they are rate limited.
maaku大约 9 年前
Using Github as your CDN is a dick move. Kudos to GH for not banning the project out-right, but CocoaPods should seriously reconsider what they are doing.
xemdetia大约 9 年前
As a current maintenance developer&#x2F;systems guy I can definitely feel the tempered annoyance from mhagger here. It&#x27;s definitely nice to not remind yourself that it&#x27;s not only your set of recurring issues in front of you that people have to deal with.
noahlt大约 9 年前
Go&#x27;s package manager, `go get`, also downloads from GitHub. I don&#x27;t know the details of how `go get` and CocoaPods work, but I would be interested in learning why one is unscalable and the other seems to work.
评论 #11247314 未加载
评论 #11247245 未加载
SuperKlaus大约 9 年前
In fact, they are maxing out five <i>CPUs</i> - not five nodes, big difference.
fokinsean大约 9 年前
I found the solution humorous. Ironically shallow clones are causing the problems, so fetch the max :)<p>$ git fetch --depth=2147483647
评论 #11247141 未加载
kodablah大约 9 年前
Has any consideration been given to Bintray[1] as an alternative store for this stuff?<p>1 - <a href="https:&#x2F;&#x2F;bintray.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;bintray.com&#x2F;</a>
rcthompson大约 9 年前
Reading the issue, it seems that one of the problems is a single directory with lots and lots of files in it, which is something of a pathological case for Git. Now, this could be &quot;fixed&quot; by splitting the files in that directory into subdirectories, but the one giant directory will still exist in all the past commits. So would this actually fix anything, or just keep it from getting worse?
评论 #11251030 未加载
kmm大约 9 年前
Funny thing is that the repo is only 7 MB gzipped (or 4 with lzma). Not that surprising, since it&#x27;s just metadata of course. They say they have about 1 million fetches&#x2F;clones per week, so that would make about 16 TB per month. I&#x27;m not sure how much bandwidth costs, but wouldn&#x27;t some sympathetic CDN host that for free, since they&#x27;re OSS?
soheil大约 9 年前
I was up until 2am last night trying to publish my Pod [1] and Github kept timing out.<p>I had no idea it was just CocoaPods repo because my other repos were working fine. I accepted defeat, went to bed and everything was working great in the morning.<p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;soheil&#x2F;SwiftCSS" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;soheil&#x2F;SwiftCSS</a>
sly010大约 9 年前
I would be interested to know what are the other top GitHub repositories. Afaik the Nix package manager uses a similar model (using a GitHub repo as a database), however they periodically release snapshots and the default configuration uses those instead of git.
评论 #11246279 未加载
zoul大约 9 年前
Another reason I consider <a href="https:&#x2F;&#x2F;github.com&#x2F;Carthage&#x2F;Carthage" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;Carthage&#x2F;Carthage</a> a better solution of the dependency management problem.
评论 #11245851 未加载
评论 #11247021 未加载
评论 #11248004 未加载
Negative1大约 9 年前
When he says approaches similar to &#x27;other packaging systems&#x27;, which ones is he referring to? I can see why this is a bad approach but am unfamiliar with what would be considered a better practice (outside of just hosting a .tar on CloudFront).
评论 #11246686 未加载
评论 #11246179 未加载
superuser2大约 9 年前
Just last night, all my pod installs were timing out after ~30ish minutes. That explains it.
评论 #11246615 未加载
评论 #11246959 未加载
joeblau大约 9 年前
I just installed Cocoapods last night and tried to clone down the repo. It took about 5 minutes and I thought to myself &quot;Is my 150MB&#x2F;s connection slow?&quot; This definitely clears up what was going on.
debacle大约 9 年前
Why aren&#x27;t the packages distributed? Composer is incredibly distributed and likely doesn&#x27;t cause nearly the same headaches for GitHub.<p>Seems like a poor design decision on the CocoaPods side.
评论 #11248405 未加载
voltagex_大约 9 年前
It&#x27;s difficult to run an open source project on a budget of $0. You&#x27;re always relying on the goodwill of others.
LoneWolf大约 9 年前
While I do not have much knowledge on the subject, why not using rsync?
nimish大约 9 年前
hopefully we can now move to using real artifact repositories.
rdancer大约 9 年前
tl;dr: &quot;Using GitHub as your [free-of-charge] CDN is not ideal, for anybody involved.&quot;
speps大约 9 年前
Why is everyone talking about CocoaPods where the title is CoacoaPods anyway? :)
评论 #11248252 未加载
whitehat2k9大约 9 年前
Only the Apple development community would think it&#x27;s OK to have 16,000 subdirectories in one place and abuse GitHub as a free CDN instead of putting some actual effort in and develop their own repository infrastructure - you know, like almost every other package manager in existence.
评论 #11248282 未加载
Const-me大约 9 年前
I don’t think GitHub acts wisely here.<p>Short term sure, they’re doing the right thing, implementing a nice way to manage the free rider problem without hurting them too much.<p>But long term it’s different.<p>Financially, one average programmer = $80k&#x2F;year, one average cloud server = $4k&#x2F;year. And, GitHub has hundreds of millions of venture capital. More than enough to provision a few more servers, even if they will be installing new servers just for those pods.<p>The way they act now will lead to someone will develop a decentralized git+torrent hybrid. When that happens, sure, those pods will no longer consume precious GitHub’s resources. However, for the rest of the github users, there will be no reason to stay on GitHub either.
评论 #11251025 未加载