TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Experiment: The costs of slow build times

167 pointsby pimterryover 2 years ago

31 comments

tjoffover 2 years ago
This is also a big reason to self-host.<p>Without this beta github offer: 2-core CPU (x86_64), 7 GB of RAM, 14 GB of SSD space<p>Your last development machine that was thrown away is so much faster than this (and we do have great tools for administer stuff like this nowadays). Hell, the computer I bought as a student in 2008 is comparable(!) (though it didn&#x27;t have an SSD). And it will have so much better network connectivity with everything else on your network.<p>Whenever you hear cloud, realize that the dedicated time you will get (unless you specifically pay for it (in which case it will be more expensive than self-host)) is in the same order the phone you had two generations ago. That is why gmail search sucks. Because they can&#x27;t afford to really search your messages and can not even do exact matching properly.<p>So yes, apparently github is fixing this now and if paying for this problem makes sense for you do it. But this is a problem that was partly invented by the cloud in the first place.
评论 #33921055 未加载
评论 #33919924 未加载
评论 #33920071 未加载
评论 #33920721 未加载
评论 #33923698 未加载
评论 #33919734 未加载
e1gover 2 years ago
I recently moved our GutHub CI to a self-hosted runner and it reduced CI times for all jobs by ~5x.<p>One frustrating part is that a single GitHub runner can handle one job at a time, and our repos have 4-8 highly parallelized jobs. So we need multiple runners. To do this in an economic way, I made a Docker image and run 10 instances of it a single dedicated host from Hetzner. For ~$50&#x2F;mo, we have unlimited CI and job run as fast as theoretically possible.<p>The CI box has no inbound ports open to the internet, and the OS is set to auto-update, so the maintenance burden is low.
评论 #33920444 未加载
评论 #33920089 未加载
评论 #33922314 未加载
solaticover 2 years ago
Please, step me through this. In a start-up:<p><pre><code> a) Developer builds locally b) Developer tests locally c) Developer pushes to a repository d) CI starts e) CI ends f) Wait for human code review and approval g) Merge and deploy h) Observe that nothing broke &#x2F; no need to revert. </code></pre> Is it even possible for (e) - (d) to be short enough, let alone (f) - (d), to keep the developer&#x27;s attention instead of context switching? Most devs I know just context switch immediately after (c). If you care about developer productivity, you&#x27;re much more likely to get results from focusing on (a) and (b), by hooking development laptops into caching and restricting system&#x2F;test scope, than you are by reducing CI times, unless your CI takes some ungodly amount of time to run (several hours).<p>For point of comparison: article examines how using monster machines can reduce the build time of Fedora to 27 minutes (not really a comparable example to most companies, but OK). My devs complain about an (admittedly unoptimized) CI time of 20 minutes (on a much simpler project than Fedora) that introduces context switching. Is the article really trying to get me to believe that Fedora developers wouldn&#x27;t context switch on a 27 minute build, twiddling their thumbs for 27 minutes, but that a 35 or 55 minute build would? Something about getting under the 30 minute bar gets developers to keep their focus? I call bullshit.
评论 #33921525 未加载
评论 #33921041 未加载
评论 #33921275 未加载
评论 #33921736 未加载
评论 #33922613 未加载
评论 #33921442 未加载
评论 #33921252 未加载
yositoover 2 years ago
&gt; The Bottom Line<p>&gt; It’s cheaper—and less frustrating for your developers—to pay more for better hardware to keep your team on track.<p>I agree with this point, 100%. And not just in the context of build hardware. It&#x27;s often cheaper to pay for specialized services rather than paying in-house developers to do those things. I&#x27;ve lost count of the number of times in my career that I&#x27;ve spent days of effort (at $200&#x2F;hr) so that the company I was working for could avoid paying for some $15&#x2F;mo SaaS.
评论 #33920068 未加载
评论 #33923394 未加载
评论 #33919304 未加载
评论 #33920348 未加载
评论 #33919812 未加载
alkonautover 2 years ago
Build is one of those things that I can&#x27;t see the economy in buying cloud VMs for if you have lots of load.<p>Buy a rack and a dozen cheap-ish rack chassis with some middle range CPUs with lots of cores but not necessarily the highest specs otherwise (Last gen we used 9700K and for a recent refresh it&#x27;s AMD 7700X). You can get quite far at $1k per node and $12k for a rack. Then you can run 3-4 build agents per node so you have a 48 agent cluster in your rack. Electricity and management will be on top of the purchase price of course, but it&#x27;s still a bargain.<p>Even if you do use cloud compute for builds, it&#x27;s worth having some self-hosted nodes for the base need and just using the cloud for peak use scaling.
评论 #33920292 未加载
fefe23over 2 years ago
This is not an experiment. This is an ad by Github who want you to buy their larger build hosts.<p>Notably absent from the comparison is the cost for buying a build host yourself.
评论 #33920403 未加载
nottorpover 2 years ago
Isn&#x27;t the real problem that no one does partial local builds any more? And maybe the lack of a local testing environment.<p>Build servers are fine for the main branch, but why does a dev have to build the whole thing to quickly check the last 50 lines that they&#x27;ve written?
评论 #33921143 未加载
评论 #33920341 未加载
评论 #33923343 未加载
评论 #33921840 未加载
评论 #33921554 未加载
pronover 2 years ago
Focusing on the cost side of software development is not very convincing for managers who need to make purchasing decisions. I believe <i>most</i> software is a high-margin product because manufacturing costs are close to zero. Even a relatively high difference in cost is not as much as a small difference in value. The one cost-side metric that has a big effect on value is time-to-market.<p>What could be more convincing is the effect on product <i>value</i> that build times could have, and they certainly could (and I, for one, believe that they do, especially if they can be made to be less than 15 seconds or so). For example, it&#x27;s certainly conceivable that very short build times make it easier to write more tests, which could result in more correct software with better features and&#x2F;or a shorter time-to-market.<p>Some products targeted at developers also make the mistake of focusing their marketing message on the cost, perhaps because that&#x27;s the part of the business that developers personally feel, but it&#x27;s not the message that would convince their employers.
评论 #33920473 未加载
评论 #33920153 未加载
carreauover 2 years ago
I my experience if you get more builders&#x2F;faster hardware someone higher up will end up asking for more items in the build-matrix, and the CI time will balloon up again.<p>It&#x27;s rarely only a question of just allocating money to more&#x2F;better hardware, it&#x27;s also a question of policy and willingness of your organisation to keep CI time short&#x2F;feedback fast.
评论 #33919782 未加载
评论 #33919449 未加载
评论 #33926899 未加载
评论 #33919440 未加载
thinkafterbefover 2 years ago
Disclaimer: Co-founder of <a href="https:&#x2F;&#x2F;buildjet.com&#x2F;for-github-actions" rel="nofollow">https:&#x2F;&#x2F;buildjet.com&#x2F;for-github-actions</a><p>While more cores can certainly help with certain types of projects, such as those that can be easily parallelized, this is not always the case. For example, web app projects won&#x27;t benefit as much from additional cores.<p>Another important factor to consider is the single-core performance of each vCPU. Many server-class CPUs, such as those used by GitHub, are built with a very high-core count but with a very low single-core speed. In contrast, BuildJet uses consumer CPUs, such as the 5950x, which offer slightly less core count but an excellent single-core speed.<p>It&#x27;s quite astonishing how slow &quot;the cloud&quot;&#x2F;server-class CPUs can be, we compared my old MacBook Pro 2015 vs. a 2vCPU GitHub actions runner and the MBP 2015 won most of the time.<p>BuildJet&#x27;s bet is that single-core performance is critical for a fast CI, and it appears that the self-hosting comments here on HN also agree.<p>(We&#x27;re working our own CI, DM me if you&#x27;re interested in the fastest CI on the market)
评论 #33920481 未加载
评论 #33920613 未加载
评论 #33928338 未加载
tikkabhunaover 2 years ago
We use GitLab CI and solve this with tags. We have runners allocated with set resources and tagged with a t-shirt size. Developers can then tag their job based on this size. Single threaded jobs (curl-uploading a file) can be tagged &quot;small&quot; to get 1 CPU core. Building projects can be medium or large based on how big they are and developers can choose whether the large runner is needed. Because its simple to switch tags, developers can create a feature branch to swap the tag and observe how long it takes.<p>All our runners are self hosted. There are fewer large runners, so developers are inclined to go small. Hypothetically, you can run all your jobs in large but then you may have to wait for availability but in practice we haven&#x27;t seen any real contention.<p>We run over 15,000 jobs a week on 6 physical servers with this pattern.
tjpnzover 2 years ago
Are there really people who still need convincing of this? At the same time you probably shouldn&#x27;t be throwing more hardware at the problem without trying to understand it first. I&#x27;ve witnessed numerous times throughout my career what can be saved by tuning a few parameters.
评论 #33919507 未加载
评论 #33920236 未加载
Hardwired8976over 2 years ago
I worked from home on Wednesday and I managed to watch one episode of Better Call Saul while waiting for the compilation of my code lol.
评论 #33919494 未加载
sluongngover 2 years ago
This is also a reason why many large organizations recently started to adopt a matured build tool solution such as: Bazel &#x2F; Pants &#x2F; Buck. They did the math.
评论 #33920025 未加载
ilytover 2 years ago
Back in ye olde we just linked a bunch of our machines in distcc cluster. Not really workable for remote work or really anything else but C&#x2F;C++ but was pretty much all benefit for that.<p>I&#x27;d kinda love to have transparent &quot;workplace&quot; that builds local by default but just have &quot;build in cloud&quot; option for those few cases where I do need to do a ton of compute at once (say getting to bisect something) but moving completely to the cloud seems just worse experience overall compared to just relatively beefy machine
avodonosovover 2 years ago
If you want to speedup build of a large Maven project, check <a href="https:&#x2F;&#x2F;github.com&#x2F;avodonosov&#x2F;hashver-maven-plugin" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;avodonosov&#x2F;hashver-maven-plugin</a>
no_wizardover 2 years ago
If you&#x27;re in the frontend space and are migrating to or already work in a Monorepo, I can&#x27;t speak highly enough of NX[0] and their cloud builds. Very very fast. The caching algorithm they use is pretty nice and a little smarter than Turbo, and the distributed caching helps both in CI and developer machines.<p>My only complaint is I haven&#x27;t been able to quite figure out how to make the developer machine cache independent from the CI&#x2F;CD one, which has caused a couple of small issues we were able to work around.<p>One thing I have to say though, is I don&#x27;t use NX plugins except for `@nrwl&#x2F;jest`. I just use their Task runner capabilities. I found the plugins were a hassle if you weren&#x27;t adopting the integrated[1] approach. For whatever reason, Jest worked well in the package based monorepo (for whatever reason, our IDE of choice, JetBrains, doesn&#x27;t support TSConfig Paths properly in TSX files so it killed DX. So we had to go package based)<p>That said, it was easy enough to get up and moving with the task runner and we&#x27;ve seen major benefits. Did require us to think a little bit differently in how to run tasks in a parallel friendly way though.<p>[0]: <a href="https:&#x2F;&#x2F;nx.dev&#x2F;" rel="nofollow">https:&#x2F;&#x2F;nx.dev&#x2F;</a><p>[1]: <a href="https:&#x2F;&#x2F;nx.dev&#x2F;concepts&#x2F;integrated-vs-package-based" rel="nofollow">https:&#x2F;&#x2F;nx.dev&#x2F;concepts&#x2F;integrated-vs-package-based</a>
simonhampover 2 years ago
I have never had to run a build that takes more than a few minutes, so I&#x27;m unfamiliar with these sorts of long-running builds.<p>I definitely agree on investing to make builds faster, but when they&#x27;re a few minutes and the build will go down to slightly fewer minutes, those &#x27;marginal&#x27; gains don&#x27;t seem worth the increased build cost.<p>Just throwing more compute at the problem isn&#x27;t always the answer either. The build tools I&#x27;ve been using over the past few years (Webpack and now Vite) have gotten so much faster on their own, which shows that there&#x27;s loads of slack in our code and lots of room for improvement there before we need to throw more compute at it.<p>Also, I&#x27;m certainly not just twiddling my thumbs while it&#x27;s running...<p>Am I the odd one out? Feels like a bit of a straw man argument from MS&#x2F;GH trying to justify a blanket switch to more powerful runners &#x27;because the data says so&#x27;<p>Do the math _for your business_ and _your use-case_
评论 #33920757 未加载
Dowwieover 2 years ago
This makes languages such as Rust a tough sell for startups that can&#x27;t yet afford $3000 M1 laptops for each Rust developer. Rust has incremental compilation and it helps to make incremental builds faster. You also have to wait for compilation every time you run tests. Further, you wait for CI test suites to finish. Optimizing builds in CI, taking advantage of caching at the right steps, isn&#x27;t widely understood nor does optimization completely address the problem with compile times. Throwing hardware at the problem makes sense, if you can afford to, but you still need to optimize the entire chain.<p>Is a remote dev environment an economically viable alternative? I suspect not but haven&#x27;t run the numbers. It&#x27;s not clear to me what tier of virtual server would beat an M1 macbook pro with 32gb ram and sufficient number of cores.
malinensover 2 years ago
One of the main metric for build time for me is to make it fast enough so you do not want switch to other task at all
评论 #33919965 未加载
评论 #33920047 未加载
marginalia_nuover 2 years ago
Excepting C++ development, is slow build times really a hardware problem?<p>CI software in general is very slow. But even building locally, there are many projects that have just ridiculous build times because it&#x27;s just not a priority at all. I&#x27;ve seen small-medium sized Java projects with a 25 minute test suite.<p>This has been in part due to extremely inefficient tests that keep assembling the entire system over and over each test, often with extreme redundancy in coverage, ensuring you need to keep running the entire suite whenever a test breaks. Nobody is going to run tests for half an hour unless you hold a gun to their head. So you make your change by 10 AM, usually something trivial, and then you spend the afternoon fixing the 30 tests that broke, running the suite over and over to find more broken tests to fix. It&#x27;s almost comically inefficient.
评论 #33919751 未加载
评论 #33920088 未加载
ohgodplsnoover 2 years ago
Most people&#x27;s CI issues can be solved in three easy steps.<p>1&#x2F; Get an actual, real machine. We run all of our CI for 10 devs out of a single 13900K &#x2F; 64GB RAM machine, from a Jenkins setup that I&#x27;ve done over a weekend that has a single runner. 5-10 minute runs for PR validations (500kloc app, shitloads of tests, etc), 3 minutes for app builds. Buy it and host it yourself, or rent from Hetzner. You don&#x27;t need a sysadmin for that. Machine is fucked ? Just restart it lol. Dependencies out of date ? Just upgrade your dockerfile, or your ansible setup and reinstall it all. Used to use Gitlab runners, Bitrise, Github Actions, etc. They&#x27;re all too slow, because all these services make money on allocating as little as possible to you to stack as many clients as possible on the same server. Pretty UIs are not worth 45 minute builds.<p>2&#x2F; Cache. Cache. Cache. Cache. Cache. It isn&#x27;t hard if you&#x27;re not looking for perfect. Nobody cares if sometimes, randomly you have one small module that rebuilds even though it shouldn&#x27;t have: the beefy machine is here to make sure it takes 10 seconds anyways. All that matters is that you never have a cache that says even though it shouldn&#x27;t. And to solve this: don&#x27;t make the cache yourself. Your build tool either has it (Gradle, Maven, Webpack, etc), or there&#x27;s a widely used tool that works (ccache, etc.). How do you store your cache ? Not on the build machine. Because that fucking sucks. Make it a remote cache (If using gradle, you can pay for enterprise if you&#x27;re feeling optimistic, host a gradle remote cache yourself, or use <a href="https:&#x2F;&#x2F;github.com&#x2F;androidx&#x2F;gcp-gradle-build-cache" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;androidx&#x2F;gcp-gradle-build-cache</a> and forget about it. The running costs of it have been well under a hundred dollars over a year).<p>3&#x2F; Less important, but know your build tool. I&#x27;ve spent a lot of time rewriting Gradle projects to avoid cross configuration, allow configuration caching, allow parallel configuration, etc. The same applies to everything else. Your build tool should be part of your knowledge.
ttyyzzover 2 years ago
As a Developer, build times sometimes are a blessing :D But thats beside the point. In my JavaScript frontend Framework &#x2F; Nodejs backend world, build times are often times horrific due to the Single-threaded nature of JS. Hardware is often times not to blame.
评论 #33919938 未加载
评论 #33919936 未加载
评论 #33919937 未加载
jjallenover 2 years ago
This long waiting time for builds and testing to complete is a big part of the reason Go was created. Couldn&#x27;t find a direct link. But yeah, using Go is great, build times are crazy low.
comprevover 2 years ago
We use Github Actions on self-hosted runners with the exception of &quot;human interaction&quot; steps which usually run on the normal public runners.<p>So far no complaints to be honest.
benibelaover 2 years ago
If that was written in Pascal, it could be compiled in 25 minutes on a single core rather than using 300 minutes on a dual core
kunleyover 2 years ago
Hmm, in addition to that, what about un-bloating the typical CICD to speed up the build?
评论 #33919763 未加载
lowbloodsugarover 2 years ago
Who is paying $15&#x2F;hour for 64core 256gb??? That’s like 5x what it should be.
jurmousover 2 years ago
[deleted by author]
评论 #33919999 未加载
iLoveOncallover 2 years ago
&gt; In short, if a developer is waiting on a build to run for one hour and doing nothing in that timeframe<p>This should be at the very top of the article. It basically negates entirely the whole experiment because no developer will just sit and wait for hours for a build (unless they planned on taking that break anyway).<p>The cost is not 0 but it is absolutely negligible.
评论 #33919777 未加载
评论 #33919957 未加载
评论 #33919650 未加载
rpigabover 2 years ago
That&#x27;s why you should only hire contractors, and shouldn&#x27;t pay them while a build is running.<p>Saves more money that way, and you don&#x27;t have to care much about build duration.
评论 #33919421 未加载
评论 #33919435 未加载
评论 #33919540 未加载
评论 #33919462 未加载