Meta-comment, and I’m <i>not</i> claiming this is what’s true about Discord.<p>Every time I see a company post about how they’re changing programming languages, I tend to believe it <i>actually</i> happened because engineers with enough political clout decided they wanted everybody to change to their preferred language, not truly because of technical reasons, but because they just wanted to be paid to write in a cool language. But, of course, you’d be lambasted by your colleagues if you <i>didn’t</i> provide technical reasons, so you go and hunt for technical reasons that seem insurmountable and build a case for it, turning a blind eye to perhaps less exciting solutions that don’t require a programming language change.<p>(It’s sort of like how a cop in America can find you doing something illegal in any circumstance if they look hard enough. Except in this case, it’s a matter of finding technical deficiencies.)<p>This strategy highly depends on clout though, which is why I think technical arguments are frequently really a façade. If, say, a junior programmer doesn’t have clout, even presenting an equivalent technical argument won’t lead to a change in programming language, especially if it’s the senior’s favorite. I’m sorry to say, but most tech organizations aren’t meritocratic or egalitarian in this way (and perhaps they shouldn’t be).<p>As it pertains to the article—which again I don’t claim is all “fake” politicking (I don’t know anything about or anyone at Discord)—they mostly ripped on garbage collection. They specifically talked about how GC was bad because their LRU cache had to be scanned, which was part of why they observed spikes in their metrics. I wonder why they didn’t try doing things like putting the LRU cache into foreign, unmanaged memory to avoid garbage collecting it. Easier said than done, but why not?<p>I think using <i>any</i> language with any serious and large project will find deficiencies in the language. At some point in a product lifecycle it’s more profitable to just work with and ameliorate those deficiencies than do rewrites where the <i>new</i> deficiencies will be completely unknown.
GO promised to be very fast, but if you check the actual benchmarks:<p><a href="https://www.techempower.com/benchmarks/#section=test&runid=8ca46892-e46c-4088-9443-05722ad6f7fb&hw=ph&test=plaintext" rel="nofollow">https://www.techempower.com/benchmarks/#section=test&runid=8...</a><p>High-level C# (asp.net) is almost twice as fast as GO in the benchmark... Rust is also in the top 3.<p>So, why should I use GO instead of C#, if ASP.NET/C# is so blazing fast and requires a lot less lines of code to achieve the same? -> It makes absolutely sense to switch to C# or Rust.
This was covered a year ago, and is literally the most popular post for "Rust" ever on this site <a href="https://news.ycombinator.com/item?id=22238335" rel="nofollow">https://news.ycombinator.com/item?id=22238335</a>
Original author here. Cool to see it posted again.<p>We're still using and investing in rust at Discord. For example, in the past year we successfully pulled part of the core messaging system out of python and into a rust service. Rust has been a boon for developer productivity and correctness of code.<p>We've also been very happy with the way the rust ecosystem is evolving and the amount of adoption the language has seen.<p>I'm happy to answer questions here.<p>P.S. We're still growing and hiring (including remotely). You can find my email in my profile.
2021 will be about them switching back. All jokes aside, it's really common for engineers to randomly switch between solutions incurring huge costs for the companies they work for. Often as it is in this case with no gain for the company, other than some resume building for the involved team members. We use Go here at Stream and power activity feeds and chat for a billion end users. Works perfectly.<p>Cockroach also wrote about GC optimization: <a href="https://www.cockroachlabs.com/blog/how-to-optimize-garbage-collection-in-go/" rel="nofollow">https://www.cockroachlabs.com/blog/how-to-optimize-garbage-c...</a><p>For caching specifically there is this library as well: <a href="https://dgraph.io/blog/post/introducing-ristretto-high-perf-go-cache/" rel="nofollow">https://dgraph.io/blog/post/introducing-ristretto-high-perf-...</a><p>Nothing wrong with Rust, exciting project, but personally think Go is better for building performant micro services like this. Rust seems more suited for building browsers, drivers etc. Rust's performance is slightly better, but it's not as refined around go routines and developer productivity.<p>Edit: On the other hand, Discord could have made the very valid decision that they have an easier time recruiting for Rust than for Go. Which is in fact a good reason to switch.
Sounds like Discord was running gigantic in memory Go caches.<p>Not that it's the wrong approach, but this is a very real-time system-y problem that Rust is designed for. So I understand why this made sense.<p>Obviously rewriting in Rust worked for them, but I'm curious if they tried the Ballast[0] approach, along with smaller cache's per service. That would make the GC scan on the cache smaller while still avoiding some of the Go startup heap issues. I'm not sure if the target heap size ever was accepted by the Go team, but it may have been easier than a full rewrite.<p>[0] <a href="https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap-26c2462549a2/" rel="nofollow">https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...</a>
In its early days, this forum was full of excited young devs who had read PG's old essays and we're actively <i>looking</i> for technologies that would give them a competitive edge.<p>Now, it's completely the opposite.<p>Now, when a company successfully leverages technologies as a competitive edge and becomes a multi-billion dollar success, the thread is dominated by stodgy criticism for not picking the absolute most mainstream technologies possible.
> When starting a new project or software component, we consider using Rust. Of course, we only use it where it makes sense.<p>Big takeaway here for those who are married to their tools. Rust happens to be just another tool for this company, and it works well for them; where it makes sense of course.
Off-topic: who is in charge of choosing colors for graphs? Being red-green colorblind, choosing blue and purple is makes it impossible for me to read the charts. I know its stylish, but every time i make a graph i choose very high contrast colors. Another point: is it just being color blind or do other people struggle in charts/maps that have shades of a color? like different shades of blue represent different rates of covid deaths across a map. just curious if anyone else struggles with this
i'm curious if allocating a memory ballast would have helped discord in this case?<p>a one line patch in main() with something like<p>ballast := make([]byte, 10<<30)<p>might help their GC spikes, since garbage collection would only trigger when the heap size reaches 20GB, at which point the heap will be reduced back down to 10GB or so.<p><a href="https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap-26c2462549a2/" rel="nofollow">https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...</a>
Switching languages is THE most efficient way to refactor a messy codebase.<p>The past color the idea that a language switch is a waste of time, because in the past the choice of languages was from <i>worse to bad</i> or for <i>same to same-ish</i>.<p>But MODERN languages have something that make this compelling: Actual good practiques baked in the language, as defaults.<p>---<p>So, if a codebase is in trouble you can:<p>- Try to refactor it with "better" idioms...<p>but if this is not the norm in the language/ecosystem you depend in discipline, and that have proved to fail at scale.<p>- Change to "shinny lang" where do the right thing is the default...<p>And now if the developers try to do wrong, them will fight against the current.<p>---<p>Switch to Go is a great way to solve a project with concurrency issues: Go is good at that, and make a compelling case of build good concurrency apps. Similar if switch to Elixir/Erlang.<p>The case of Rust is similar: it make the "best" practiques the <i>default</i>, and <i>very painful to do wrong</i> (and sometimes to do right, but this is not the focus here :) ).<p>So, a language switch is the ultimate clean up for a project.<p>I have done several in the past (I'm contractor) and let me tell you: Any codebase I get is in deeeeeep trouble.<p>The best results have been a language switch, when I move from "bad" to "better". Just the fact the new language have something that is truly better make the move great.<p>Now, the opposite is true: If you are switching and NOT because the new language have that something that is truly in need or worse, is a language with the need of a LOT of discipline (like C, PHP, JS,...) then yeah, is doomed...
I've never seen anyone look into just <i>why</i> Go forces a GC on 2 minute intervals. The code originally appeared in <a href="https://codereview.appspot.com/5451057" rel="nofollow">https://codereview.appspot.com/5451057</a> and there's a bit of discussion on <a href="https://groups.google.com/g/golang-nuts/c/WBlJjT-zu7E" rel="nofollow">https://groups.google.com/g/golang-nuts/c/WBlJjT-zu7E</a>.<p>After reading this, I still don't understand. But I find <a href="https://codereview.appspot.com/5451057#msg38" rel="nofollow">https://codereview.appspot.com/5451057#msg38</a> particularly entertaining: there was going to be an environment variable to control the interval, which might have allowed Discord to keep using Go, but Russ Cox shot it down.
I wouldn't mind looking into Rust to eventually replace our Java and Python based stacks, but every time I look at it, I tend to give some thought to the difficulty of writing Rust as opposed to reading/writing/maintaining Java for an entire team. Frankly I'm worried that not all team members will become good Rust developers, and the codebase will develop serious rot over time. Any experiences with this?
Feels like there might have been a middle ground short of a scratch rewrite. LRU caches are always suspect. LRU is pretty much the worst replacement policy you can choose. I wonder if a smaller cache with a better replacement policy (2Q, ARC, ...) would have solved this specific problem.<p>Another approach I've used with caches in Go is to just use a huge byte slice as the cache without any pointers, so the GC can ignore it.
> Discord has never been afraid of embracing new technologies that look promising. For example, we were early adopters of Elixir, React, React Native, and Scylla. If a piece of technology is promising and gives us an advantage, we do not mind dealing with the inherent difficulties and instability of the bleeding edge. This is one of the ways we’ve quickly reached 250+ million users with less than 50 engineers.<p>this sounds like a recipe for disaster in every other company i have worked with. are Discord's engineers just THAT good or is there something else going on here?
These threads are always full of people criticizing these moves by companies. But nothing is gained of nothing is risked, right? Sometimes these things pay off, and we would never know without early adopters.
Discussed at the time:<p><i>Why Discord is switching from Go to Rust</i> - <a href="https://news.ycombinator.com/item?id=22238335" rel="nofollow">https://news.ycombinator.com/item?id=22238335</a> - Feb 2020 (640 comments)<p>also <a href="https://news.ycombinator.com/item?id=23874336" rel="nofollow">https://news.ycombinator.com/item?id=23874336</a> - July 2020 (7 comments)
I really enjoyed reading this article last time it was posted. I particularly like it because it shows evidence that current tooling wasn't doing the job. I believe I remember seeing some comments on how Discord was on an old version of go that wasn't receiving the benefit around gc collections, but we'll never know if that would've been a solution.<p>That all said, I've noticed that people here are really off golang in favor of rust. I believe I sort of understand. I've recently switched from python to go at work, and I frequently bring up why I like it so much (caveat I'm overly enthusiastic about the things I like related to my work). But again, that all said, at the end of the day, both Python and golang will definitely get the job done.
Incredibly brave of them to go with nightly build of Rust for async functionality. Not as in the weeds, but I would have been afraid of release being delayed forever and being stuck with the nightly release for a long time.
This post is getting a bit long in the tooth, at least regarding relatively new and fast-moving ecosystems like Go and Rust. I wonder if the Go team has determined any way to mitigate these long-running GC pauses for apps that don't allocate often since this article was published.<p>I also wonder how well Rust has continued to work for Discord, if it's been a problem to keep up with the rapid movement particularly in the async ecosystem in the last year or so.
Note: this is an account of using Rust in a single service<p>> The service we switched from Go to Rust is the “Read States” service.<p>And the conclusion doesn't indicate "switching", but rather Rust is an "option" for Discord, and already used in the stack:<p>> When starting a new project or software component, we consider using Rust. Of course, we only use it where it makes sense.
I think the response time / latency to use Cassandra on the "read state" will be higher, and mind you cassandra is written in java (JVM) and has more garbage collection magic and latency than Go. From my point of view, it's not the language, it's your data store.
How timely!<p><a href="https://stackoverflow.blog/2021/02/22/choosing-java-instead-of-c-for-low-latency-systems/" rel="nofollow">https://stackoverflow.blog/2021/02/22/choosing-java-instead-...</a><p>I do believe that there is nothing better tuned for performance vs developer productivity than the JVM.<p>The language semantics may be a question for personal distaste..but then you do have Kotlin (<a href="https://jooby.io/" rel="nofollow">https://jooby.io/</a> ) which ranks fairly near the top on techempower benchmarks <a href="https://www.techempower.com/benchmarks/#section=data-r20&hw=ph&test=composite&l=wqmbun-sf" rel="nofollow">https://www.techempower.com/benchmarks/#section=data-r20&hw=...</a>
Very misleading title makes it sound like they are rewriting all code wholesale - in fact it's a targeted performance rewrite of a smallish component.
The main thing that bothers me about rust is that there is one complete implementation of the language (and one or two very incomplete implementations) and the singular complete one isn't even self hosting.<p>Coming from scheme where everyone and their dog has an R5RS compatible REPL that's pretty disturbing.