This doesn't surprise me and matches my own experience. Rust literally makes a codebase nearly void of most bug classes with the exception of logic bugs (Unfortunately, in a huge codebase, there can still be tons and tons of logic bugs).<p>Still, when I migrated my Python codebase to Rust I got rid of whole classes of bugs and honestly code faster in Rust on a "per debugged line of code" basis. In Python, every line MUST be run or could trigger an exception (and so many do and it is hard to see). This now requires a test harness that tests every single line which is a huge pain. In Rust, the things that can obviously panic are pretty easy to see and are limited (unwrap, expect, indexing, etc.) making it much easier to write robust code, even with a more limited test suite.
> Dev velocity, which was supposed to be the claim to fame of Python, improved dramatically with Rust.<p>I don't doubt this in the least. I've been a professional Python developer for 15 years, and I can't believe Python ever had the reputation for "high dev velocity" beyond toy examples. In every real world code base I've worked in, Python has been a strict liability and the promise that you can "just rewrite the slow parts in C/C++/Numpy/etc" has never held (you very often spend at least as much time marshaling the data to the target language format than you gain by processing in the faster language <i>and</i> you have all of the maintainability problems of C/C++/Numpy). Python trades developer velocity for poor runtime performance.<p>I don't think Rust is the paragon of high dev velocity languages either, but it seems to be more predictable. You don't have something that works as a prototype, but then you run into a bunch of performance problems (which virtually cannot be worked around) when you go to productionize, nor do you get all of the emergent quality and maintainability issues that come about from a dynamic type system (and last I checked, Python's optional static type system was still pretty alpha-grade).<p>I strongly recommend avoiding Python for new projects. Use Go if you're looking for an easy GC language with max productivity and a decent performance ceiling. Use Rust if you're writing really high performance or correctness-is-paramount software. I'm sure there are other decent options as well (I've heard generally good things about TypeScript, but I'd be concerned about its performance even if it is better than Python).
So I was one of the 3 hackers who wrote the LSM that predated Rocks at FB, and as a result I distinctly remember when Rocks overtook it in features by trading off performance, and again when it met and exceeded the performance while keeping the greater capability. RocksDB kicks ass and has for a long time, I’d recommend it to anyone in the market for that sort of thing.<p>FAISS is also great, and has been for years. I don’t doubt that someone has done a meaningful rev on it.<p>But as someone who likes Rust and wants to like it more: don’t lead with that. The software speaks for itself. Or should.
> ..complex runtime issues which were almost impossible to reproduce or isolate<p>> That’s when internal murmurs about a complete rewrite started brewing…<p>> We decided to move our entire codebase to Rust<p>> there was still one minor problem - no one on the team knew Rust<p>"We have runtime issues so we will fix them by a complete rewrite in a language that no one on the team knows."<p>I would not call that a "hard decision", rather "rolling the dice".
It awesome that they managed to make it work.
Writing Rust at scale, I concur with some of these findings. Nowadays, my team exclusively writes Rust. Our rationale is two-fold.<p>1) We want to write software that is fast and inexpensive to run. Our rust services are quickly growing, but cumulatively, they remain in the sub-1000 core count, while processing many millions of requests per second across our service with excellent latency.<p>2) We want to write software that won't break and page us in the middle of the night, or blow up when we deploy it. We operate some of the core persistence primitives at our company, so the consequence for error in the worst case could be corruption or loss of user data. Our experience thusfar with Rust is that we're able to confidently iterate and deploy our services. The developer velocity is indeed higher, and continues to grow with your experience in the language.<p>Our team of 10 (with 3 engineers having started in the last 2 weeks) owns all of the database clusters, and all of the services surrounding them. Our on-call is surprisingly quiet, with maybe 0-3 low sev incidents a week, with most of them being due to the underlying database/hardware, and not the rust services. We are actually writing a Rust control plane for our databases soon, which seeks to automate most of the response and remediation of most of the incidents we experience in a given week, and also to automate a bunch of manual work that we currently do around operations of the databases.<p>The learning curve can be a bit rough, but our team of engineers is motivated to learn, and we have a great culture of mentorship and teaching from the engineers who are familiar with Rust.
I like this "and an overzealous compiler increased engineers’ confidence in pushing changes".<p>If you think about it, you can try to optimize a Rust program through trying brute force stuff and trying weird things that you are not even sure will break something or not. But you can try ~without 100% knowledge or confidence~ and boom the compiler will tell you if you are shooting yourself in the foot. Now try to that in C++. Even with 95% knowledge and confidence I will be very worried as a Lead or CTO.
> Dev velocity, which was supposed to be the claim to fame of Python, improved dramatically with Rust. Built-in testing, CI/CD, benchmarking, and an overzealous compiler increased engineers’ confidence in pushing changes, and enabled them to work on the same code sections and contribute simultaneously without breaking the code base. Most impressively though, real time operational events dropped almost to zero overnight after the original release.<p>I think Rust is really shaping up to be in a sweet spot.<p>One thing that is not talked about enough with Rust, is that you do not need to start with fighting the borrow checker from day one.<p>Making copies and using Rc can bypass a lot of the trickier parts of the borrow checker and still allow you to access much of the benefits of Rust in terms of performance and correctness, especially compared to a dynamic language like Python.<p>As you get more experience, you can then use references and other techniques to eliminated unnecessary copies, etc.
I don't want to be dismissive but how much of this success belongs to the team of rock solid senior engineers with enough experience under their belt rather than rewrite in Rust?
Writing something for the second time is a significantly different experience so that makes the conclusion much less definitive to me. Did they need python at all the first time - maybe not.<p>Python "velocity" is, to me, more about the ease with which you can make changes without breaking other code. You can still break other code and if you don't have a proper test suite then you're wasting your time - which is something that smells slightly in the story.
Maybe it’s just me but it seems irresponsible to move your entire team to a programming language none of them knows.<p>Rust is great, but modern C++ is great too.
Something I still struggle to understand with these systems is why indexing 1-10M things is hard / their sweet spot, when that is basically a small dataframe in pandas or a small SQLite, vs the 10B+ rowa // GBs+ I was expecting them to be discussing
Frankly on larger projects overall time to create and test the whole thing in modern C++ or (I assume) in Rust should not take longer than in Python. I do not understand this idea of mixing languages where one is enough.<p>In my own case I rewrote decent size Python app in C++ and it had taken me about the same time as for original developers. And no I was not porting code but rather working with the specs so did my own design.
I'd be interested in a bit more detail on the technical migration strategy. Did they just cut over to a version of the project built in Rust? (I.e. vN is Python+C/C++, vN+1 is Rust?) Or something more gradual?<p>How did they verify that the new code was conformant with the old app's behavior/logic?<p>> To make matters worse, we would discover issues only after deploying (or in production!) due to Python’s run time nature.<p>I don't want to infer too much, but this makes it sound like perhaps they didn't have a very robust set of E2E/Acceptance tests, which would make a full-cutover migration scary to me. If you're finding Python bugs only after deploy, how can you find the inevitable rewritten-Rust-code incompatibilities before deploying/production?<p>I've been digging into Rust/Python interop recently using <a href="https://github.com/PyO3/pyo3" rel="nofollow">https://github.com/PyO3/pyo3</a> and maturin, and this points to an interesting migration strategy; PyO3 makes it quite easy to write a Python module in Rust, or even call back and forth between them, so you could gradually move code from Python to Rust. A migration path to full-Rust might be:<p>1. Incrementally replace all your fast-path C/C++ code with Rust, still having Python call these compiled modules. End-state: You now have a Python/Rust project instead of Python/C/C++.<p>2. Gradually grow the surface area of your Rust packages, moving logic from Python into Rust. Your existing Python tests can still run, and your existing entrypoints are the same Python.<p>3. At some point, you presumably need to cut over the entrypoint layer (the API?) to use pure Rust, instead of Python. This should be a much less scary migration since both versions are calling into the same underlying Rust library code. Depending on your architecture, if you have an API Gateway you can split your service backends to migrate one endpoint at a time to the new Rust API service, while keeping the old Python API service around to fail back to. (They are using k8s so you can do this with your Ingress for example).<p>I'm interested in others' experiences with Rust/Python interop, are there any rough edges worth knowing about?
Dev velocity with new projects is often higher so I am not convinced by that argument.<p>With that said. I learned Rust recently. I think it is a viable choice for a business core. Rust is not at all that difficult to learn as some claim. The tuts are great. Some things are very different though.
> we would discover issues only after deploying (or in production!) due to Python’s run time nature<p>I love Rust but seems like they just needed to write some test before shipping to prod?<p>> Built-in test, CI/CD<p>Does it have anything to do with the language?
Python has built in testing and C++ has libraries like catch2. I would have liked some initial comment that said "we had a good suite of tests". Python has plenty of benchmarking tools available.<p>IMHO - a db is all core. Everything should be fast. Even the CLI tools - Rust is pretty good at cli tools. Why is he building a database out of python? How much python is in this repo anyway?<p>Also this is a Python + C++ app - not python. And the article critiques python, not c++. Which is weird, because I would have thought the source of problems would have been C++ not python.<p>Don't get me wrong - Rust is great. Cargo is great too. Compiled programs are great to deploy rather than python (pip hell) and C++ ({CMAKE_HELL}). But this is apples to oranges comparison.<p>The things that make Rust better than C++ / Python are not the things this article talks about.
Would you guys start a GraphQL Api today using Rust? I'm very tempted to hint at that but am still insecure about my 'options'. This is coming from a very comfortable python + fastapi + sqlalchemy ecossystem
The steps some people to go to to avoid writing tests is shocking. (I say this as someone who has written Python programs and been tired of deploying them only to find a runtime bug, and solved it by writing tests)
Leading with several pages of what amounts to smug self-congratulation does not inspire finishing.<p>Rust deserves mention in the title only if actually getting it done in Rust was particularly difficult.
> We decided to move our entire codebase to Rust (and Go for the k8s control plane).<p>> there was still one minor problem - no one on the team knew Rust.<p>Is this real or satire? I can't tell.
Expectation: this article.<p>Reality: It's insanely hard to find talented developers, but Rust makes people excited, so they are just using it as a marketing tool.
> Simultaneous fetches of thousands (sometimes tens of thousands) of objects started becoming inefficient, especially when fetching from collections of tens of millions of objects<p>Why not try leveldb. We're doing random reads of 170,000 vectors (/130M) a second. No startup needed.