We turned our monolith into a bunch of micro services almost 6 years ago to the day. For a long time I was very happy with the new pattern but over the years the weight of keeping everything updated along with the inevitable corners that fall behind and have...questionable..security due to how long they sit neglected has really left me wondering if I am happy with it after all.<p>I would love hear some thoughts from others that made the move, especially anyone that decided to move back to a monolith repo.
Go is a very weird language :-(.<p>It's very limited when you started to do complex thing. Example, let's say you are building websocket. You will have a hard time to write type safe websocket handler to process the payload from client for all the events...<p>I started to do Rust/Crystal and both of them are better than Go(performance, type system).<p>Yet, whenever I build something for work, I come back to Go :-(. I told myself to use Rust or Crystal.<p>Then I realized that Go is a practical language. It compiled fast so it makes testing easier. The cross compiler just make it so easy to build binary run on everything thing. And the limitation of Go makes it very consistent on how you do thing. This makes working with Go become faster event by the fact that it slows you down on other parts.<p>So I think Go is a language that people easier to fall into because it has the speed of interpreter language like Ruby/Python(or even faster) during development and have a better performance/type safe story.
just trying to understand - you guys think moving a Python2 monolith to Python 3 is too painful, and so you are going to port all the code from Python2 to a completely new language (Go), change the architecture (monolith -> microservices) and move the HTTP API to React + GraphQL, all in one year?<p>2020 is going to be in an interesting year at Khan Academy ;-)
> If we moved from Python to a language that is an order of magnitude faster, we can both improve how responsive our site is <i>and</i> decrease our server costs dramatically.<p>I see people say things like this a lot but my experience is that while other languages are 10x or more faster than python in some benchmarks it's very rare that computation time dominates server latency or that servers are running at 60%+ cpu across all cores.<p>If 90% of your service latency is not directly on the cpu and/or you haven't profiled to see that the performance bottleneck is evenly distributed across all tasks, then it's super dangerous to migrate to a new language thinking that will fix it.<p>I hope people inside Khan Academy know this and it's just a clickbait blog. If they really think "go is 10x faster than python so we'll only need 1 server for every 10 when we migrate" then I think they'll be disappointed.
Migration from python 2 to 3 is easy and fast. I've migrated multiple large apps and it took about a day each. Most libraries that matter have been migrated. Some don't even support python 2 anymore. It's practically 2020. This should not even be a consideration. After 2 to 3 is done they should consider again If they want to redo the stack but first I'd focus on this small maintenance task.
Why do they call them "micro services" and not distributed systems? Oh right, it's because distributed systems are obviously really hard to create correctly and no sane person would ever agree to pay for that.<p>Nice: re-branding. I can't wait for the, maybe "consolidated computing" manifesto (aka turning micro services back into monoliths).
I think some of the misunderstanding in these comments comes from not fully appreciating the perspective of not-for-profit organisations. While I can't speak for Khan Academy, I know that in every NFP organisation I have worked for there is an acute awareness that funding could dry up one day and the prime directive is to ensure that in a scenario like that, the work of the organisation can continue.<p>In this case, it leads to a higher concern about minimising the cost of the operational services than you might have in a for-profit organisation. In all the strategic planning I have been involved in with NFP, we always have the "what if worst case scenario arises" plan and in that plan the ability to scale down to bare minimum operational cost is key. It may not be conscious but I suspect that may be part of the reason the performance savings from moving to Go are so attractive in this case, where most profit-making companies just ask the question of whether they can afford to pay for the servers with their current margin or not and if they can they have more important things to worry about.
So, some potential pitfalls:<p>- The decision seems to be primarily a software architecture one, without much mention of all the other architects whose input will shape how the finished product is run and supported. In a modern software development environment, all the other parts of the org should be consulted on greenfield work to "Shift Left" anything that may need to change down the pike. Design in a silo leads to ineffective products.<p>- They're going from "hmm we need to upgrade from Python 2 to Python 3", to "we need to redesign everything in a new language with a radically different software architecture". This is definitely the second system effect. It's going to take years to make this thing reliable and sunset the old product.<p>- They're porting over the logic? Even if this is actually the right move, wouldn't a clean-room implementation potentially give better outcomes?<p>- Why are they continuing to use App Engine if the writing's on the wall for 2024?
Looking at the case where khanacademy is migrating their server only after about 10 years. I realize more that I don't have to worry that much about being locked into certain technologies (unless it's clearly untransferable, e.g. storing part of customer data in 3rd party server), because after all, we might keep it only for about 10-20 years, and the thing I'm working at almost certainly will only last < 2-3 years.
> We’ll only generate web pages via React server side rendering, eliminating the Jinja server-side templating we’ve been using<p>I’ve been down this road. Deep down this road. Let me just give you a heads up on something I didn’t consider at the time: Most template languages do not parse every single node, one by one. In a sense they are just doing string concatenation. Not so with server side rendering and React. I’m not saying it can’t be done but just realize it is going to take a lot more compute power. Caching is great of course but won’t help you if you plan to customize user content during the server side rendering as well. My recommendation is that you don’t do any user authenticated stuff during SSR.<p>Also consider how you are going to handle cookies if you do plan to make authenticated requests to server side rendering. Also solvable but for some reason people had the hardest time understanding why we had to forward cookies to the domains we controlled in an API request and definitely not to any other servers.<p>I’m not sure I would pick React for an SEO driven website. It is hard to get a competitive “time to first byte”. Unless of course you can pre warm a cache of every one of your pages.<p>Lastly, you’re going to need Node for the SSR. I’m sure you know this but that might take you out of app engine and into cloud compute. Not a big deal but thought I’d mention.<p>Good luck! It is doable. If you ever want to chat about how we solved some of these problems I’d love to save you some time if I can. Hit me up in my profile email.
The article says this: "Moving from Python 2 to 3 is not an easy task."<p>I disagree with this. It's a Python project's dependencies that make it hard to move from 2 to 3, and most libraries have been updated.<p>Of course, you could argue that it isn't easy to migrate a codebase from one major version of a language (or framework, or database) to another, but when you eliminate <i>easy</i> from your vocabulary it becomes harder to describe different levels of difficulty.
I've been in a similar boat. We've been splitting up or converting large Python 2.6/2.7 applications into Go services (and doing the same to large Perl applications) for a long time now.<p>Go has consistently been 10-20x performant (allowing for dramatically reduced hardware needs), easier to maintain, and more productive to produce code in than our previous Python (Twisted) and Perl (AnyEvent).<p>Hopefully KhanAcademy has solid telemetry data in both legacy and new code so they can quantify benefits. They will also have a learning curve for managing multiple micro services vs monoliths. Accessing shared data will be a problem they will likely have to solve. We've opted for each service controlling its own data - no reaching into another service's data behind its back. Everything through APIs. This gives the microservice the ability to alter its datastore as it needs to and not be blocked by other teams' need to update how they access the data.<p>Debugging a distributed solution is much harder than a single service. Distributed tracing, consistent structured logging with log aggregators that let you do fancy searches (like Splunk), and application telemetry and metrics will be even more important than before.
I heard about this project from a friend who works at KA. I am concerned about the strategy, and I think the following approach would yield better results:<p>1. Write in Go an exact reimplementation of the current Python codebase. Use the same database schema, front-end HTML/JS, test suite, and so on. To whatever extent possible, use the same names for classes and functions. Check the reimplementation correctness by using a comparison tool that calls both the Python and Go version of a page/function/search and making sure that they produce the same results.<p>2. Change the production code over to the Go version, perhaps using a ramping strategy where X% of servers are running the Go code, and you gradually increase X, while monitoring vital statistics like server load and response time.<p>3. Now that the production site is running Go, incrementally split off components into their own services.<p>This approach leads you to the same destination, but with a lot less risk. It is very unhealthy to have a situation where the production site is running one codebase but all the developers are working on another codebase. Note that you will realize the benefits of Go (performance, type safety) after step 2, which is much sooner than OP's plan.<p>Joel Spolsky's classic essay about how you should never do full codebase rewrites is worth reviewing:<p><a href="https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/" rel="nofollow">https://www.joelonsoftware.com/2000/04/06/things-you-should-...</a>
It is crazy to see they are still using python 2. Seeing how slow the conversions to python 3 have been, was creating python 3 a good decision for python community? Can it be argued that developing python 2 further in a backward compatible way would have been better for the community? I know that evaluating this kind of thing is hard as metrics are bound to be subjective and speculative. But I am curious if there was any serious attempt to figure it out.
> Now, in 2019, Python 3 versions are dominant and the Python Software Foundation has said that Python 2 reaches its official end-of-life on January 1, 2020 , so that they can focus their limited time fully on the future. Undoubtedly, there are still millions of lines of Python 2 out there, but the truth is undeniable: Python 2 is on its way out.<p>The Python 2/3 split is by far the most annoying thing about Python. I don't develop software in Python but about half the time I've had to use a Python library or program the problem of 2/3 incompatibility has cropped up. Some projects don't make it clear whether one or the other is required, leading to further confusion.<p>If anything the Python 2 EOL could make a bad situation worse. Like Khan Academy, each Python 2 package maintainer will be forced to make a decision: move to Python 3 or abandon and maybe move to an entirely new language. It think many will choose to abandon, leaving these packages to rot.<p>Second on the list are the multiple package managers (or things looking like package managers).<p>Third on the list of annoyances are native extensions, driven by the poor performance of Python itself. These extensions make it difficult to use certain libraries across operating systems.<p>So as a non-Python developer I don't look forward to the occasions when I must use a Python-based piece of software.
As much as I personally don’t enjoy writing Go I really can’t fault them.<p>I still find it interesting that for a relatively obvious feature set of fast compiles, fast startup and fast runtime there really isn’t anything mainstream out there to compete with Go.<p>I really hope something like Kotlin, Swift, ReasonML or even AOT JVM/.NET brings something to the table soon. Or perhaps I’ll just have to wait for WASM to really take off server side.
Seems like such a waste. Is switching to python 3 really that hard? Is hardware that expensive? If this is indeed the right call it doesn't bode well for traditional scripting languages as the web scales to fewer high traffic apps. We might start to see more jvm, go (apparently) or even rust and c(++), rather than speed of development languages like Python or Ruby. Trend seems to be the reverse though, with python the second and most rapidly growing language.
Yikes! It's a lot of effort to reduce memory use. They might be better off creating a new Go entrypoint/server that can call into CPython to reuse all their existing/tested modules (treat their Python as a microservice called by Go). They could then use Go to create/call new microservices or replace various routes on a selective basis.
I think the real problem is that they didn't properly maintain their code. Rewriting it in Go won't prevent them from dealing with this in a few years for when this Go version reaches end of life. I would have liked to see an article on "introducing process" side of programming.
Unpopular opinion, writing Go is faster than Python. With the compiler, strong typing, and no versioning hell, I'm much more productive in Go.<p>Whenever I use python I run into problems with versions and dependencies. And the whole community just tells me to use pyenv or virtualenv and it will "fix all my issues". Only it doesn't.
The article mentions Go's superior compile time, when compared to Kotlin. I have done a lot more Java development than Kotlin, but my recollection is that both of them compiled fairly fast.<p>Is Go really significantly faster to compile for similarly sized projects?
What a mistake.<p>Not only going from a monolith to microservices but also changing the language? This is a mistake that rookies make. This will be one of those post-Mortems where they will sheepishly admit they bit more than they could chew, and it wasted years of productivity.<p>There’s no reason to move to Go. Stick with Python for now. Migrate safely to python 3. Once everything is stable, start breaking things up into thrift or protobuf services. They don’t even need to be microservices but you need the contract. Once that is stable migrate to whatever language you want. But at this point you will have the well-defined api and test cases. Trying to do too much all at once is a no-brained disaster.
You can start porting your live codebase to python 3 right now - no downtime at all. You can have a 2+3 compatible codebase live in a week or two, it'll get you about 98% the way there [1]<p>My projects (e.g. <a href="https://tmuxp.git-pull.com" rel="nofollow">https://tmuxp.git-pull.com</a>, <a href="https://unihan-etl.git-pull.com" rel="nofollow">https://unihan-etl.git-pull.com</a>) are both python 2/3 compatible.<p>I learned by reading through other projects like Sphinx, Flask, Werkzeug, and SQLAlchemy:<p>- Flask: <a href="https://github.com/pallets/flask" rel="nofollow">https://github.com/pallets/flask</a><p>- Werkzeug: <a href="https://github.com/pallets/werkzeug" rel="nofollow">https://github.com/pallets/werkzeug</a><p>- Sphinx <a href="https://github.com/sphinx-doc/sphinx/tree/v1.8.5" rel="nofollow">https://github.com/sphinx-doc/sphinx/tree/v1.8.5</a> (see compat: <a href="https://github.com/sphinx-doc/sphinx/blob/v1.8.5/sphinx/util/compat.py" rel="nofollow">https://github.com/sphinx-doc/sphinx/blob/v1.8.5/sphinx/util...</a>)<p>- <a href="https://github.com/sqlalchemy/sqlalchemy" rel="nofollow">https://github.com/sqlalchemy/sqlalchemy</a><p>Helpful blog posts: <a href="http://lucumr.pocoo.org/2011/1/22/forwards-compatible-python/" rel="nofollow">http://lucumr.pocoo.org/2011/1/22/forwards-compatible-python...</a>, <a href="http://lucumr.pocoo.org/2013/5/21/porting-to-python-3-redux/" rel="nofollow">http://lucumr.pocoo.org/2013/5/21/porting-to-python-3-redux/</a><p>As for automation tools, for 2/3 compatibility, I used futurize and huge codebase to great success: <a href="https://python-future.org/futurize.html" rel="nofollow">https://python-future.org/futurize.html</a>. I started with this:<p><pre><code> futurize --write --stage1 --unicode-literals --nobackups <files>
</code></pre>
and <a href="https://docs.python.org/2/library/2to3.html" rel="nofollow">https://docs.python.org/2/library/2to3.html</a><p><pre><code> 2to3-2.7 -w -n <files>
</code></pre>
Really helped.<p>Also, wire Travis / your CI system to have your tests run on python 2 and 3.<p>If you want to test your python 3 codebase live on a subdomain / same SQL/NoSQL DB, be careful about jobs/tasks! Pickle version mismatches and stuff. Use a separate redis/whatever DB for the deployments.<p>After you're finally on python 3, you can use <a href="https://github.com/asottile/pyupgrade" rel="nofollow">https://github.com/asottile/pyupgrade</a> to modern your code.<p>[1] For the other 2%, use conditionals in your requirements file:<p><pre><code> enum34==1.0.4;python_version<"3"
ushlex==0.99.1;python_version<"3"</code></pre>
Ok this is going to sound ignorant, as my only experiences in backend services have been Go and Python. I don't like either. Is there something I'm missing? For simple CRUD apps, both are sufficient. But (in my limited experience), the moment I've wanted to create more complex business logic with stricter constraints, neither has been quite up to the task.<p>Go doesn't make things easy. It asks you to repeat yourself. I don't like the lack of basic functions like a generic map / filter function.. I know Rob Pike just says "use for loops", but it feels so unnecessarily unexpressive. When I see map, I know what's going on almost immediately. For loops take more reading to understand. Nil pointers shouldn't be, yet still are, a thing (developers aren't perfect - why can't the type system help?). It feels like a straight downgrade from Python from a code clarity perspective. And it's typed, sure, but the type system doesn't let me express constraints that other languages allow me to do that would prevent entire classes of bugs. It doesn't feel worth it compared to Python. Yes, the core language ends up being comparatively "simple", but simple building blocks doesn't guarantee a simple overall system. And my company is very diligent from an architectural perspective.<p>But then when I look at Python, I'd rather just use Javascript with Lodash, esp. when it comes to the treatment of functions as first class objects[0]. Throw Typescript in there, and you get, in my opinion, a better type system than Go, so unless language performance is a major constraint (which it hasn't been for my company, our DB usage patterns it the biggest thing instead), why would I want to use either of these rather than Typescript?<p>[0] Edit: Dumb and wrong, I meant its treatment of anonymous functions. I don’t like lambdas.
> Go, however, used a lot less memory, which means that it can scale down to smaller instances.<p>I could be mistaken, but this sounds like they went ahead with the default JVM settings, where it tends to use as much memory it is allowed to (which makes sense from a utilization and efficiency perspective). If memory usage is a concern, the JVM can be tuned for such.
Kotlin looks nice now but on the long term it's a bad choice since they're stuck on Java 8 they will always lag behind the real JVM and won't be able to catch up since they need heavy modifications.
Interesting read. It always surprises me that companies go many years running major apps and infrastructure with interpreter-based languages like Python and Ruby to begin with. It's an incredible waste of energy, compute, and for web apps sometimes, users' time.<p>Developers need to be a lot more disciplined about performance and efficiency. I'm glad Khan went to Go, but man all those years wasted.