Great article. Main takeaway: microservices is not the only option when managing big codebases. In a parallel universe I imagine that the coolest trend in software development right now is a tool for monoliths: all code in a single repo, independent deployable components, contracts in the boundaries and mockable dependant components where needed. As opposed to our universe in which building microservices is the non-official way to go.
A number of years ago, I worked on a team (~20 engineers in total) that successfully carved off two relatively independent portions of a large Rails app using engines. I'm happy to see that Shopify is also using that strategy.<p>I'm curious to know more what sorts of challenges they have around managing dependencies across engines — I think what we were doing was fairly vanilla Rails, and we didn't have the opportunity to run into those sorts of issues.
we are actually doing precisely the same thing at instacart (breaking our 1+ million lines of code monolith into discrete components, which we call "domains"), and typing the boundaries and as much of the internals of these domains as possible with sorbet types.<p>this has the benefit of ruby dynamicism (fast development within domains, you can use all the nice railsy tooling, activerecord, and all the libraries we've built over the years), with type safety at the boundaries (we've also put in timeouts, thread separation, and error handling at the boundaries).<p>the additional benefit for using sorbet is that it makes making typed RPC calls (over twirp or graphql) much easier as you can introspect the boundaries trivially.<p>really cool to see other companies evolving similarly given the same starting conditions!
This is a brilliant brilliant article.<p>Does anyone know how Shopify created it's Architecture Guild and grew it ? The author talks about "should have done it earlier"
Have y'all seen any issues around autoloading of classes/modules in development? I've been working on a rails app composed of a handful of engines and I've noticed that every so often classes aren't loaded. 6 seems to be a lot better about it than 5 was.
Interesting article. We use a similar approach for our mobile apps to allow multiple teams to develop their own modules independently.<p>Can anyone speak to what the advantages and disadvantages to such an approach are as opposed to going full Kubernetes / Microservices? Is it that deploys are riskier and you can't scale separate pieces independently?
Does anyone know if the Storefront rendering described here[0] is running Rails or something else?<p>[0] <a href="https://engineering.shopify.com/blogs/engineering/how-shopify-reduced-storefront-response-times-rewrite" rel="nofollow">https://engineering.shopify.com/blogs/engineering/how-shopif...</a>
There’s so much truth in this. It’s full of lessons I tell clients at the outset of similar endeavors yet they often do not heed until they experience the pain first hand.
Interesting read. I've seen a component based rails architecture work wonders for cleaning up a codebase and allowing for the benefits of a SOA encapsulation while still keeping everything under a monolithic architecture (and avoiding the networking nasties). Not such a fan of sorbet though, but hopefully something better comes along.
I can imagine that this has been a huge effort, and kudos to the team, but this is a solved problem; there are ample methodologies to resolve the big ball of mud.<p>IMHO the shopify team could have saved a lot of time by getting some schooling about strategic DDD, and consulting one or more DDD experts to draw a first version of their context map.
I think it's kind of bad that we have this trend to use "walls" to enforce modularity. This whole thing about using "walls" to enforce "developer behavior" is, in my humble opinion, the wrong direction.<p>If you think about it, almost all lack of modularity comes from shared mutable variables. Segregate mutability away from the core logic of your system and the smallest function in your architecture will become as modular as a microservice.<p>Really, any function that is stateless can be moved anywhere at anytime and used anywhere without fear of it being creating a permanent foothold in the architectural complexity of the system. So if the code is getting to structured where you become afraid of moving things... do this rather than build classes and walls around all your subroutines.<p>Remember as long as that add function doesn't mutate shared state you know it has zero impact on any part of the system other than it's output... you can replace it or copy it or use it anywhere.... this is really all you need to do to improve modularity of your system.<p>>Again and again we pondered: How should components call each other?<p>I think this is what's tripping most people up. They think DI IOC and OOP patterns are how you improve modularity. It's not. Immutable functions are what improves modularity of your program. The more immutable functions you have and the smaller they are the more modular your program will be. Segregate IO and mutations into tiny auxiliary functions away from your core logic which is composed of pure immutable functions.<p>>Circular dependencies are situations where for example component A depends on component B but component B also depends on component A.<p>I've never seen circular dependencies happen with pure functions. It's rare in practice. I think it occurs with objects because when you want one method of an object you have to instantiate that object which has a bunch of other methods and in turn dependencies that could be circular to the current object you're trying to call it from. In essence this kind of thing tends to happen because when you call a method you're actually calling a group of methods and state within a class and upon all those dependencies as well increasing the chances of a circular dependency.<p>Still I've seen this issue occur with namespacing when you import files. Walls aren't going to segregate this from happening. You need to structure your dependencies as a tree.
I think it's kind of bad that we have this trend to use hardware to enforce modularity. If it's a performance issue, sure break it up into more hardware. If it's just code modularity than by shifting to microservices you are adding additional complexity of maintaining multiple services on top of modularizing the system. In short it's overkill. This whole thing about using hardware to enforce "developer behavior" is stupid. You can use software to enforce developer behavior. Your operating system, your programming language is already "enforcing" developer behavior.<p>Additionally, your microservices are hard lines of modularization. It is very hard to change a module once it's been materialized because it's hardware.<p>If you think about it, almost all lack of modularity comes from shared mutable variables. Segregate mutability away from the core logic of your system and the smallest function in your architecture will become as modular as a microservice.<p>Really, any function that is stateless can be moved anywhere at anytime and used anywhere without fear of it being creating a permanent foothold in the architectural complexity of the system. So if the code is getting to structured where you become afraid of moving things... do this rather than go to microservices.<p>>We can more easily onboard new developers to just the parts immediately relevant to them, instead of the whole monolith.<p>Correct me if I'm wrong but don't folders and files and repos do this? Does this make sense to you that it has to be broken down into hardware?<p>>Instead of running the test suite on the whole application, we can run it on the smaller subset of components affected by a change, making the test suite faster and more stable.<p>Right because software could never do this in the first place. In order to test a quarter of my program in an isolated environment I have to move that quarter of my program onto a whole new computer. Makes sense.<p>>Instead of worrying about the impact on parts of the system we know less well, we can change a component freely as long as we’re keeping its existing contracts intact, cutting down on feature implementation time.<p>Makes sense because software contracts only exist as http json/graphql/grpc apis. The below code isn't a software contract it's only how old people do things:<p><pre><code> int add(x: int, y: int)
</code></pre>
Remember as long as that add function doesn't mutate shared state you know it has zero impact on any part of the system other than it's output... you can replace it or copy it or use it anywhere.... this is really all you need to do to improve modularity of your system.<p>Editing it on the other hand could have some issues. There are other ways to deal with this and simply copying the function, renaming and editing it is still a good solution. But for some reason people think the only way to deal with these problems is to put an entire computer around it as a wall. So whenever I need some utility function that's located on another system I have to basically copy it over (along with a million other dependencies) onto my system and rename it... wait a minute can't I do that anyway (without copying dependencies) if it was located in the same system?<p>>Again and again we pondered: How should components call each other?<p>I think this is what's tripping most people up. They think DI IOC and OOP patterns are how you improve modularity. It's not. Immutable functions are what improves modularity of your program. The more immutable functions you have and the smaller they are the more modular your program will be. Segregate IO and mutations into tiny auxiliary functions away from your core logic which is composed of pure immutable functions. That's really the only pattern you need to follow and some languages can enforce this pattern without the need of "hardware."<p>>Circular dependencies are situations where for example component A depends on component B but component B also depends on component A.<p>I've never seen circular dependencies happen with pure functions. It's rare in practice. I think it occurs with objects because when you want one method of an object you have to instantiate that object which has a bunch of other methods and in turn dependencies that could be circular to the current object you're trying to call it from. In essence this kind of thing tends to happen with exclusively with objects. Don't group one function with the instantiation of other functions and you'll be fine.<p>Still I've seen this issue occur with namespacing when you import files. Hardware isn't going to segregate this from happening. You need to structure your dependencies as a tree.