It's been three years since the last thread (https://news.ycombinator.com/item?id=16166645), maybe there are more mature solutions now.<p>Interested to hear about current setups, and how it works for you.
We just made all the microservices into one big monorepo and we deploy all at the same time.<p>To be honest we tried to avoid the monorepo but it was hellish. Maybe if each microservices was larger and our team was larger but then are they microservices any more?
I don't keep track. All microservices use continuous deployment pipelines. If you check in code and it passes all the tests, it will make it out to prod some time in the next few hours.
If you need to keep track its probably too late. What makes them services is that they should be able to be deployed without a bunch of orchestration of other services. You can solve this by having backwards compatible apis.<p>That said, to know what changes would actually break things you'd ideally have a suite of tests.
Frankly, we don't do a great job of it. We have some Ansible deploying to OpenShift via openshift applier, that gets run from some Jenkins jobs. We use a form of Git Flow to do branching and tag releases. It's messy.<p>I've been looking at Sentry for this, recently. They have a specific feature for tracking releases (and even relating them to errors vs. commits) which looks very interesting. Haven't tried it yet though.
We're building Cortex (<a href="https://www.getcortexapp.com/" rel="nofollow">https://www.getcortexapp.com/</a>) to solve this problem :) We help you track all your microservices and integrate with all your 3rd party tooling to build a single pane of glass for your architecture. Happy to give you a demo if you're interested!
Two ways I've seen it done reasonably well.<p>The somewhat more modern way with Kubernetes deployments is the Helm "chart of charts" pattern, where your system level deployment is a single Helm chart that does nothing but pull in other charts, specifying the required semantic version of each sub-chart in the values.yaml file.<p>The older, but also much more flexible way I've seen it done is through something a local system architect developed a while back that he called a "metamodule." This was back when Apache Ivy was a preferred means of dependency management when Apache Ant was still a popular build tool and microservices were being deployed as Java OSGi components. Ivy defines a coordinate to uniquely identify a software dependency by organization, module, and revision. So a metamodule was just a module, but like the chart of charts, it doesn't define an actual software component, but rather a top-level grouping of other modules. Apache Ivy is significantly more flexible than Helm, however, allowing you to define version ranges, custom conflict managers, and even multiple dependencies that globally conflict but can be locally reconciled as long as the respective downstreams don't actually interact with each other.<p>Be aware both of these systems were for defense and intelligence applications. Personally, I would just recommend trunk based development and fail fast in production for most consumer applications, but for things that are safety or mission critical, you can't do that and may have very stringent pre-release testing and demonstration requirements and formal customer acceptance before you can release anything at all into ops, in which case you need the more complicated dependency management schemes to be able to use microservices.<p>Arguably, in this case, the simplest thing to do from the developer's perspective is don't use microservices and do everything as a monorepo instead, but government and other enterprise applications usually don't want to operate this way because of being burned so much in the past by single-vendor solutions. It's not totally impossible to have a monorepo with multiple vendors, but it's certainly a lot harder when they tend to all want to keep secrets from each other and have locally incompatible standards and practices and no direct authority over each other.
I'm curious to see how micro service management evolves over time and learn whether or not it will become viable for small companies. Hopefully one day its as cheap as writing a function is.<p>As it stands, with what I've seen and heard about microservices, I'd say the best way to deal with micro service anything is to use a monolith 90% of the time and for the rest of the time make sure your micro service could stand as it's own SAAS if given enough love.<p>Not a direct solution to your problem but might be an indirect one.
Flux with GitOps approach, using Helm charts.<p>All of our of microservices have deployment charts, with frozen image versioning. That way, we can can rollout a whole release knowing they are all compatible with each other and can easily fall back just by using git rollback.<p>CI/CD updates image versions in affected YAMLs on every backend release and Flux keeps staging in sync. When we are happy, we sync to production branch, Flux syncs and it's done.<p>If we spot an issue that we didn't see in staging, we either release a hotfix or rollback.
In a previous job had tons of microservices and tons of environments, so it was getting difficult to track what was deployed where. We opted for a simple solution to this: we wrote a very simple CLI that makes the deployments and at the same time registers the deployment in a DynamoDB table. Then to get a picture of a certain environment we just had to list all services for that environment. You could also list the history of releases for a certain service in a certain environment.
We don't. I'm just waiting the day we reach more than 100 microservices and my company realises that microservices was a bad idea to begin with. That's usually the way it works: learning the hard way.<p>To elaborate:<p>- I do think there is value in "utility microservices". For example: a microservice to send email, a microservice to filter spam, etc. These are the next level libraries (because they do need to run as services 24/7). Management usually don't like these kind of microservices because these "domains" usually don't belong to any particular team, so managers cannot "own" their success.<p>- I don't think there's much value in building microservices for the core of your business (e.g., a checkout microservice, a payments microservice, etc.). The usual argument management gives is: "we'll make teams more independent and they will be able to delivery stuff faster than with a monolith!". While this is sometimes true, "faster software delivery" is not on my top list of prioritites when it comes to build software.
My usual setup is pretty simple with each service in its own git repository with a Gitlab pipeline:<p><pre><code> * build code
* run tests (unit + integration using database)
* build docker image
* push to gitlab registry
* deploy to staging k8s environment by using a custom image that just templates a .yml and does `kubectl apply` against the staging cluster
* optional extra "deploy to production" that works in the same way but is triggered with a manual button click in the pipeline.
</code></pre>
I don't do canary deploys or anything. Just deploy to staging, and if it works, promote to production.<p>For some projects I have "staging test scripts" which I can run from my devmachine or CI that check some common scenarios. The test scripts are mostly blackbox using an HTTP client to perform a series of requests and assert responses. (signup flow scenario for example)<p>I would like to move to a monorepo, but I have not yet figured out an easy way to have a separate pipeline for each service that is only triggered when that service has changed.<p>edit: formatting
Founder of <a href="https://Komodor.com" rel="nofollow">https://Komodor.com</a> here, we track changes and alerts across your complete K8s-based stack, analyzing their ripple effect and then providing devs, DevOps and SRE teams the context they need to troubleshoot efficiently. Independently.<p>Feel free to ask question or reach out :)
You've broken the microservice abstraction if this a problem.<p>A team should own a microservice, you release as soon as the team able to.<p>You version your apis, so you don't break any services which rely on yours.
Push to master -> jenkins runs linting, tests, applies migration (or fail, requiring manual intervention), build sdocker image, k8s deploys to canary, monitors canary for a bit for errors, k8s deploys to production, tags docker image, notifies slack.<p>In the past, instead of canary, we used a staging environment with manual promotion. That was costing us a cool half a million in AWS overpriced machines (but we were committed to spend a certain amount of money per year in exchange for discounts, so it's hard to price things) and it was doubling the testing process (promote to staging, test, promote to prod, test). We have been bitten by issues happening in production and not in staging. With the canary, prod only approach we have higher risks of messing up with real data but we have safeguards in place and the canary approach means that a small portion of the users will see problems. We also have the option to deploy to a canary for devs only.<p>I'm not happy about using / running / maintaining jenkins (terrible UI, upgrade path, API to add plugins, etc) but it does the job and it improved a fair bit over the last 5 years. Jenkinsfile are especially nice, even though not being able to easily run them locally is a bit annoying.
When I worked at Atlassian, we had this issue as well, given all the many services that were deployed for products. A few of us left and created Sleuth [1] to solve it for Atlassian and folks like you. Sleuth helps you know what is deployed, its health, and helps with workflow automation. It also tracks the DORA metrics so you know how healthy a service release processes is.<p>[1] <a href="https://sleuth.io" rel="nofollow">https://sleuth.io</a>
It depends a little on you definition of “microservice”, but we keep track of a lot of our “mostly single responsibility” data-processes that make up the builk of our AD, IDM and organisational database for 10.000 employees and 300+ IT systems with a mix of azure automation runbooks and local tasks that are activated by azure automation to. This gives us a clear picture of when what is run, alert humans on errors and halts processes.<p>For all-ways-on systems we have a simple dash-board that each service interacts with.<p>We don’t have a fancy CI/CD pipeline or anything like that, just a set of rules that you have to follow.<p>Database-wise a service has to register itself with one of our data-gatekeepers, which involves asking for permission for the exact data used with a reason. But beyond that services are rather free to make “add” changes, often in the forms of new tables that are linked with views. It’s not efficient, and we have a couple of cleanup scripts that check if anyone subscribed to all the data, but we’re not exactly Netflix, so the inefficiency is less expensive than doing something about it.
Since the specific question is "how do you keep track of", my build & deploy script copies a quick one-liner dump of git information (SHA, date, environment, branch, etc.) to a directory on a shared server, as a text file. Later I can go to that server and `cat versions/* | sort` to get a report of what is deployed where/when and so on.<p>It helps that I have One Deployment Script To Rule Them All (or really, a couple DSTRTA's). When every service has its own special build & deploy script you have to ask nicely and hope people keep up with it. A lot of CI/CD systems force you into that corner because of an implicit assumption that each build & deploy is its own special one-off.<p>Anyhow, text files rule, at least as an ad-hoc solution.
My team is responsible for about 200 microservices being deployed, some of them have 10 or more pods. We don't do continuous delivery. Instead it's done by 5 different groups deploying once every week or two.<p>Our production deployment jobs are in Jenkins and isolated. It's easy to check what was deployed when. We also have a script written that can run an environment report to see what versions and which microservices have been deployed. Along with their CPU/memory allocations, number of pods etc.<p>Release management tracks which JIRA stories are in which release, they do it mainly by looking at master merges between prod deployments.
Have you heard of Komodor? They just held a joint webinar with Epsagon regarding this very issue!
You can see a recording of the webinar here: <a href="https://www.youtube.com/watch?v=J32ZoiRVvPg" rel="nofollow">https://www.youtube.com/watch?v=J32ZoiRVvPg</a>
Or the product overview here:
<a href="https://www.youtube.com/watch?v=Qgio3vF1sPE&t=6s" rel="nofollow">https://www.youtube.com/watch?v=Qgio3vF1sPE&t=6s</a>
We use Octopus Deploy. On commit it autodeploys to dev and sends the team a slack message ([environment] version x (previous was y) deployed by z). Prod deployments are also done through octopus but "manually" by the team when we are ready to make a release. Usually every week or two.
Looks like someone from AirBnb can shed a light on the topic. They seem to be nailed the Microservices deployments :-)<p><a href="https://www.altoros.com/blog/airbnb-deploys-125000-times-per-year-with-multicluster-kubernetes/" rel="nofollow">https://www.altoros.com/blog/airbnb-deploys-125000-times-per...</a>
We work on a solution - <a href="https://relizahub.com" rel="nofollow">https://relizahub.com</a><p>Our community Discord Server (questions on DevOps and DataOps, not limited to Reliza Hub) - <a href="https://discord.gg/UTxjBf9juQ" rel="nofollow">https://discord.gg/UTxjBf9juQ</a>
Sauron is a solution to help to track as many microservices as you have, indexing these information into an elasticsearch<p><a href="https://github.com/freenowtech/sauron" rel="nofollow">https://github.com/freenowtech/sauron</a>
I hate to state the obvious... but don't do microservices? It's the lunacy of the 2000's ESB craze but without all the formal testing.<p>The only way to accomplish what you're asking for would be extremely thorough mock testing.
CI/CD can make it manageable with Git and Atlassian tools, or you can build a custom web dashboard if needed. Personally I like version tagging and release management based on semantic versioning.
if you want to know how to track a micro services please visit <a href="<a href="https://iostweaks.net">ios" rel="nofollow">https://iostweaks.net">ios</a> Tweaks</a>