Not gonna lie. I designed a system that looks exactly like this. However, the plugin system was an operator instead of annotating a service which is kinda cleaner but it still felt dirty. A better approach is having a service registry to track the versions, values and clusters where this has been pushed but that means you needed a different API or extend the k8s API and deal with the whole auth/authz system.<p>I think the biggest hurdle to implement a solution like this is your application MUST be a 12 factor app which sadly is not the case from what I've seen. Many devs hardcode values when shipping their code and containers which make it really difficult to add nodes to a DAG. Had this change, SDLC testing would be waaay easier.<p>In the same space, labeling of all services is a thing. As you are annotating it can be tracked but if you had different gateways or destination routes or other things that need to be tweaked at the mesh level, it could be daunting to track it. In my case, I ensure the whole mesh representation of a version could be tracked by a label so if you had a app=vote1,version=release-4,feature=shiny you could track the shiny components across the whole cluster.<p>Another hurdle is you are tied to a service mesh implementation. Istio is ok but it can be a beast. It also constraints your org to implement other networking alternatives which is something we wanted to explore.<p>I do like the project uses Nix =).