Why are we templating YAML?

263 pointsby jaxxstormover 6 years ago

36 comments

joeduffyover 6 years ago

My belief is that we've been slowly building up to using general purpose languages, one small step at a time, throughout the infrastructure as code, DevOps, and SRE journeys this past 10 years. INI files, XML, JSON, and YAML aren't sufficiently expressive -- lacking for loops, conditionals, variable references, and any sort of abstraction -- so, of course, we add templates to it. But as the author (IMHO rightfully) points out, we just end up with a funky, poor approximation of a language.I think this approach is a byproduct of thinking about infrastructure and configuration -- and the cloud generally -- as an "afterthought," not a core part of an application's infrastructure. Containers, Kubernetes, serverless, and more hosted services all change this, and Chef, Puppet, and others laid the groundwork to think differently about what the future looks like. More developers today than ever before need to think about how to build and configure cloud software.We started the Pulumi project to solve this very problem, so I'm admittedly biased, and I hope you forgive the plug -- I only mention it here because I think it contributes to the discussion. Our approach is to simply use general purpose languages like TypeScript, Python, and Go, while still having infrastructure as code. An important thing to realize is that infrastructure as code is based on the idea of a goal state. Using a full blown language to generate that goal state generally doesn't threaten the repeatability, determinism, or robustness of the solution, provided you've got an engine handling state management, diffing, resource CRUD, and so on. We've been able to apply this universally across AWS, Azure, GCP, and Kubernetes, often mixing their configuration in the same program.Again, I'm biased and want to admit that, however if you're sick of YAML, it's definitely worth checking out. We'd love your feedback:- Project website: <a href="https://pulumi.io/" rel="nofollow">https://pulumi.io/</a>- All open source on GitHub: <a href="https://github.com/pulumi/pulumi" rel="nofollow">https://github.com/pulumi/pulumi</a>- Example of abstractions: <a href="https://blog.pulumi.com/the-fastest-path-to-deploying-kubernetes-on-aws-with-eks-and-pulumi" rel="nofollow">https://blog.pulumi.com/the-fastest-path-to-deploying-kubern...</a>- Example of serverless as event handlers: <a href="https://blog.pulumi.com/lambdas-as-lambdas-the-magic-of-simple-serverless-functions" rel="nofollow">https://blog.pulumi.com/lambdas-as-lambdas-the-magic-of-simp...</a>Pulumi may not be the solution for everyone, but I'm fairly optimistic that this is where we're all heading.Joe

评论 #19111994 未加载

评论 #19110709 未加载

评论 #19110779 未加载

评论 #19110941 未加载

评论 #19112417 未加载

评论 #19111777 未加载

评论 #19112106 未加载

评论 #19111195 未加载

评论 #19114460 未加载

评论 #19111806 未加载

评论 #19111190 未加载

评论 #19111630 未加载

评论 #19112621 未加载

评论 #19110371 未加载

评论 #19110400 未加载

评论 #19113784 未加载

评论 #19111818 未加载

评论 #19110842 未加载

评论 #19110644 未加载

BossingAroundover 6 years ago

I know I'm in a minority, but I really dislike YAML... I recently did a lot of Ansible and boy, at the beginning, I was just struggling a lot. Syntactic whitespace kills me.I don't like it in Python either, but for some reason, when I write Python, it's a lot easier. Maybe YAML is just a bit more complex (and Python has better IDE support..?)

评论 #19111313 未加载

评论 #19110069 未加载

评论 #19110071 未加载

评论 #19110261 未加载

评论 #19111669 未加载

评论 #19109836 未加载

评论 #19109147 未加载

评论 #19109476 未加载

评论 #19111167 未加载

评论 #19114030 未加载

评论 #19111469 未加载

评论 #19112341 未加载

aetherlordover 6 years ago

I didn't see this mentioned anywhere else, so another alternative (that I've seen and really like conceptually, but haven't used so far) to all this wildness with YAML and JSON -> <a href="https://github.com/dhall-lang/dhall-lang" rel="nofollow">https://github.com/dhall-lang/dhall-lang</a>, and for kubernetes specifically -> <a href="https://github.com/dhall-lang/dhall-kubernetes" rel="nofollow">https://github.com/dhall-lang/dhall-kubernetes</a>

评论 #19109460 未加载

评论 #19109443 未加载

评论 #19129904 未加载

miohtamaover 6 years ago

If you have been around long enough you still remember the world that was excited about XML and templating it using XSLT. As a hindsight it was a horrible world.Even though YAML is not optimal, it is a human friendly compromise between too verbose XML and machine only JSON. It lacks native templating, leading to funny constructs e.g. with Ansible files. However human kind has made progress and will make progress further, so it is just a matter of time until someone comes up with sane "native templated YAML" and all projects will adopt it.

评论 #19109957 未加载

评论 #19112385 未加载

评论 #19111723 未加载

评论 #19112001 未加载

jrockwayover 6 years ago

I saw this title and immediately knew the article would be about Helm. I don't think anyone wants to use Helm. People use it for a set-and-forget thing that they don't care about (who cares that it's called impressive-leopard-kubernetes-dashboard, after all.)kustomize is much more sane for your own stuff: <a href="https://github.com/kubernetes-sigs/kustomize" rel="nofollow">https://github.com/kubernetes-sigs/kustomize</a>It is actually a little bit too magical for my taste, but I continue to use it because it hasn't done anything stupid. I have one file that maps logical names to images in a container repository. If I create a service called "foo" pointing to selector.app.label="foo" in the base, then in production it's called foo-prd and the label magically updates to foo-prd for the selector. It actually understands what it's generating, and while they might have taken it a little bit too far, it's far better than just dumb text replacement.

评论 #19111808 未加载

评论 #19110859 未加载

SamWhitedover 6 years ago

As others in this thread have said: I ask this question all the time, except s/templating/using/.YAML is insanely over complicated; it's as bad or worse than XML for config files, and it doesn't even have the nice streaming mode. Not to mention that it's a bit of a security nightmare (seriously, who put pointers into the YAML spec?).And, on a more subjective note, YAML is just confusing: between all the significant whitespace and the random single character symbols that no one ever remembers what they do, I never get a YAML document right on the first try.Templating it really does add a whole new level of headache too.

评论 #19109941 未加载

评论 #19111033 未加载

评论 #19119885 未加载

评论 #19113137 未加载

niftichover 6 years ago

File-based configs are a troublesome abstraction: they package unrelated concerns into a rigid document whose form must take a particular, application-dependent shape, and the assembly and disassembly of that document essentially becomes an API where key-value pairs are mixed with complex glue code. The application has to do this internally, but anyone who's generating their configs are also doing parts of this externally.Templates try to bandage over that by drilling down the abstraction to key-value pairs themselves. And imperative constructs that sneak into templating languages are an artifact of wanting to gain expressiveness without losing the benefits of declarative form -- but really, the two are at odds.YAML is a red herring -- we had the same headaches with XML a decade prior. The problem is always that there's relationships among the data (or even multiple instances of the config) that we care about, but that the structure of a single config file at rest cannot model.Databases -- let's say, an SQL one -- are actually among the better solutions, because they allow the universe of config items to live in structured places without overspecifying the exact form the data must take when serialized into a file. Then, data can be normalized where it makes sense to avoid repetition and introduce propagation. An SQL database gives all the tools needed to accomplish this, using mostly declarative code.Databases in a KV sense are often used for configuration, and SQLite's rise has increased richly structured configs that are specified at a higher level than what's typically done with other serialization formats, but the full approach has not caught on outside big enterprise systems and complex applications. Which is a shame, because it's hardly more complex than the current awkward pairing of a full serializer and a templating engine.

评论 #19113803 未加载

TeMPOraLover 6 years ago

Wait, what?I feel this article is missing the bigger problem - one that for some reason just cannot die.The problem is that of gluing strings together. YAML is not an unstructured text file, it's a tree notation. Whatever "templating" or "generation" mechanism you want to use, it needs to respect the tree nature of the language it operates on. It needs to respect semantics.Gluing strings together is literally what causes SQL Injection to exist. It caused countless of defacements on the web, and countless of broken websites. I would think we've learned our lessons, but for some reason, I see these template languages still alive and kicking.

评论 #19112699 未加载

评论 #19112519 未加载

epageover 6 years ago

There have been times I've wanted to templatize my configuration but I don't want to do it with text-based templates but templates within the configuration files syntax (be it yaml, toml, or something else). Not sure what this is called, I've been calling it "structural templating".So far the only things close to this are- Azure pipeline's syntax: <a href="https://docs.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops" rel="nofollow">https://docs.microsoft.com/en-us/azure/devops/pipelines/proc...</a> - Something called Jasonette: <a href="https://docs.jasonette.com/templates/" rel="nofollow">https://docs.jasonette.com/templates/</a> - Something called Jsonnet: <a href="https://jsonnet.org/" rel="nofollow">https://jsonnet.org/</a>Azure Pipeline's approach I think is closest to what I've been looking for.Anything else in this space?

评论 #19110381 未加载

评论 #19110238 未加载

fiddlerwoaroofover 6 years ago

Dhall-lang ( <a href="https://dhall-lang.org" rel="nofollow">https://dhall-lang.org</a> ) is another, somewhat interesting, attempt to solve this program: it comes with a non-Turing complete programming language, so you can bring some abstraction to your configuration files without having to worry about things like infinite loops.

MrStonedOneover 6 years ago

The real question is why are we using yaml at all?

评论 #19109315 未加载

评论 #19109258 未加载

评论 #19108969 未加载

评论 #19109049 未加载

评论 #19108982 未加载

评论 #19109048 未加载

评论 #19108971 未加载

评论 #19109405 未加载

评论 #19109058 未加载

equaluniqueover 6 years ago

Maybe we should be exploring using Dhall instead of YAML.>One of the clearest signals I’ve gotten from users is that Dhall is “the YAML killer”, for the following reasons:>Dhall solves many of the problems that pervade enterprise YAML configuration, including excessive repetition and templating errors>Dhall still provides many of the good parts of YAML, such as multi-line strings and comments, except with a sane standard>Dhall can be converted to YAML using a tiny statically linked executable, which provides a smooth migration path for “brownfield” deployment.Source: <a href="http://www.haskellforall.com/2019/01/dhall-year-in-review-2018-2019.html?m=1" rel="nofollow">http://www.haskellforall.com/2019/01/dhall-year-in-review-20...</a>

rhackerover 6 years ago

LOL we are doing the same with k8s. We deploy to any environment, where each environment is a different k8s namespace. We even have a namespace for each developer. The variables are things like the image name (the tag is effectively a git commit id, or sometimes different for not-committed yet stuff). Beyond that just about nothing really needs to be a variable, but we do have different RAM amounts.We have a couple ways that we template this out, but mostly we literally just do this in bash:<pre><code> sed -e "s/\$CI_COMMIT_SHA/$CI_COMMIT_SHA/" kube-deploy.template.yaml | kubectl -n $ENV apply -f - </code></pre> (Where CI_COMMIT_SHA comes from gitlab) ENV comes from our gitlab CI file.That all being said, the extent of our k8s integration is lots of stuff like that. We could write a JS file that creates a JSON k8s template, but honestly, that would be more work and more learning than we had to for what we're doing. Why would we do more just because we want to avoid templating in a YAML file?

CryoLogicover 6 years ago

I think they are missing the real selling point of JSON. It's basically interoperable with JavaScript objects.That means you write it, send it, store it, operate on it, etc. with little or no modification.The author says "converting between the two is trivial" which may be true, but the developer overhead is less trivial. And it will always be JSON in the client - JS doesn't support YAML objects.

评论 #19109017 未加载

评论 #19108979 未加载

评论 #19109001 未加载

评论 #19111620 未加载

the8472over 6 years ago

Helm reminds me of the 90s level of webtech. I.e. php cgi files mixing html, logic and includes.

评论 #19110189 未加载

fasteoover 6 years ago

Off-topic: The whole discussion is about deploy-time configuration management, but our problem is more about run-time configuration management.We have done the classical memcached+database custom solution, but I was wondering if there is any accepted library/tool to change application run-time behavior. We have tried consul KV store [1], but does not quite fit in our environment.My ideal solution would be a webapp with some text editor (think codemirror). Changes in this text file would push the configuration data to a running application.[1] <a href="https://learn.hashicorp.com/consul/getting-started/kv" rel="nofollow">https://learn.hashicorp.com/consul/getting-started/kv</a>

nickjjover 6 years ago

Who knows.It makes me throw up a little in my mouth every time I see hundreds of lines of YAML to configure something like Traefik with Kubernetes. The worst is when people say they prefer that because "I don't have to write a config file for my backend". That's true but instead now you have extremely verbose configuration mixed in with other verbose configuration.But in YAML's defense I think it's more of a problem with the tools that use it more so than YAML itself. Ansible is a great example of how amazing YAML can be to manage complex configuration in a concise way.

评论 #19110503 未加载

peterwwillisover 6 years ago

YAML is a data stream, not a program. Please do not shove programs into data.Your data does not need to be "expressive", it just needs to provide input to a program. If your data files need to be complex, you need a program to generate them for you.I've danced the dance of ini -> json -> yaml -> weird hybrid -> embedded logic, and it ends with "program that asks for what the thing you want looks like and generates data files". Industrial software design figured this out ages ago.

评论 #19112240 未加载

descover 6 years ago

My approach to this these days is:* all configuration, without exception, is XML.* all configuration may be generated from any other format imaginable, but it's sure as fuck going into the Big Main Godlike Application as XML.Separation, interfaces, etc. Disclaimer: I work in .NET almost exclusively. The .NET configuration APIs generally work, as long as you only ever use them for reading; treating config as something the application itself can fiddle with is a fast route to madness.

louiskottmannover 6 years ago

I feel like Ansible generating YAML with Jinja templates is really a sweet-spot, with idempotency and reusability.I find it pushes me to write plain YAML files for variables and defaults (Ansible), while allowing strong templating of generated files (Jinja) and letting the result be readable (YAML). By readable I mean minimum programming bloat (code spread on many lines just to write a for loop) and minimum extra syntax that clutters the screen (brackets and quotes). It also lets me write very little custom code (aside from variables, obviously).If I had to use a more "powerful" YAML-like replacement, it would mix all these into files, written differently by people with different styles, and it would have bloat all over the place.The main issue I have with helm is that values.yml is not templatable by default so you have to generate it if you want reusability.YAML does one thing and does it well, it's readable and bloat-free. Maybe we need more tools like "kubectl explain" to know the syntax though.

mschaefover 6 years ago

Reading the commentary on whitespace, my thoughts immediately jumped to the C preprocessor. Even though it's built into the language, it has the same sorts of problems as a template engine has with generating YAML: the preprocessor just wasn't enough aware of the syntactic structure of the language to make it easy to generate anything of significant complexity.I'm not proud of this (and like to think I could come up with something better these days), but this code was a bit of a nightmare for that reason:<a href="https://github.com/mschaef/vcsh/blob/master/vm/number.c#L256" rel="nofollow">https://github.com/mschaef/vcsh/blob/master/vm/number.c#L256</a>Lisp macros do better, but they have the problem that the macros (and their potentially unusual evaluation rules) can easily just blend in with ordinary function calls.

sankyoover 6 years ago

In the Clojure world we use EDN (extensible data notation) which is a subset of Clojure. <a href="https://github.com/edn-format/edn" rel="nofollow">https://github.com/edn-format/edn</a>You can extend it, convert it to JSON if necessary, and it is easy to read.

fredstedover 6 years ago

Helm charts are great, and accomplishes the task, but things get really weird and complex if you're not careful. Oh man, the indentation and _helper.tpl nightmares. It's not obvious at all. And there's still a lot of repetition for each chart; writing a deployment or whatever is so verbose in K8s.On the other hand, Ansible uses yaml and there it works great. I feel like Ansible uses yaml in a way that's easier to understand and the way it was meant to be written. With Ansible, you're writing configuration, not templates of configuration. I don't think a layer on top of Ansible like Helm is for K8s wouldn't make sense.

ssspajuover 6 years ago

I'd rephrase the question: "Why are we templating YAML with text-based templating tools?"I and @akx wrote a templating tool called Emrichen that's specifically designed for producing YAML and JSON from YAML templates:<a href="https://github.com/con2/emrichen" rel="nofollow">https://github.com/con2/emrichen</a>In contrast to other template systems, Emrichen templates are not just "based on YAML", they _are_ YAML. YAML tags like "!Var varname" are used to perform things like variable substitution, loops etc. Variables can be of any JSON type, not just strings, and the template is evaluated top–down.

alxarchover 6 years ago

Over the last week I created a tool processing YAML and JSON files using Jsonnet. It's called [ycat](<a href="https://github.com/alxarch/ycat" rel="nofollow">https://github.com/alxarch/ycat</a>) and is inspired by `jq` but uses Jsonnet for processing. It can also be used just as `cat` to concatenate JSON/YAML files. It's still young but very usefull, especially for handling complex kubernetes configurations.

zapharover 6 years ago

Wholly agree with this which is why I've been experimenting with a project similar in goals to jsonnet as a side project of my own. I didn't know about jsonnet when I started or I'd probably have just used/contributed to that instead.Why template yaml when you could just generate it? Or generate json, or toml, or xml, or environment variables...

评论 #19110965 未加载

nikolayover 6 years ago

Jsonnet is nice, but not actively developed, unfortunately. I highly recommend you to support UCG [0], which is written in Rust (Jsonnet has two implementations, which is one source of the slowness).[0]: <a href="https://github.com/zaphar/ucg" rel="nofollow">https://github.com/zaphar/ucg</a>

illumin8over 6 years ago

His main argument is that you need to template differently for different environments (dev/stage/prod) and cloud regions (us-west/us-east/emea/apac).This is actually a solved problem, and you shouldn't be doing it in your YAML/JSON templates. You should be using an external parameter store to do this, and using a single template for everything.See <a href="https://aws.amazon.com/blogs/compute/query-for-the-latest-amazon-linux-ami-ids-using-aws-systems-manager-parameter-store/" rel="nofollow">https://aws.amazon.com/blogs/compute/query-for-the-latest-am...</a>This is a simple use case: I want to deploy the latest AMI (Amazon Machine Image) in any region, so I always get the latest patched Linux base image to run my application on. I don't (and shouldn't) want to update my YAML/JSON every time a new image is published.So, why are people having to go to these crazy templating macro lengths? Just store the changing bits in an external config/parameter store like etcd and let your infrastructure as code templates remain unchanged.

评论 #19111516 未加载

评论 #19110220 未加载

评论 #19109897 未加载

l0b0over 6 years ago

YAML is the bastard offspring of XML. A bunch of ways to write semantically identical stuff is bullshit. Let data files be data files, and build them using languages actually suited for the job. JSON is plenty complex for 105% of the use cases of YAML, with much fewer downsides.

jeremysalwenover 6 years ago

When I joined Google, I realized that it was kinda surprising I hadn't seen something like GCL in the outside world (not to say it doesn't exist, just it's ubiquitous enough for me to hear about it). This seems to be an example of that hole being filled.

kyberiasover 6 years ago

Why is it so hard to write the meat of the argument in the first paragraph? So much setup...

shussonover 6 years ago

I've recently settled on typescript object literals for creating complex config files.Adv:- It's still quite declarative.- supports expressions.- supports merging objects using the spread `...` operator. This enables breaking up large configs files into smaller files.- supports type constraints.- serialises to JSON easily.

评论 #19113071 未加载

zoobabover 6 years ago

I use j2cli to template my YAMLs, pretty powerful to template stuff without using ansible. You just need to right environment variables. And you can put if logic if you want.

gjvcover 6 years ago

We cannot now be far off some ansible [1]-like tool using LISP s-expressions for its configuration language.[1] or any of that ilk

spacesuitman2over 6 years ago

A consistent observation has been that all config languages tend to become turing complete over time.

Animatsover 6 years ago

Next, inheritance for JSON! Look how well it worked for CSS.