Hey all, I'm the creator/primary maintainer of pyinfra! Super excited (a little terrified) to see this on the frontpage, happy to answer any questions :)<p>I also hang out on the Matrix room: <a href="https://matrix.to/#/#pyinfra:matrix.org" rel="nofollow">https://matrix.to/#/#pyinfra:matrix.org</a><p>Another thing: the GH repo points at currently in beta v3 and the docs for this are here: <a href="https://docs.pyinfra.com/en/next" rel="nofollow">https://docs.pyinfra.com/en/next</a> (highly recommend starting with v3, I just haven't had any time recently to wrap up the release, but it's stable).
I current use Ansible to setup both local and remote hosts. I've been very happy with it, and love that Pyinfra intends to support the Ansible connector.<p>My main gripe with Ansible is the YAML specification. Ansible chooses to separate the task specification and task execution.
Pyinfra chooses to directly expose the Python layer, instead of using slightly ugly magic functions/variables. I like this approach more since it allows standard Pythonic control flow instead of using a new (arguably ugly and more hassle to maintain) grammar.<p>Excited for Pyinfra!
I just started using Pyinfra to wrangle a bunch of servers and it is a breath of fresh air compared to Ansible. I moved all of my server OS installs to Fedora CoreOS which doesn't ship with Python in the OS and since Pyinfra doesn't need Python on the host node I can kick off tasks in bulk to do server things. It is great. I cannot wait to see where the Pyinfra project goes.<p>On a side note, one of the most hacky things I came up with to get Ansible working on Fedora CoreOS was to bind mount a container rootfs that had python 3 and then symlink it into the right spots. You can of course add Python in with rpm-ostree if you want but I wanted to avoid layering packages at the time. I wasn't proud of it. But it worked.<p><a href="https://github.com/forem/selfhost/blob/main/playbooks/templates/forem.yml.j2#L67-L118">https://github.com/forem/selfhost/blob/main/playbooks/templa...</a>
I think Puppet hits the sweet spot in this area. It's default is a series of idempotent "here's how this should be configured" statements, but it can be used as a full programming language in its more advanced capacity, and it's reasonably extensible (in Puppet-lang and Ruby) to support specific custom applications.<p>I also think that the facts/manifest/apply separation is conducive to nicely testable infra code, and useful dry-run output.<p>I'm always surprised that Puppet isn't still more popular. My theory is that it's passed over because of its age/cruftiness/bad vibes in some cases, and that a couple of technical flaws mess it up for some key userbases:<p>For folks who just want a quick-to-start management tool for a small set of config, Puppet's ugly and clunky client/server model and the hyper-YAML-ification of its best practices (which is pursued to a fault by the community, and not helped by the Hiera pitch that the Puppet stack can also be sort of an asset tracking/catalog system) make small-scale usage and prototyping hard. Puppet doesn't <i>have</i> to be used that way (it can be used just like pyinfra/Ansible with a local-apply or via Bolt, hitting a nice sweet spot between ad-hoc/non-idempotent commands and nice declarative/idempotent Puppet code), but I think the puppetmaster/hiera-all-the-things legacy in the community does Puppet and potential new users a disservice.<p>From the other side, I think a lot of more cloud-oriented users looking for a "better Terraform for server state" end up annoyed by the quality of modules on the Puppet forge and Puppet's lack of a statefile equivalent (meaning that it doesn't support deletes or infrastructure state snapshots in the same way TF does).
This will tie nicely into my favorite way to deploy services these days:<p>1. Use PyInfra to set up Docker and Tailscale on remote hosts and any other setup. Open the Docker port to your Tailnet.<p>2. Use the Docker provider for Terraform to set up and manage containers on those hosts from your dev machine or from a CI/CD tool. Tailscale allows containers on different machines to communicate privately, or you can open a port to the web.<p>This makes for such an easy-to-use and bulletproof setup. In the past I would have used Kubernetes but I've come to realize that's overkill for anything I do and way harder to debug.
Agree with those saying the landing page needs work. But terraform/docker integration sounds interesting.. after many years of ansible you’d think there is a more comfortable way to replace a hundred lines of hacky bash in dockerfiles.<p>Also, can I just say that cm is extremely frustrating? Not sure this is the fix, but hopefully the story isn’t over. In my experience the maintenance of cm codebases never, ever stops. At first I thought it was a matter of expertise, but experts typically agree and just call it the cost of doing business.<p>Shelve something for three months and it will break on the next run, on the same os/host where it used to work. Blame the package manager, blame the os choice, or the cm tool. But it’s embarrassing and insulting for Devops teams after putting in the effort to do things right, and evangelizing to everyone else about repeatability. I’d rather just see tighter integrations with containers moving forward and never think about it again. Not everyone is using k8s but in the 2020s everyone probably should default to using docker before doing things of even marginal complexity directly on hosts.
Thanks for making Pyinfra.<p>It's one of the tool that get out of your way and let. you get the work done. The tool works for you instead of you fighting with the tool.<p>Pain point of ansible: storing state and checking later, coordinate state between server is all a breezy with Pyinfra because you write the Python code to perform those check.<p>The system is very well though out. No need to hack around host file, inventory is just a python script that export resource definition.<p>No more static, ad-hoc host var, you get a real python script to define and return your variable.<p>Using pyinfra I was able to focus more on the "compute". the state such as credential, inventory can managed and store outside such as in SSM or just call python ec2 api to filter instance by tag.
Ansible needs a working Python interpreter on the target machine.<p>Pyifra doesn't even need that. Just needs a shell.<p>Subjective opinion but it is heavily under recognized piece of software.<p>Ansible is really great but you soon end up writing Python in Yaml strings.<p>So why not straight up Python?
Should this be considered some kind of alternative to tools like Ansible?<p>Also CDKTF should be in the space for imperative infrastructure as code definitions.<p>- <a href="https://developer.hashicorp.com/terraform/cdktf" rel="nofollow">https://developer.hashicorp.com/terraform/cdktf</a>
This is great. We tried ansible and gave up as it was difficult to keep configuration DRY and annoying to create conditions with no control structure.<p>It was before ansible 2, so probably things are better now.<p>Then we started using Python fabric. Wow it was so freeing. Any helper methods were easily extracted and writing conditions felt natural.<p>Now I am using Python invoke to maintain my local setup.
I worry about using python for this kind of thing.<p>It's very hard to be confident about python code.<p>If you have a good code review feedback loop and so on then it can be OK but proper types enable lots of good things when dealing with configuration and state.
Was there any thought to perhaps do a version with an agent? I really like how fast Saltstack can be as compared to Ansible.<p>I've been using my own homegrown project that does just this - Python roles, server/client, Mako templates: <a href="https://github.com/mattbillenstein/salty">https://github.com/mattbillenstein/salty</a><p>It's very very fast to do deploys on long-lived infrastructure, but it hasn't been optimized for large clusters yet; I expect the server process will be a bottleneck with many clients, but still probably faster than Ansible for most setups.
Very cool. One question: Can Pyinfra create container-like objects and objects inside them? Example: create an RDS database, create a user inside that database, and assign the user a role.<p>Terraform cannot deploy such a configuration in a single config, since its planning stage requires that all containers already exist. Terraform crashes when planning the user and role changes, saying that the database doesn't exist. This is a large pain-point when using Terraform. How does Pyinfra handle such deployments?
Python seems like a really poor choice for infrastructure.<p>- Python is not easy to build into portable binaries<p>- The package ecosystem is very hard to use in a reproducible way<p>- The language is not truly typed - types add massive value for infrastructure and scripts because they are less likely to be unit-tested<p>- The lack of a "let" or "var" keyword makes simple programming errors more likely (again, this code is less likely to be unit-tested)<p>Maybe I'm missing something? I don't know why I would want to introduce Python in this domain.
Does it allow me to run a script against an EC2 instance, say, and it spins it up and take care of everything? Something like packer would but without creating an AMI
IMHO Ruby is better for creating DSLs, so I wrote a small thing to scratch my own itch: <a href="https://github.com/marius/koch">https://github.com/marius/koch</a><p>This is not meant to scale to more than a handful of machines, but you get the idea how nice straight Ruby is for a machine specification DSL.
Seems like an interesting generalized mix of something like <a href="https://github.com/cloudtools/troposphere">https://github.com/cloudtools/troposphere</a> and Ansible from a glance.<p>The value add would be unifying provisioning and configuration management in a Python-y experience? The lifecycle of each is distinct and that's traditionally where the headaches of using a single tool for both has come in
Is this something that would be a good fit to automate node reboots/restarts of complex clustered systems? Think Kafka, Elasticsearch or Flink, where you can’t restart the next node without revalidating the state of the cluster and the rejoining of the previous node. Please feel free to suggest other tools for this purpose.
I use pyinfra through molecule for testing sensible roles, it’s made it possible to have a process resembling TDD and have automated tests for my roles and playbooks. I actually don’t know how else to do it than with molecule and pyinfra, being able to have automated tests on ansible “code” made a big difference for me!
Yeah, I like this approach.<p>There's something about YAML that just sucks the joy out of programming. It seems like a giant step backwards when we have plenty of amazing programming languages in existence.<p>Even when infrastructure yaml like cloudformation are wrapped by some SDK, it can still be a pain because you end up with stuff like...<p><pre><code> do_something("___((!-prickly_config_string_::might as well use yaml _blah-blah:blah))")
</code></pre>
Back in the days of java and xml, there used to be a distant promise of "binding" the xml to code (remember jaxb?) so that you could then just manipulate it fluently as code and then "marshall" it out back to xml when you were done. Those days and that promise are gone, right?
this feels like Michael DeHaan's OpsMop project that existed for like a week before he pulled all the code offline.<p><a href="https://news.ycombinator.com/item?id=18717422">https://news.ycombinator.com/item?id=18717422</a><p>Interesting to see all the Ansible comments here. I'll have to check this out asap.
I would like to point to a virtual machine or a set of virtual machines that I have configured and make the tool reproduce / translate the state of these „model machines „ to some hosting environment.<p>Can this or any other tool do that?
Does anyone have any info on if saltstack is going to be enshittified? That is the situation that would get me to go looking for a replacement such as this
Maybe this is the best thing ever, but the documentation doesn't answer one simple question: What problem is it solving?<p>It is a configuration management tool, like Ansible?<p>Is it meant for running one-off commands across the infrastructure, like Salt?<p>It says it integrates with Terraform, so it's not a provisioning tool...<p>What does it do different (and presumably better) than other tools?<p>The Getting Started guide doesn't cover this.
The FAQ doesn't cover this, and the Docs doesn't have an Introductory section to cover this.<p>It's disheartening to find a potentially interesting project, but not really know what it does and how it might fit in your workflow.
We build a similar tool except we focus on AI workloads. Also support on-prem clusters now in addition to GPU clouds. <a href="https://github.com/dstackai/dstack">https://github.com/dstackai/dstack</a>
Should probably just stick with the Terraform CDK or Chef if you need this level of expressibility.<p>This is no where near the level of readiness needed to be reliably used in a production environment.<p>Verbose logging is not a reason to introduce a non-standard tool into your stack.
so, it's Ansible...?<p>Configuration Management tools (that's what this, and Ansible, are) are a nice idea, but get very complicated very quickly. The tools themselves get complicated, the configuration gets complicated, you're constantly finding ways that the state gets broken that you need to re-incorporate into your script, it has to work in a variety of states, and you have to keep re-running and re-running and re-running it, monitoring for problems, investigating, fixing. Very complex, lots of maintenance, lots of potential problems. The "Pets" model from the phrase "Cattle, not Pets." I <i>strongly</i> recommend you do not raise Pets.<p>Instead, use Immutable Infrastructure: build an immutable image one time that works one way. Deploy that image. If you need to change it, change the build script, build a new image (with a new version), deploy a new instance with the new image, take the old one out back and shoot it. (The "Cattle" of "Cattle, not Pets") If the state gets out of whack or there are problems, just shoot it and deploy a new one that you know works.<p>This is the single most revolutionary concept i've seen in over 20 years of doing this job. It is an absolute game-changer. I would not go back to Configuration Management for all the tea in China.
As someone who had to write infrastructure in Python, every time from scratch, for large projects: pyinfra isn't it (and neither is Ansible, if you care about that).<p>It will probably work for some simple and common cases, but they barely need any automation anyways...<p>The problem isn't even the tool itself, it's the lack of standards. Every large enough system is too unique to be easily managed by cookie-cutter tools like this one. Some people will bite the bullet anyways, and try to adapt general-purpose infra tools to their case. I've seen that too. This is a very miserable experience. Frustrating in that very obviously simple and necessary things are sometimes described as "impossible" due to how the chosen framework works. To contrast that, the home-brewed systems usually suffer from the lack of generality, worse user experience in general, quickly start lagging behind the underlying technology updates...<p>Also, out of popular languages, Python would be somewhere towards the bottom of the hierarchy if I had to choose a language to manage infrastructure. The only redeeming quality of Python is its popularity. On engineering merits alone its unremarkable at best.<p>----<p>PS.<p><pre><code> import click
</code></pre>
If I see this in the project source code, I blacklist it and never look at it again. This is a red flag, a sure sign that the person writing it are clueless.