I am curious how people are doing their staging and production deployments on AWS. Do you bake everything into the AMI and do nothing at boot? Do you boot a vanilla AMI and do all configuration during boot? Something in the middle? If you don't fully bake, is it because it is too hard to manage?
I make complete AMIs with packer, configure them entirely using environment variables in userdata, configuration data in etcd, and shell scripts, and run all services in docker containers, which I also build using packer. With all services in containers, AMIs are almost never rebuilt and there is no need for configuration management/mutating infrastructure.<p>Building containers with packer is easier than switching to Dockerfiles for existing builds, but does not support fast, incremental build and deploy or tagging. Even without those features, I see no advantages in traditional CM other than the convenience of familiarity and legacy.
I create an AMI with a bare minimum OS. Then I use a configuration management tool to install all software packages, libraries and configurations. My new favorite is Ansible (ansibleworks.com) but Chef and Puppet are others.<p>Updates are easier this way versus having to rebake images.
I use Ansible for all configuration management. Boxes that belong to ASGs use Ansible to create a pre-baked AMI, while the rest are just handled with Ansible on a case-by-case basis.