We run clusters of machines and whenever there's an update via USN / DSA or whatever I end up manually patching each cluster with cluster-ssh.<p>This is less than ideal, but seems to work.<p>What do you do?<p>Note; I'm talking about binary packages distributed by your OS : apt upgrades / rpms.. not config files (Hi, puppet/chef), or deprec for capistrano style stuff.
I used to manage around 300 servers myself. The only way it was possible was to have a completely stripped OS. All apps we used I installed under:<p>/apps/<appname>/<app version>
example: /apps/perl/5.8.12
then I would symlink /apps/perl/5.8.12 to /apps/perl/current<p>The profiles on the machine would add /apps/*/current/bin to the path.
This allowed upgrades and roll backs just by changing the symlink to the one I wanted to be current. This also allowed me to push out versions of software ahead of time, and they just change the link when we were ready to use it.<p>Each machine would rsync /apps from a master distro nightly and of course I could force it with a for i in `cat hosts.list`...
Create a custom repository of your packages and point all the machines in your cluster to it? If it is CentOS or RHEL just add a new repo to /etc/yum.repos.d. If you are upgrading system repos then just set the priority of your repo to be higher. Of course this implies that you have successfully rpm-fied your packages. We did that to all our packages and configuration.
I was at devopsdays in Mountain View CA a week ago and one of the panel discussions was on package management. I was a bit surprised to see it as a topic, as there are numerous known ways to solve the problem.<p>It turns out, that's exactly the problem. The topic was hotly debated and yes, there are endless possible ways to distribute packages across a large environment.<p>I think choosing a pattern involves deciding for yourself how secure/auditable you need your environment to be and how tightly you want to couple your deployment process to your current architecture (ie. some package managers only work on some systems, etc.). That will narrow your choices down to a handful, and then you get to dig into the implementation details and decide from there.
We manage our own dists. We package our own software as .debs, so everything gets managed the same way. All security updates, release deployments, rollbacks, etc., are managed with apt-get. I don't exactly understand pinning, but it's also important to how we manage packages.<p>Depending on the release and the kind of server we're deploying to, we may do them all in one night or in batches of a few hundred over a week. All our boxes install security updates regularly (because we promptly add security updates to our dists.)
I run a squid caching proxy and direct all my (Fedora or RHEL) systems to use it along with a non-mirror list repository that is fairly close to me. I'm not totally sure what your question is though.<p>If you want to tightly control the packages that get updated then there isn't much to be done other than manually managing your own repo. Although if it is this big of a concern you should probably be running a distribution that is less volatile than Ubuntu. Say RHEL, CentOS, or Debian.
You can do two things.<p>1. Mirror a yum repository plus updates. For each day create a hardlinked directory of updates named reponame.YYYYMMDD. On the hosts you wish to update sed -i to the new day in your yum repo config file and run yum -y upgrade. When you know it works, do it on the other hosts too.<p>2. Use a configuration management tool like Puppet.<p>For your sized infrastructure you probably want the first.
I've heard good things about MCollective for distributed package updates:<p><a href="http://www.puppetlabs.com/mcollective" rel="nofollow">http://www.puppetlabs.com/mcollective</a>