<i>Luckily, there is an alternative: we can move all of the configuration of Jenkins into source control. This means that when we deploy Jenkins it is deployed fully configured, so we can treat the instances as cattle instead of pets.</i><p>I would take that two steps further:<p>1. make an OS package of the configuration, and keep the OS package files plus the configuration under revision control (Bitkeeper, git, Mercurial, ...);<p>2. ditch Docker, because at that point, it's superfluous (the OS' software management subsystem worries about the payload and deployments at that point, for example with Kickstart in RHEL / CentOS, or AutoYAST in SLES, or...) - no code required for installation.<p><i>We might lose the job build histories when we redeploy (only if we wipe the old Jenkins and redeploy a fresh, but treating Jenkins as cattle implies that this can happen at any time). If keeping the build histories intact is very important to you then you’ll have to solve this yourself or keep treating Jenkins as a pet.</i><p>That's easy to solve (I already solved it): if Jenkins the application, and Jenkins' configuration is packaged, then the same applies for backup software (Bacula), and backup software's client configuration.<p>Using the AT&T / SVR4 filesystem hierarchy standard specification, Jenkins can be <i>engineered</i> to deliver:<p>- payload in /opt/;<p>- configuration in /etc/opt/;<p>- <i>data</i> in /var/opt/jenkins/.<p>Since the client will be automatically configured via the OS package to backup only /var/opt (no point in backing up packaged application and configuration, since they're under revision control and can be reinstalled at will to replay the machine state), only <i>data</i> can be restored and the instance can continue where it left off, <i>without any loss of state</i>.<p>I already do this at scale. No Docker needed at all, and servers, be they virtual or physical, are completely disposable.
Great post. I've seen first hand the problems that occur with all the uncontrolled 'state' in Jenkins installations when something goes wrong... lots of manual work!<p>It's like moving Jenkins towards a static website model where you can always get back to a known state quickly and easily.<p>At hyper.sh we've been working to take this a step further by providing fast starting (couple of seconds), secure and highly-parallelized on-demand CI worker nodes.<p>We started with Buildbot (released yesterday: <a href="https://blog.hyper.sh/serverless-ci-hyper-docker-integration-for-buildbot.html" rel="nofollow">https://blog.hyper.sh/serverless-ci-hyper-docker-integration...</a>) and the Jenkins plugin is coming up next.<p>That makes a dream situation whereby you have a Jenkins master that can be easily restored from version control and all workers are created on-demand and billed per second meaning that there is no worker node maintenance at all and you only pay when you're building.<p>We're calling it 'Serverless CI'. Would love to talk to you about this further. You can find contact details on the website if interested!