Back when I was doing Python deployments (~2009-2013) I was:<p>* Downloading any new dependencies to a cached folder on the server (this was before wheels had really taken off)
* Running pip install -r requirements.txt from that cached folder into a new virtual environment for that deployment (`/opt/company/app-name/YYYY-MM-DD-HH-MM-SS`)
* Switching a symlink (`/some/path/app-name`) to point at the latest virtual env.
* Running a graceful restart of Apache.<p>Fast, zero downtime deployments, multiple times a day, and if anything failed, the build simply didn't go out and I'd try again after fixing the issue. Rollbacks were also very easy (just switch the symlink back and restart Apache again).<p>These days the things I'd definitely change would be:<p>* Use a local PyPi rather than a per-server cache
* Use wheels wherever possible to avoid re-compilation on the servers.<p>Things I would consider:<p>* Packaging (deb / fat-package / docker) to avoid having any extra work done over per-machine + easy promotions from one environment to the next.
Their reason for dismissing Docker are rather shallow, considering that it's pretty much the perfect solution to this problem.<p>Their first reason (not wanting to upgrade a kernel) is terrible considering that they'll eventually be upgrading it anyways.<p>Their second is slightly better, but it's really not that hard. There are plenty of hosted services for storing Docker images, not to mention that "there's a Dockerfile for that."<p>Their final reason (not wanting to learn and convert to a new infrastructure paradigm) is the most legitimate, but ultimately misguided. Moving to Docker doesn't have to be an all-or-nothing affair. You don't have to do random shuffling of containers and automated shipping of new images—there are certainly benefits of going wholesale Docker, but it's by no means required. At the simplest level, you can just treat the Docker contain as an app and run it as you normally would, with all your normal systems. (ie. replace "python example.py" with "docker run example")
We do something similar at embedly, except instead of dh-virtualenv we have our own homegrown solution. I wish I new about dh-virtualenv before we created it.<p>Basically, what it comes down to a build script that builds a deb with the virtualenv of your project versioned properly(build number, git tag), along with any other files that need to be installed (think init scripts and some about file describing the build). It also should do things like create users for daemons. We also use it to enforce consistent package structure.<p>We use devpi to host our python libraries (as opposed to applications), reprepro to host our deb packages, standard python tools to build the virtualenv and fpm to package it all up into a deb.<p>All in all, the bash build script is 177 LoC and is driven by a standard build script we include in every applications repository defining variables, and optionally overriding build steps (if you've used portage...).<p>The most important thing is that you have a standard way to create python libraries and application to reduce friction on starting new projects and getting them into production quickly.
We fixed that issue at Datadog by using Chef Omnibus:<p><a href="https://www.datadoghq.com/blog/new-datadog-agent-omnibus-ticket-dependency-hell/" rel="nofollow">https://www.datadoghq.com/blog/new-datadog-agent-omnibus-tic...</a><p>It's more complicated than the proposed solution by nylas but ultimately it gives you full control of the whole environment and ensure that you won't hit ANY dependency issue when shipping your code to weird systems.
<a href="http://pythonwheels.com/" rel="nofollow">http://pythonwheels.com/</a> solves the problem of building c extensions on installation.
Yes, someone should build the one way to ship your app. No reason for everybody to be inventing this stuff over and over again.<p>Deploys are harder if you have a large codebase to ship. rSync works really well in those cases. It requires a bit of extra infrastructure, but is super fast.
The fact that we had a weird combination of python and libraries took us towards Docker.
And we have never looked back.<p>For someone trying out building python deployment packages using deb, rpm, etc. I really recommend Docker.
We use a devpi server, and just push the new package version, including wheels built for our server environment, for distribution.<p>On the app end we just build a new virtualenv, and launch. If something fails, we switch back to the old virtualenv. This is managed by a simple fabric script.
We just commit our dependencies into our project repository in wheel format and install into a virtual env on prod from that directory eliminating PyPi. Though I don't know many other that do this. Do you?<p>Bitbucket and GitHub are reliable enough for how often we deploy that we aren't all that worried about downtime from those services. We could also pull from a dev's machine should the situation be that dire.<p>We have looked into Docker but that tool has a lot more growing before "I" would feel comfortable putting it into production. I would rather ship a packaged VM than Docker at this point, there are to many gotchas that we don't have time to figure out.
> curl “<a href="https://artifacts.nylas.net/sync-engine-3k48dls.deb”" rel="nofollow">https://artifacts.nylas.net/sync-engine-3k48dls.deb”</a> -o $temp ; dpkg -i $temp<p>It's really not hard to deploy a package repository. Either a "proper" one with a tool like `reprepro`, or a stripped one which is basically just .deb files in one directory. There's really no need for curl+dpkg. And a proper repository gives you dependency handling for free.
Note that the base path /usr/share/python (that dh-virtualenv ships with) is a bad choice; see <a href="https://github.com/spotify/dh-virtualenv/issues/82" rel="nofollow">https://github.com/spotify/dh-virtualenv/issues/82</a> for a discussion.<p>You can set a different base path in debian/rules with export DH_VIRTUALENV_INSTALL_ROOT=/your/path/here
"Distributing Docker images within a private network also requires a separate service which we would need to configure, test, and maintain." What does this mean? Setting up a private docker registry is trivial at best and having it deploy on remote servers via chef, puppet; hell even fabric should do the job.
No No No No! Or maybe?<p>Do people really do that? Git pull their own projects into the production servers? I spent a lot of time to put all my code in versioned wheels when I deploy, even if I'm the only coder and the only user. Application and development are and should be two different worlds.
I recently created vdist (<a href="https://vdist.readthedocs.org/en/latest/" rel="nofollow">https://vdist.readthedocs.org/en/latest/</a> - <a href="https://github.com/objectified/vdist" rel="nofollow">https://github.com/objectified/vdist</a>) for doing similar things - the exception being is that it uses Docker to actually build the OS package on. vdist uses FPM under the hood, and (currently) lets you build both deb and rpm packages. It also packs up a complete virtualenv, and installs the build time OS dependencies on the Docker machine where it builds on when needed. The runtime dependencies are made into dependencies of the resulting package.
I've had decent results using a combination of bamboo, maven, conda, and pip. Granted, most of our ecosystem is Java. Tagging a python package along as a maven artifact probably isn't the most natural thing to do otherwise.
Unfortunately, this method seems like it would only work for libraries, or things that can easily be packaged as libraries. It wouldn't work that well for a web application, for example, especially since the typical Django application usually involves multiple services, different settings per machine, etc.
Here is the process I use for smallish services -<p>1. Create a python package using setup.py
2. Upload the resulting .tar.gz file to a central location
3. Download to prod nodes and run
pip3 install <packagename>.tar.gz<p>Rolling back is pretty simple - pip3 uninstall the current version and re-install the old version.<p>Any gotchas with this process?
For installing using .deb files, how are db migrations handled. Our deployment system handles running django migrations by deploying to a new folder/virtualenv, running the migrations, then switching over symlinks.<p>I vaguely remember .deb files having install scripts, is that what one would use?
Weirdly I am re-starting an old project doing this venv/ dpkg (<a href="http://pyholodeck.mikadosoftware.com" rel="nofollow">http://pyholodeck.mikadosoftware.com</a>). The fact that it's still a painful problem means Inam not wasting my time :-)
> Building with dh-virtualenv simply creates a debian package that includes a virtualenv, along with any dependencies listed in the requirements.txt file.<p>So how is this solving the first issue? If PyPI or the Git server is down, this is exactly like the git & pip option.
Great article. I had never heard of dh-virtualenv but will be looking into it.<p>How has your experience with Ansible been so far? I have dabbled with it but haven't taken the plunge yet. Curious how it has been working out for you all.
Seems this method wouldn't work as well if you have external clients you deploy for. I'd use Docker instead of doing this, just to be in a better position for an internal or external client deployment.
Here's how I deploy python code:<p><pre><code> cf push some-python-app
</code></pre>
So far it's worked pretty well.<p>Works for Ruby, Java, Node, PHP and Go as well.
> The state of the art seems to be ”run git pull and pray”<p>No, the state of the art where I'm handling deployment is "run 'git push' to a test repo where a post-update hook runs a series of tests and if those tests pass it pushes to the production repo where a similar hook does any required additional operation".