Stack Overflow: How We Do Deployment

How people manage git and source control tells you a lot of things about a company&#x27;s culture. They said that most commits go directly onto master and this works for them, which indicates:<p>- Good rapid communication about who is working where. People are generally not touching the same code or else you&#x27;d run into frequent collisions (solvable via rebasing of course but they would be doing more branching if it were a thing to happen very frequently I&#x27;d suspect)<p>- The developers are given autonomy and have assumed some level of mastery over whatever their domain is. Trust in each developer&#x27;s ability to commit well-formed and considered code.<p>- They have a comprehensive test stack, which helps verify the above point and keep it sane

I was always wondering if there is some way of database deployment which does not suck. And all I see - every professional team ends with bunch of 074-add-column-to-table.sql files. I mean, code deployment can be organized much better. You can make graceful restarts, transactional updates, etc. Actually, almost nobody backups old code version because deployment of new one may fail, but database upgrades are so fragile and making backups is a must not only because you afraid of upgrade process may be interrupted and leave database in inconsistent state, but because properly done upgrade may result in corrupted state.

Additional anecdata: At my place of employment, after the required code review, we must write an essay about the change and have at least two coworkers approve it. Then we must email the essay to a mailing list of several hundred people. One of those people is the designated external reviewer, who must sign off. However, lately, this final reviewer has been requesting 10+ modifications to the deployment essay due to a managerial decision to &quot;improve deployments&quot;. Moreover, deployments are not allowed 1&#x2F;4th of the year unless a vice president has signed off.<p>Any code change requires this process.

This feels really clunky to me, but maybe I&#x27;m just not getting it. I&#x27;m trying to implement a more automated build&#x2F;deploy process at my current place of employment and am basically modeling it off of Github&#x27;s [0], which seems to have a better feel.<p>Obviously the quality of the process needs to be high, but when it&#x27;s effortless and &quot;fun&quot; then everybody wins.<p>[0] <a href="http:&#x2F;&#x2F;githubengineering.com&#x2F;deploying-branches-to-github-com&#x2F;" rel="nofollow">http:&#x2F;&#x2F;githubengineering.com&#x2F;deploying-branches-to-github-co...</a>

<p><pre><code> Fun fact: since Linux has no built-in DNS caching, most of the DNS queries are looking for…itself. Oh wait, that’s not a fun fact — it’s actually a pain in the ass. </code></pre> Surely that should just be a very fast lookup in &#x2F;etc&#x2F;hosts?

I think if you&#x27;re deploying .net code you&#x27;re almost certainly going to follow similar build architecture with TeamCity doing most of the grunt work. We have a very similar build structure but a bit more polished I think. Our TeamCity build agents build nuget packages, deploy them to Octopus, run unit and integration tests. Octopus handles promotions of code from dev to qa, to staging and all production environments. We also write migrations for database updates using FluentMigrator which works with just about any database driver. It&#x27;s a joy deploying code on an environment like this.

&gt; <i>Fun fact: since Linux has no built-in DNS caching, most of the DNS queries are looking for … itself.</i><p>This is wrong in two ways, and isn&#x27;t factual at all.<p>First, the cause of the queries is nothing to do with whether DNS query answers are cached locally or not. There is no causal link here. What causes such queries is applications that repeatedly look up the same things, over and over again; not the DNS server arrangements. One could argue that this is poor design in the applications, and that they should remember the results of lookups. But there&#x27;s a good counterargument to make that this is <i>good</i> design. Applications <i>shouldn&#x27;t</i> all maintain their own private idiosyncratic DNS lookup result caches. History teaches us that applications attempting to cache their own DNS lookup results invariably do it poorly, for various reasons. (See Mozilla bug #162871, for one of several examples.) Good design is to hand that over to a common external subsystem shared by all applications.<p>Which brings us to the second way in which this is wrong. A common Unix tradition is for all machines to have local proxy DNS servers. Linux operating systems have <i>plenty</i> of such server softwares that can be used: dnsmasq, pdnsd, PowerDNS, unbound ...<p>One of the simplest, which does <i>only</i> caching and doesn&#x27;t attempt to wear other hats simultaneously, is even named &quot;dnscache&quot;. Set this up listening on the loopback interface, point its back-end at the relevant external servers, point the DNS client library at it, and -- voilà! -- a local caching proxy DNS server.<p>* <a href="http:&#x2F;&#x2F;cr.yp.to&#x2F;djbdns.html" rel="nofollow">http:&#x2F;&#x2F;cr.yp.to&#x2F;djbdns.html</a><p>* <a href="http:&#x2F;&#x2F;cr.yp.to&#x2F;djbdns&#x2F;run-cache.html" rel="nofollow">http:&#x2F;&#x2F;cr.yp.to&#x2F;djbdns&#x2F;run-cache.html</a><p>* <a href="http:&#x2F;&#x2F;homepage.ntlworld.com&#x2F;jonathan.deboynepollard&#x2F;FGA&#x2F;djbdns-problems.html#wrong-icann-root" rel="nofollow">http:&#x2F;&#x2F;homepage.ntlworld.com&#x2F;jonathan.deboynepollard&#x2F;FGA&#x2F;djb...</a><p>* <a href="http:&#x2F;&#x2F;cr.yp.to&#x2F;djbdns&#x2F;install.html" rel="nofollow">http:&#x2F;&#x2F;cr.yp.to&#x2F;djbdns&#x2F;install.html</a><p>I run server machines that have applications that repeatedly look up the same domain names again and again. Each runs a local dnscache instance, which ameliorates this very well.

It&#x27;s funny to see that stackoverflow came to exactly the same solution for database migrations on the Microsoft stack as my team did, even down to the test procedure.<p>Simple, safe and very effective :)

&gt; If Gitlab pricing keeps getting crazier (note: I said “crazier”, not “more expensive”), we’ll certainly re-evaluate GitHub Enterprise again.<p>Shots fired :P

When did everybody decide that chatbots were the new hotness for deployments?

<p><pre><code> A developer is new, and early on we want code reviews A developer is working on a big (or risky) feature, and wants a one-off code review </code></pre> This implies you don&#x27;t normally do code review??

SE&#x2F;SO is for it&#x27;s size an amazingly high proformance team. It&#x27;s rare to hear them say it, in fact, I don&#x27;t recall ever hearing them say this.<p>The importantance of the cohesion and trust amoung their team is critical to their deployments. In fact, I would say it&#x27;s vital to how they&#x27;re able to get away with minimal amounts of code reviews for example.<p>It&#x27;s dangerous to believe this is easy or reproducible. New teams needs extensive controls in place to make sure the quality of the deployments will not negatively impact the group.

A few things strike me as odd&#x2F;sub-optimal:<p><pre><code> - migration id&#x27;s freeform in chat -&gt; why not central db with an auto-increment column? - Use chat to prevent &#x27;migration collisions&#x27; -&gt; Same central db&#x2F;msmq&#x2F;whatever to report start&#x2F;stop of migrations and lock based on that...</code></pre>

The article says:<p>&gt; there is a slight chance that someone will get new static content with an old hash (if they hit a CDN miss for a piece of content that actually changed this build)<p>Anyone has a solution to this problem?

Stack Overflow uses windows? Any particular reason to do this?

I was surprised they weren&#x27;t using Kiln [<a href="https:&#x2F;&#x2F;www.fogcreek.com&#x2F;kiln&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.fogcreek.com&#x2F;kiln&#x2F;</a>], a Fog Creek product. I know SO is independent from Fog Creek now, but still a bit surprised at it. I wonder if there was a migration off at some point.

Did you choose polling over webhooks for a reason? Or was webhooks recently added as a feature to Gitlab?

TeamCity is a great CI system.

What&#x27;s with all the backslashes?