The sad state of sysadmin in the age of containers

This bothers me as well. Even tasks as simple as adding a repository are now being &quot;improved&quot; with a curl | sudo bash style setup[1].<p>However, installing from source with make was (and remains) a mess. It may work if you&#x27;re dedicated to maintaining one application and (part of) its stack. But even then it usually leads to out of date software and tracking versions by hand.<p>Many people have this weird aversion to doing basic sysadmin stuff with Linux. What makes it weird is that it&#x27;s really simple. Often easier than figuring out another deploy system.<p>(The neckbeard in me blames the popularity of OSX on dev machines.)<p>[1] <a href="https:&#x2F;&#x2F;nodesource.com&#x2F;blog&#x2F;nodejs-v012-iojs-and-the-nodesource-linux-repositories" rel="nofollow">https:&#x2F;&#x2F;nodesource.com&#x2F;blog&#x2F;nodejs-v012-iojs-and-the-nodesou...</a>

I agree that many of these convenient setups are embarrassingly sloppy, but it&#x27;s the sysadmin&#x27;s responsibility to insist on production deployments being far more rigorous. No one can tell you how to build hadoop? Well, figure it out. Random Docker containers being downloaded? Use a local Docker repo with vetted containers and Dockerfiles only.<p>I don&#x27;t even allow vendor installers to run on my production systems. My employer buys some software that is distributed as binary installers. So I&#x27;ve written a script that will run that installer in a VM, and repackage the resulting files into something I&#x27;m comfortable working with to deploy to production.<p>If a sysadmin is unable to insist on good deployment practices, it&#x27;s a failure of the company or organization or of his own communication skills. If a sysadmin allows sloppy developer-created deployments and doesn&#x27;t make constant noise about it, then they aren&#x27;t doing their job properly.

make is the least-auditable build tool imaginable. You don&#x27;t have to obfuscate a Makefile, they come pre-obfuscated; you could put the &quot;own me&quot; commands right there in &quot;plain&quot; Make. Not to mention that it&#x27;s often <i>easier</i> to tell whether a Java .class file is doing anything nefarious than whether a .c file is. How many sysadmins read the entire source of everything they install anyway?<p>Maven, on the contrary, is the biggest single source of signed packages around. Every package in maven central has a GPG signature - the exact same gold standard that Debian follows. The problems Debian faces with packaging Hadoop are largely of their own making; Debian was happy to integrate Perl&#x2F;CPAN into apt, but somehow refuses to do the same with any other language.<p>&gt; Instead of writing clean, modular architecture, everything these days morphs into a huge mess of interlocked dependencies. Last I checked, the Hadoop classpath was already over 100 jars. I bet it is now 150<p>That&#x27;s exactly what clean modular architecture means. Small jars that do one thing well. They&#x27;re all signed.<p>Bigtop is indeed terrible for security, but its target audience is people who want a one-stop build solution - not the kind of people who want to build everything themselves and carefully audit it. If you are someone who cares about security, the hadoop jars are right there with pgp signatures in the maven central repository, and the source is there if you want to build it.

At my last job we used a micro-service architecture on AWS (EC2 and RDS). Using Ansible playbooks for various types of servers and roles for each service, we created a new server instance every deploy. All servers were running FreeBSD and using daemontools to control services. For testing, hotfixes, and manual checking of logs, it was easy to complement with manual ssh. Save old and new instance in case something goes wrong. Ansible is just a thin layer on top of shell scripts, and reasonably straightforward to understand and parameterize. Worked wonderful in most cases (possible exception of build server because of shared libraries and a complex workflow with git pull&#x2F;trigger, but I don&#x27;t think that was the fault of the overall architecture).<p>That said, I agree that sbt is an abomination and doesn&#x27;t lend itself to a sane and secure workflow, unfortunately.<p><a href="http:&#x2F;&#x2F;martinfowler.com&#x2F;bliki&#x2F;PhoenixServer.html" rel="nofollow">http:&#x2F;&#x2F;martinfowler.com&#x2F;bliki&#x2F;PhoenixServer.html</a><p><a href="http:&#x2F;&#x2F;www.ansible.com&#x2F;" rel="nofollow">http:&#x2F;&#x2F;www.ansible.com&#x2F;</a>

I love a good rant as much as the next guy, but unfortunately, rants are rarely actionable.<p>&gt; Maven, ivy and sbt are the go-to tools for having your system download unsigned binary data from the internet and run it on your computer.<p>The root of the problem is that out of the total number of libraries available in language X, only a small subset is packaged in Debian&#x2F;RHEL. This may be more egregious with large, Java enterprisy software, but you could easily end up with the same problem in Ruby or Python.<p>You cannot reasonably expect developers to package and maintain all their dependencies properly. The least worse solution would be to:<p>- still use maven to manage dependencies<p>- create a Debian&#x2F;RHEL package incorporating the dependencies (effectively vendoring them in the package)<p>Unfortunately, it is not that simple, because you need to make sure that your vendored-in-the-package dependencies are somewhere where they will not conflict with another package with the same idea and the same dependencies (or better, the same idea and a different version of the same dependencies). Which means you need to keep them out of &#x2F;usr&#x2F;share&#x2F;java and make sure the classpath points at the right location.<p>However, it seems that developer tend to avoid this kind of rigmarole and instead go for the &quot;install dependencies as a local user&quot; for certain classes of application (eg, webapps) because packaging is not fun.

&quot;Stack is the new term for &quot;I have no idea what I&#x27;m actually using&quot;.&quot; - this made my day!

I was doing sysadmin the &quot;right way&quot; a long, long time ago, and I don&#x27;t see much difference. Maybe the author regularly does full audits of the source code of every package he downloads, and of course disassembles every executable and library in the underlying OS, but most of us don&#x27;t. There&#x27;s no wisdom or security to be gained from the act of running &quot;make&quot;, much less &quot;make install&quot;.

The contract between operations and dev (as concepts, not as people) is in need of renewal.<p>To my mind, that was what &quot;devops&quot; was supposed to be, but it&#x27;s been a bit of a dogpile in the years since the term gained popularity.<p>Systems are opaque to most developers, and many developers wish to make their software opaque to the system on which it runs. This is a failure on behalf of our entire profession, not any one group.<p>Infrastructure software is in a bit of a renaissance period, but it&#x27;s very early days. Packaging software is a total mystery to most developers. I don&#x27;t even need to back that up with examples, most of us can recall the last time we can across a well packaged piece of software with joy due to sheer rarity. I&#x27;d be very surprised to find the average age of a Debian maintainer was trending anything but upwards, and steeply.<p>Containers are being misused, but that&#x27;s because the alternatives we&#x27;ve been building for ourselves have not kept up with the strong user experience narrative of web and mobile software.<p>We need to do better.

This 1 page poorly titled wrong rant is the #2 story on this site?<p>&quot;Ever tried to security update a container?&quot; lol. you are doing it wrong.<p>&quot;Essentially, the Docker approach boils down to downloading an unsigned binary, running it, and hoping it doesn&#x27;t contain any backdoor into your companies network.&quot; nope <a href="https:&#x2F;&#x2F;blog.docker.com&#x2F;2014&#x2F;10&#x2F;docker-1-3-signed-images-process-injection-security-options-mac-shared-directories&#x2F;" rel="nofollow">https:&#x2F;&#x2F;blog.docker.com&#x2F;2014&#x2F;10&#x2F;docker-1-3-signed-images-pro...</a><p>&quot;»Docker is the new &#x27;curl | sudo bash&#x27;«&quot; no it&#x27;s not. most intelligent companies are building their own images from scratch.<p>People that care about what&#x27;s in their stack take the time to understand what&#x27;s in there &amp; how to build things.

So much truth in this.<p>We&#x27;ve been doing some work with Elastic Beanstalk lately, and - while it certainly does one or two things that are extremely clever and useful - in the end it just feels like this bizarre mix of complete magic and incredibly convoluted arcana. Everything feels very out of our control and locks us into an ecosystem that considerably limits our choices and flexibility (unless we invest the time in becoming experts in EB, which really isn&#x27;t particularly something we have the time for). And, as the author of this post says, the security ramifications, while orthogonal, are also deeply troubling.

<p><pre><code> This rant is about containers, prebuilt VMs, and the incredible mess they cause because their concept lacks notions of &quot;trust&quot; and &quot;upgrades&quot;. </code></pre> Prebuild VMs? Sure, I wouldn&#x27;t touch them except for evaluating a project, and for commercial software you may not have a choice.<p>But docker containers at least usually provide a dockerfile that describes exactly how a binary image is built. You just clone the source repo, audit the few lines of build commands and then build your own private registry. It&#x27;s nearly no more trust than following the instructions of README or INSTALL. Just because fools are pulling down pre-build images and running in their datacentre, doesn&#x27;t mean that&#x27;s what way <i>you</i> should do it. And the problem with &#x27;old-school&#x27; sysadmins is they are often far too quick to reject new practices, citing tired excuses based on misunderstandings of the technologies.<p><pre><code> Ever tried to security update a container? </code></pre> Yeah I have. It&#x27;s easy if you have already built your &#x27;stack&#x27; to scale horizontally (which means you have at least 2 or more of everything in a HA or LB config). You rebuild against a fully patched base-OS container, spin-up, send some test load to it &amp; validate, then bring into service. Repeat for rest of nodes at that tier.<p>If you are trying to be an old-school sysadmin that expects to console or SSH in and run &#x27;yum upgrade&#x27; or &#x27;apt-get upgrade&#x27; your containers then you are doing containers wrong...

As a Java &#x2F; Hadoop &#x2F; Spark &#x2F; Scala fan, all I can say is, it&#x27;s a little embarrassing, not sure how the Java ecosystem around hadoop became so sloppy (I witness it first hand on a daily basis). I wish more people who are concerned with security &#x2F; ease of build would turn into contributing to maven, sbt, ivy and the hadoop project. Instead of hating the Java ecosystem, why not join it and make it better? Hadoop is ubiquitous, maven (and ivy &#x2F; sbt) are the de facto dependency management and built tools for that ecosystem, and if it&#x27;s broken, (or alienating people who are used to just have make &#x2F; rpm &#x2F; deb for anything) then those people should join and try to make it better.<p>Whether you like java &#x2F; maven &#x2F; ivy &#x2F; sbt or not, good chances you&#x27;ll end up forced to work with Hadoop (Java) or Spark (Scala), both of which use maven &#x2F; sbt for dependency and build.<p>I say, It&#x27;s all open source, if it&#x27;s broken, and you know where it&#x27;s broken, I think the Hadoop &#x2F; Java community will be happy to get suggestions &#x2F; pull requests to improve it.

It seems rather easy and effective, for NSA-like agencies, to hide crude exploits in complex projects. An unintended effect of Snowden&#x27;s whistle blowing is that it has become easier, because it has let them know that they don&#x27;t need plausible deniability requirement anymore.<p>Until Snowden, they were very cautious not to be caught, because, you know, what might happen if public opinion knew what a bunch of crooks they were? Now, they know that public opinion doesn&#x27;t really care, and that if they&#x27;re caught, they can mostly shrug it off, with politicians&#x27; complicity.<p>So, shoving a rather crude and detectable exploit in a messy product has become practically doable. If I were in charge of subsidies distribution for some 3-letters agency, I&#x27;d pour more money on Docker, Maven etc. than on TLS.

If you read german I can recommend you <a href="https:&#x2F;&#x2F;plus.google.com&#x2F;+KristianK%C3%B6hntopp&#x2F;posts&#x2F;gPpHx5Trec6" rel="nofollow">https:&#x2F;&#x2F;plus.google.com&#x2F;+KristianK%C3%B6hntopp&#x2F;posts&#x2F;gPpHx5T...</a> and <a href="https:&#x2F;&#x2F;plus.google.com&#x2F;+KristianK%C3%B6hntopp&#x2F;posts&#x2F;54v3MNX8ud7" rel="nofollow">https:&#x2F;&#x2F;plus.google.com&#x2F;+KristianK%C3%B6hntopp&#x2F;posts&#x2F;54v3MNX...</a>

Every once in a while someone figures out that we could entirely solve the dependency problem by packaging all the dependencies with the application. Everyone gets excited. After a while everyone gets unexcited when the problems associated with this approach become obvious.<p>Docker is merely a more extreme example of the &quot;package everything with the application&quot; idea...

An interesting point that I didn&#x27;t see the author bring up is the concept of how Docker images can be built in a layered fashion, and the potential for a false sense of security.<p>For example, you start with some sort of base image -- say phusion&#x2F;baseimage-docker[1] -- and proceed to layer your application on top of it. You &quot;trust&quot; Phusion. They do Phusion Passenger, it&#x27;s a real piece of software you heard of, and it&#x27;s not some random person on the internet.<p>At some point, there&#x27;s a bug, a problem, a security flaw, and you&#x27;re waiting on them to fix it... nothing, nothing. Maybe they get hacked and their base image is now infected. I haven&#x27;t bothered to look, but I&#x27;m guessing it would be a trivial amount of work to start the process of culling the most popular base images used by public Dockerflles, looking for the biggest trojan horse.<p>It seems like the whole model is ripe for pushing an understanding of what is actually running on a machine -- soup to nuts -- to the way side, and establishing a non-existent trust on the building blocks you&#x27;re using, lulling people into a false sense of security about their containers. A lot of people already believe that they&#x27;re already doing something much more secure by running containers, and arguably, they are... except for all of the places where malicious software can be added in, and the potential container breakout techniques.<p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;phusion&#x2F;baseimage-docker" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;phusion&#x2F;baseimage-docker</a>

System administration is as important as ever. Docker and other containers just simplify system administration across many different machines. The standard Unix user land tools are excellent and very flexible, but they are fucking god awful at configuration management. Docker solves the problem of &quot;how do I make sure I have the same versions and configurations of everything on all 500 of my compute nodes without having to lock them down completely?&quot; This question is meaningless if your base system image sucks, so you still need a proper sysadmin to build your Docker images.<p>Many places are rolling this type of sysadmin work up into DevOps. This scares graybeard sysadmins, because they see DevOps automating them out of a job. What they fail to see is that DevOps is a step up for them: it&#x27;s an explicit admission that system administration is as important as software development, and needs to be integrated into the software development process and managed through whatever management processes and tools the core dev team uses.<p>The ultimate driver behind this is a shift in the way technology organizations are managed. A few years ago, you would have functional silos: development, operations, product, etc. that would all contribute to one or more products. Employees reported up through the functional lead, and incentives were doled out based on cost effectiveness. This didn&#x27;t work well. So what started happening is that engineering executives began building product-focused silos instead. A development manager is no longer in charge of just software developers, but also QA, scalability and deployment. If the operations folks fuck up the deployment, the development manager gets chewed out about it. So the dev manager is going to bring as much of that under her control as she can.<p>Docker&#x2F;Maven&#x2F;etc. are the abstraction layer between the teams that manage the infrastructure (physical servers, VMWare pools, storage, network, etc) and the teams that manage the applications. This is no excuse for bad sysadmin practices; you still need good sysadmins in the DevOps role. But here&#x27;s the kicker: DevOps often pays more than system administration! And if you&#x27;re a SME in a very specific thing (say, Cassandra administration) you can be in a support role across a number of different teams, making sure their DevOps folks deploy Cassandra in a sane way.<p>(Yes, I realize all of this is centered on huge companies with massive engineering organizations. Small organizations have always required sysadmins to wear multiple hats, so none of this is new.)

&gt; Update: it was pointed out that this started way before Docker<p>Yes, like in the 90&#x27;s, at least, when people started using Java. Even prior to Maven there were jars, and we didn&#x27;t really know what was in them. And prior to that, I didn&#x27;t understand how every piece of software or hardware worked.<p>I was a big proponent of Gentoo when it came out because of building everything from source, but the fact is: I don&#x27;t have time to look through and understand every line of code. Even compilers can and have injected malicious behavior in the past. Firmware cannot even be trusted.<p>Some level of trust and reliance on others needs to be there. While it is true that there will always be people that betray that trust, without the trust, we would be hermits living alone off the land- which may not be so bad, but that&#x27;s another story.

&gt; »Docker is the new &#x27;curl | sudo bash&#x27;«.<p>Fully agree with this.<p>Maybe I am just only another grey beard grumpy developer, but the new generations that grew up with GNU&#x2F;Linux instead of UNIX, bash the security of other OSes and then go running such commands all the time.

I think the author is mixing up a few different topics. If you&#x27;re going to blame container frameworks for people sharing software in insecure ways you might as well blame the fact that executables are portable between compatible systems. Might as well blame the fact that there&#x27;s a network while you&#x27;re at it. We run docker throughout our infrastructure, but it is a deployment and dependency management technology, not a vector for infection. We run only our own images, which are all built from source or validated binaries. So what do insecure or unreliable practices have to do with containers, specifically?

For people who are interested in learning more about the problem. This is a really great paper : <a href="https:&#x2F;&#x2F;www.informatik.tu-darmstadt.de&#x2F;fileadmin&#x2F;user_upload&#x2F;Group_TRUST&#x2F;PubsPDF&#x2F;BNPSS11.pdf" rel="nofollow">https:&#x2F;&#x2F;www.informatik.tu-darmstadt.de&#x2F;fileadmin&#x2F;user_upload...</a>

&quot;Maven, ivy and sbt are the go-to tools for having your system download unsigned binary data from the internet and run it on your computer.&quot; You should setup a maven repository (Nexus, Artifactory) for your organisation if you want to have more control on binaries. Seems that artifactory can host docker files: <a href="https:&#x2F;&#x2F;www.jfrog.com&#x2F;confluence&#x2F;display&#x2F;RTF&#x2F;Docker+Repositories" rel="nofollow">https:&#x2F;&#x2F;www.jfrog.com&#x2F;confluence&#x2F;display&#x2F;RTF&#x2F;Docker+Reposito...</a>

Is it a coincidence that all the technologies the OP complains about are Java (Hadoop, Apache Bigtop, Maven, ivy, sbt, HBaseGiraphFlumeCrunchPigHiveMahoutSolrSparkElasticsearch)?

I completely agree with this except I think of it more as a problem of release engineering rather than system administration.<p>The trouble is, the sysadmin&#x27;s job is to deploy things. The developers job is to write code. Often release engineering isn&#x27;t thought of at all or if it is it&#x27;s given to the last qualified or least suspecting folks without any requirement from operations.<p>Developers aren&#x27;t taught about release engineering or deployment in school at all. In fact, it seems to me most university curricula do everything possible to hide all that from students.<p>Compounding that is the developers desire to get new code out conflicting with the sysadmins requirement to keep things stable in the face of limited QA automation. This leads to the common conflict between dev and ops.<p>This is to me a large part of what has led to the DevOps movement. This gives the developers information about the deployment and perhaps even access to it or a version of it and&#x2F;or a voice in deciding how things are deployed.<p>Hopefully we can standardize things widely enough that universities can teach this without fear of focusing on useless technologies that will be discarded in 3-5 years.

As an ex sysadmin I really like the container infrastructure. Manage the whole configuration on the main machine with puppet and deploy the blackbox applications (everything ruby and java related) with docker&#x2F;rocket.

Today I learned that the &quot;curl | sudo&quot; idiom is actually a thing people really do. Truly, everything is awful.

If you want to have guaranteed runtime linkage built from trusted source, you might want to give BOSH (<a href="http:&#x2F;&#x2F;bosh.io" rel="nofollow">http:&#x2F;&#x2F;bosh.io</a>) a look for config&#x2F;release management - it insists&#x2F;prefers on compiling all dependencies from source, from trusted links, with signature checks. For example with Hadoop, here is the build script:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;cf-platform-eng&#x2F;hadoop-boshrelease&#x2F;tree&#x2F;master&#x2F;packages&#x2F;hadoop" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;cf-platform-eng&#x2F;hadoop-boshrelease&#x2F;tree&#x2F;m...</a><p>Learning curve is a bit steep but it&#x27;s another approach to this immutable infrastructure trend that&#x27;s built for large production, enables rolling canary upgrades, etc.

And before &quot;curl | sh&quot; it was download from freshmeat and run &quot;tar xfz; cd; make install&quot;.<p>It&#x27;s not really better. Fact is we are running huge and complicated frameworks with lots of dependencies. These technologies are new and evolve fast. Distros don&#x27;t have enough volunteers to decouple this mess and thus fail to provide stable packages. There is a good chance nobody wants an old version of hadoop anyways.<p>Containers are a whole other problem. Always bothered me that no one cares about building these images themselves. The documentation is there, you can build your own docker&#x2F;vagrant&#x2F;... containers and vms. It&#x27;s just nobody seems to care anymore? Sometimes I don&#x27;t even know where these images come from, distro, community, ...?

I think it&#x27;s because people are getting worse at explaining things and writing docs.<p>As a student many times I wanted to learn how things work, but most tutorials&#x2F;docs just ask you to type in a few magical lines without much explanation. Maybe the authors think their audience won&#x27;t understand anyway, but I think it&#x27;s the authors&#x27; ineptitude if they can&#x27;t explain what their programs do in an accessible way.<p>I really hope there could be more projects like i3[1] and flask[2].<p>1: <a href="http:&#x2F;&#x2F;i3wm.org&#x2F;docs&#x2F;userguide.html" rel="nofollow">http:&#x2F;&#x2F;i3wm.org&#x2F;docs&#x2F;userguide.html</a> 2: <a href="http:&#x2F;&#x2F;flask.pocoo.org&#x2F;docs&#x2F;0.10&#x2F;" rel="nofollow">http:&#x2F;&#x2F;flask.pocoo.org&#x2F;docs&#x2F;0.10&#x2F;</a>

It&#x27;s not just system administration...<p>Minecraft is a meta-game about downloading unsigned JARs from the &#x27;net and running them with your own user account.

I&#x27;ve often thought about just offering my services as a sys-admin to the multitude of small startups that popup locally. Many, many developers -- almost no sys-admining skills amongst them. Just fire up a server on AWS and away you go.

The &#x27;curl | sudo bash&#x27; mention reminds me of OS X Homebrew. The one-liner installation script is still published in the home page front and center (though it&#x27;s running from a trusted source Github).<p><a href="http:&#x2F;&#x2F;brew.sh&#x2F;" rel="nofollow">http:&#x2F;&#x2F;brew.sh&#x2F;</a><p>One might argue that that easy of oneliner installer script is exactly what make Homebrew gains popularity. And dev machine is different from production environment in terms installation packages. Still, I agree that proper container management in local network and perhaps new security features from upstream container vendors would help the situation.

So, asking the obvious question: what&#x27;s the solution to that?

As a Windows&#x2F;VMWare&#x2F;Exchange&#x2F;Cisco admin, this is all completely foreign to me. Does this indirectly make a case for paying a vendor real money to manage their product properly? All vendors we work with provide installers that handle installing any dependencies. Updates are a few clicks. Occasionally we run into a vendor who provides installation&#x2F;upgrade instructions, that involve manually copying files and hand editing config files. We replace those vendors. Error-prone people should not be doing manual file copying&#x2F;editing or dependency checking, tasks that computers are dozens or orders of magnitude more competent at. This is B2B stuff where businesses should manage their product or risk getting sued out of business. The current environment seems to be that using free or opensource products is &quot;free, with purchase of a team of consultants&quot;. Why not just pay the money to a vendor to provide, and support, a real product? It seems backwards to call this a sad state of sysadmin. This is like Boeing providing its leftover parts and a 9000 page manual on 747 assembly, and people complaining about the &quot;sad state of mechanics&quot;. That&#x27;s backwards. Buy a 747 from Boeing if that&#x27;s what you need.

Give me the command line and I&#x27;ll build anything!

&gt; <i>Maven, ivy and sbt are the go-to tools for having your system download unsigned binary data from the internet and run it on your computer.</i><p>Not Maven.

I&#x27;m a bit puzzled. Let&#x27;s say I decide to not download the binary but build it from source. Unless I actually <i>read</i> the source, I&#x27;m trusting the community to have read it, which consists of other people thinking I have read it.<p>In my view, this is true for the OS itself. So unless I read everything, I&#x27;m fucked. And I don&#x27;t. Thus I&#x27;m fucked.<p>Do I miss something?

&gt; And then hope the gradle build doesn&#x27;t throw a 200 line useless backtrace<p>This is more the fault of the language Gradle chose for its build configuration language. Most build scripts are between 20 to 50 lines long, but reading through those Groovy stack traces eliminates its supposedly write-one-read-many-times benefits.<p>Hopefully Gradleware will fix this problem for Gradle 3. They&#x27;ve already enabled Gradle to be configured on the fly by Java code, and could by working towards allowing any dynamic language to be a build language through an API. Alternatively, they&#x27;ve just employed one of the ex-Groovy developers recently made jobless by Pivotal pulling funding from Groovy and Grails last month -- they might get him to write a better lightweight DSL from scratch that parses the existing syntax, but isn&#x27;t weighed down by all of the present cruft.

A lot of big projects are terrible to build.<p>Once upon a time minimising dependencies was considered good practice. Now I get a pasting if I write clean code without reusing someone else&#x27;s library... even if the suggestions don&#x27;t solve my problem directly or at all and comes complete with a sloppy &#x27;no one click build&#x2F;deploy&#x27; configuration... the kind I was embarrassed to produce on my stand alone projects in my teenage bedroom days.<p>Shame on developers everywhere for tolerating this mess. I (am lucky enough to enjoy the freedom of choice that I) would leave a job if not allowed to start fixing such a situation from day one.<p>That being said, good sysadmins and developers should work out these problems properly instead of shortcutting through someone else&#x27;s half arsed effort via Google.

On the one hand it&#x27;s nice that developers are using more libraries and writing less from scratch. On the other hand dependencies are out of control. The one thing that really bothers me is code that depends on a particular IDE. That shit drives me up the wall.

Is this the sad rabbit hole reality of attempting to abstract every last component?

Regarding curl PACKAGE | sudo bash...<p>Just what do you think happens when you run `yum update` or Windows Update?<p>If you don&#x27;t trust DNS or the network then you have serious challenges which frankly aren&#x27;t even solved by air gapping file transfers.

There is some truth in this, yes. On the other hand, maven (mentioned by the author) clearly was very successful at abstracting from the tools we use. I can remember how much time I wasted with build scripts and dependency management and all that before. (And I still do on some other platforms.) The problems only arise if the abstraction is not working well enough. This might indeed be true for containers - there is maybe too much complexity in there that currently can&#x27;t be properly encapsulated.

Because Docker is good at containing crap, it is used to cover a multitude of sins. Before using Docker, please simplify your install and upgrade processes.

Containers and VMs are part of the solution to this problem, not a cause. Try managing the same dependencies across N platforms rather than one container!

Fully agree with everything written there, with the exception of &quot;apps.&quot; I believe it is not a Microsoft term, but an Apple term. Is it not?

Incidentally I feel like my admin &quot;skills&quot; have never improved faster than since I started working with docker.<p>Docker lets me iterate on system configuration faster than ever, and that means learning the details and quirks of certain software faster. Then again, I usually don&#x27;t use prebuilt VMs and containers, but have to prepare them for people who don&#x27;t want to pay for good sysadmins..

Nicely said, I thought I&#x27;m the only one who noticed this ^_^ This is one of the reason why I tried Docker&#x2F;Vagrant images few times and said no thanks :) I would rather spend my time and install everything on separate server my self then have unknown set of packages or security holes. As few articles on HN shown this containers not secure at all.

I don&#x27;t think any experienced organization is going to just download containers off the internet to use on their servers. Which is why there is self-hosted Registry applications that corps and big companies buy to host their own, where they build their images to support their applications and that are vetted through traditional corp policy

This &quot;working out of the box&quot; phylosophy is the direct result of devs using Windows and OSX platforms for creating those programs. They now think &quot;Linux and *BSD should be as easy to use as Mac.&quot;. Indeed, the majority of devs is mediocre amateur sysadmins. They know next to nothing beyond their preffered language.

70s system software is really showing its age. Containers are just a hack to make its complexity sometimes easier to manage.

Sys admins got relegated to tech support so web devs could add sys admin to their work flow.

Looks like description of the projects with bad build system, not like problem with e.g. maven. Maven downloads binaries from the HTTPS server. You can always get those libraries and rebuild them from sources into your internal repository.

So I like this article and want to learn more about sys admin&#x2F;dev ops&#x2F;whatever but where do I go. Is docker bad? What is a good starting point? What are best practices?

Can someone tell me what realistic security problems this mode of operation introduces that can&#x27;t be mitigated&#x2F;avoided with sensible network and backup configurations?

I thought this post was overly cynical and full of generalizations. I don&#x27;t really understand what point is trying to be made here.<p>&quot;Everybody&quot; &quot;Nobody&quot; &quot;Nobody&quot; &quot;None of&quot; &quot;everything got Windows-ized&quot; every sentence is a broad generalization on top of cynicism so it&#x27;s hard to find any value in the point trying to be made.

This is the price of devops.

Previous discussion: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=9190955" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=9190955</a>

What is really ironic is that none of this &quot;tools&quot; solves any fundamental problem of so-called version hell and none of this containers are fundamentally different from<p><pre><code> .&#x2F;configure --prefix=&#x2F;xxxx &amp;&amp; make &amp;&amp; make -s install </code></pre> with or without following<p><pre><code> chroot &#x2F;yyyy </code></pre> The big &quot;innovation&quot; of having so-called &quot;virtual env&quot; (they call it &quot;reproducible [development] environment) for each &quot;hello world&quot; (a whole python&#x2F;ruby&#x2F;java&#x2F;etc installation with all packages and its dependencies in your [home] project directory) solves no real problem, only pushes it to the next guy (what they call devops).<p>Some idiots even advocating to have a whole snapshot of an OS attached to your &quot;hello world&quot;, and even to make it what they call &quot;purely functional&quot; or even &quot;monadic&quot; (why not, of someone pays for that).<p>Unfortunately, there is no way to ignore complexity of versions and package dependencies or easily pushing it to &quot;devops&quot;. Creating a zillion of &quot;container images&quot; with just your &quot;reproducible development environment&quot; or a whole &quot;OS snapshot&quot; just <i>multiplies entities without a necessity</i>.<p>Programmer <i>must</i> be aware of which version of what API implemented with what version of package or library he using and explicitly assert and maintain these requirements, like all the very few sane software projects (git, nginx, redis, postgress) do.<p>btw, the GNU autotools (which gives us .&#x2F;configure) is somewhat evolved real-world solution - you have to explicitly check each version of each API <i>both</i> at the compile (build) time and refuse to build in case of unsatisfied dependencies <i>and</i> at the install time (and package manager must refuse to install if case of mismatch). This is the only way back to sanity, however &quot;painful&quot; it is.

Gentoo and BSD ports exist for a reason.

I don&#x27;t have a strong opinion either way about Docker, but I understand the OP&#x27;s gripes.<p><i>Stack is the new term for &quot;I have no idea what I&#x27;m actually using&quot;.</i><p>This was great. It leaves me to wonder what &quot;full-stack&quot; means.<p>For one thing, we have a culture of trust inversion. I wrote about it in a blog post about a month ago: <a href="https:&#x2F;&#x2F;michaelochurch.wordpress.com&#x2F;2015&#x2F;03&#x2F;25&#x2F;never-invent-here-the-even-worse-sibling-of-not-invented-here&#x2F;" rel="nofollow">https:&#x2F;&#x2F;michaelochurch.wordpress.com&#x2F;2015&#x2F;03&#x2F;25&#x2F;never-invent...</a> . The &quot;startup&quot; brand (and it <i>is</i> a brand) has won and most companies trust in-house programmers less than they trust off-the-shelf solutions. This tends to be a self-fulfilling prophecy. Because few corporations will budget the time to do something well (make it fast, make it secure, make it maintainable) it only makes sense to use third-party software heavily and use one&#x27;s own people to handle the glue code, integration, and icky custom work. (That, of course, leads to talent loss, and soon enough, when it comes to build vs. buy your only option <i>is</i> to buy, because your build-capable people are gone.) At some point, however, you end up with a large amount of nearly-organic legacy complexity in your system that no one really understands.<p>Although it&#x27;s not limited to one language or culture, this is one of my main beefs with Java culture. It has thoroughly given up on reading code. Don&#x27;t get me wrong: reading code (at least, typical code, not best-of-class code) is difficult, unpleasant, and slow and, because of this, you invariably have to trust a lot of code without manually auditing it. But I like having the idea that I <i>can</i>. The cultures of C, OCaml, Haskell, and to a degree Python, all still have this. People still read source code of the infrastructure that they rely upon. But the Java culture is one that has given up on the concept of reading code (except with an IDE that, one hopes, does enough of your thinking for you to get you to the right spot for the bug you are fighting) and understanding anything in its entirety is generally not done.

The problem is you old sysadmins are so passé. Software has replaced you, and you need to get over it. Developers are finally liberated to move at full speed without hearing &quot;NO&quot;

So much truth spoken in the linked text. Thanks.<p>It has to be said. Damn the containers and windowization of Linux.

Except that Docker explicitly allows and encourages signing of core infrastructure containers.

haha you&#x27;re my new hero<p>YOU ONLY LIVE ONCE MAN! trust the (maven) system