Concourse CI

Have been working with it for a few months, and I am not positive about it.<p>UI is pretty but often breaks. Is unusable when you have a pipeline with dozens of concurrent jobs.<p>Concept of teams is fine, but when you have to switch between them, even with oauth authentication, it is a pain.<p>Job concurrency control is binary.<p>We never have been able to have worker pool scale down without a hiccup, always some darn worker hogging containers and atc not removing it, leading to stalled pipelines.<p>That overlay network it comes with, garden thingy, creates so many problems and solves just one...<p>Oh and not having BUILD_ vars available in tasks is rude, thank you very much, but there are cases when it is just mandatory and concourse makes it impossible to do.<p>At least new version has better secrets handling, previously it was a joke.

Not sure when it came about, but when I first evaluated Concourse, not being able to trigger jobs manually was a primary blocker. Glad this showed up on HN again, because they&#x27;ve added it in at some point. My favourite thing about Concourse was really its ideas around &quot;Resources&quot;. This always felt so much better than this idea of plugins due to how unified the experience was. Also, the ability to implement resources for yourself is extremely easy. So if I had an internal software running, being able to build something for it meant defining three bash scripts.<p>That said, the docs around implementing resources still needs some improvements. Whatever I learnt was from cloning and modifying existing resources.

I know everyone hates on Jenkins these days, but (at least for the workloads I am building) Concourse feels like a toy.<p>My issues:<p>- everything <i>has</i> to be in a docker container (not all things can run in garden)<p>- Just trying to find logs is a pain, and getting the UI to show a full log output is basically impossible<p>- each install of concourse may need an entirely different version of fly<p>- There was no way of retriggering a CI run on a single PR (without force pushing to the branch, which removes Github reviews)<p>Jenkins + Jenkinsfiles + Pipelines provides all of the good bits of Concourse, with none of the Pivitol enforced workflow + tools.<p>I get why CloudFoundry uses Concourse - the build process is so arcane that you basically need to run the full CI locally, but I really do not get the hype for other projects.

Mostly off-topic but I&#x27;ve been looking for more than a CI for some quite time, more on the CD side. How are you guys handling some of these cases? Bonus points for hosted options.<p>- Truly a pipeline based stages. No messing around with git branches&#x2F;tags for each environment release.<p>- Ability to combine multiple builds together.<p>- Build dependencies. Ability to trigger project-B build when project-A build succeeds. (Docker Cloud Builds have something like this.)<p>- Use the same artifact across multiple build stages.<p>- An option to promote a build to the next step either manually or automatically.<p>Amazon&#x27;s internal Apollo system was amazing. AWS Code Pipelines kind of does some of these things but it&#x27;s every limited and hard to work with.

Avoid! Concourse looks good on the surface but it&#x27;s really not that great.<p>The UI is clunky.<p>The abstraction layer is too low and leads to a lot of repeated YAML. Which leads to YAML programming.<p>There are simple scenarios like deployment rollbacks who are hard to do.<p>For some reason they decided to develop their own container engine which leads to all sorts of trouble and maintenance issue. It&#x27;s generally slow and we had 100% CPU usage when the worker was doing almost nothing.<p>I have used for 4 months and it was only problems. Gitlab CI is much better. Or even Jenkins is better.

We&#x27;re using Concourse extensively at HelloFresh (&gt;130 devs). It&#x27;s not without its quirks, but I&#x27;ve little to complain about so far, except perhaps the polish of the UI.

We use concourse.ci to successfully run cloud.gov at 18F.

Disclaimer: I work for Pivotal, on the RabbitMQ team. We push Concourse to its limits every day. We work closely with Jenkins, GoCD, Travis &amp; Concourse. They all have their limitations.<p>All things will break horribly if the conditions are right. It&#x27;s unreasonable to assume that the things which work in [insert your current CI] will work in Concourse. It&#x27;s still a new and relatively immature product, but it works well in most cases.<p>Half the secret to a good Concourse experience is not upgrading it in-place - stand up fresh deployments. The other half is gradually transitioning between Concourse deployments, because bad versions have been and will continue to be released - mistakes are only human. As long as you share the Concourse vision and are willing to keep up with the pace of change - not everyone can or wants to - then it&#x27;s an amazing CI.<p>Concourse still makes me excited, even after many years of hard lessons, because it is a genuinely innovative approach to building better software. Most miss this, and I understand why, but give it time - the ideas behind it will mature and become the norm.<p>Even though Concourse can work really well, it&#x27;s not always the best choice. Make it better if you can &amp; want to, use something else if it&#x27;s easier. There is no right or wrong, just preferences : )

I&#x27;m late to the party.<p>Concourse is difficult to come to from other CI tools. A little more aloof. There have been real, serious implementation difficulties.<p>But I kinda love it, because it&#x27;s a handful of simple ideas that unlock incredible power. It goes beyond &quot;build and test each commit&quot; to becoming a full project automation tool, a software manufacturing robot. When I talk to people about Concourse, I tell them: your pipeline and your tasks are <i>production code</i>. Keep the discipline, care and engineering practices that you bring to the apps and services you create.<p>The problem is that most of the deep tribal knowledge about how to get started and how to best apply it is locked up inside a handful of organisations, most notably Pivotal. I had been working on a video series which was meant to walk through both the concrete business of building pipelines, as well as the concepts of how best to do so.<p>Unfortunately visa conditions got in the way and I have abandoned that effort. Interested persons are welcome to email me for links to the first 4 episodes that I made, but be aware that it stops even more suddenly than Firefly.<p>Disclosure: I work for Pivotal.

Kind of odd to see the homepage show Vagrant as the install mechanism, even though it supports Docker as well. In 2017, I&#x27;d think more developers are likely to run Docker than Vagrant workloads on their machine.

I&#x27;ve been eyeing Concourse and Go.CD over Jenkins for a while.<p>The main criticism I saw on Jenkins and Go.CD vs. Concourse was that Jenkins Pipelines aren&#x27;t first class and that it&#x27;s easier to export configuration(in that regard Concourse &gt; Go.CD &gt; Jenkins). On the other hand Jenkins and Go.CD supports extensions, which Concourse touts as a feature.<p>I also want the CI builds to create my base boxes with packer in multiple steps. And I somewhat want to be able to hand over the stuff to ops at some point to be able just stay alive for the next 5 years or more. Would anyone know if it makes sense to even consider concourse or go.cd or some other CI&#x2F;CD solution and if so which?<p>Obviously the boxes need to be used as artifacts and everything has be on premise as well.

My number one complaint with Concourse (which I suspect is due to Go) is that you need to have it hosted with a valid TLS&#x2F;SSL cert in order to use the fly command. At least this was an issue in the 2.6.0 days, but I couldn&#x27;t see anything to change this in the recent versions.<p>This is rather annoying if you want to run a copy on your local network say at home. Its very frustrating because the fly command solves the biggest issue with IWOMM (it works on my machine) by allowing you to run code and tests on another machine before committing anything.<p>I think from memory I tried using self signed certs and this also had issues for one reason or another.<p>That said it is still the best CI system I have used to date.

A little late to this discussion, but although I&#x27;m a big fan of many of the concepts in Concourse, I think it&#x27;s lacking a lot of polish.<p>For example, I have actually implemented patterns within Jenkins to force all jobs to run inside of containers. A job is just a container + command with some linking using the jenkins pipeline plugin that reads some json configuration to determine how jobs are linked.<p>The primary issues I have surround the fact that my company uses kubernetes, thus we have no insight into the runc containers. Load balancing in concourse is non-existent. If a worker goes down due to load, if you bring it back up, it&#x27;s going to go down immediately from all the jobs that have been triggered while the worker was offline. Not only that, the resource requirements seem pretty high. Recently a concourse worker stalled because the amount of volumes&#x2F;images it was caching was over 100 gigs, and not knowing the internals, I wasn&#x27;t sure what the best way was to clear this cache. Having to tell the infrastructure team that we just need to spend more money is a hard sell when we&#x27;ve upped the cpu, memory, storage, postgres disk, all more than once. I understand that different images have vastly different sizes, and jobs different amounts of work, but their need to be some clear suggestions for sizing. If there exist them now, I apologize but I haven&#x27;t seen them.<p>So yeah I&#x27;ve had some fun developing in it, but more help making it reliable would be really nice. Also if kubernetes is absolutely the wrong way to run it, which it seems like, I&#x27;d have to be provided a better&#x2F;easier alternative to really become an advocate.<p>Final note: has anyone actually setup metrics&#x2F;monitoring for concourse that doesn&#x27;t know BOSH? The docs describing it seem huge unless you already have the infrastructure pieces. Let&#x27;s setup riemann (we already have statsd and no experience with riemann), emit to influxDB (we have prometheus, no experience with influxDB), then use Grafana (ok we already have that). I just wanted a better idea of disk, cpu, mem, # of containers, lifecycle without having to setup all these new pieces of infrastructure. Finally, just not that interested in BOSH, which all of the example metrics repo&#x27;s are.

I took part in evaluating Concourse CI for the needs of my company (30+ devs). While it has amazing CI pipeline capabilities, we ultimately didn&#x27;t select Concourse because it felt much more like a CI toolbox, requiring some development to put those tools to use. And what we really wanted was more of a turnkey CI product.<p>Perhaps ironically, we ended up doing some development around the edges of the CI product we ultimately selected (GitLab).

Excellently designed system. Very poorly implemented.<p>Our team&#x27;s been using it since it&#x27;s initial releases. It&#x27;s been nothing short of disastrous for all but the smallest pipelines.<p>The design is great. Keeping configs in yaml instead of little white boxes in a Jenkins database is much better. Pipelines as a first class concept. It feels inspired by a functional programming language. You get great build reproducibility since there are no workers that get dirtier over time if you forget to clean up. The resource model is awesome. Very cool stuff. I&#x27;m hoping every CI system learns from what&#x27;s here. Second to none in design.<p>However it performs like a dud. No scheduling to speak of, just runs everything as soon as it can. We&#x27;ve run into nodes dying under load (-not- underprovisioned, could run all these jobs manually at once on these monsters). We&#x27;ve run into problems with volume reaping, fork bombs, ui freezes, everything under the sun.<p>I really like Concourse and will hopefully one day be able to come back to it when its implementation is as solid as its paradigms are.<p>But I&#x27;d avoid use for now.

We are using concourse at work right now and have 50-ish pipelines for various repositories. IMO, there&#x27;s definitely some work you have to put into it because you&#x27;ll sooner or later run into a problem with the existing resources and need a custom one. Writing custom resources is pretty easy however.<p>Concourse also isn&#x27;t really made to work well with the Git Flow, there is no builtin way to run the CI on multiple branches (there&#x27;s a git-multibranch community resource which requires redis at some point). we&#x27;re basically thinking about changing our workflow to trunk-based, but it still feels weird to me that we might change our workflow to fit our CI better.<p>that being said, I personally still really like concourse and it&#x27;s fun to work with.

I like how it looks, but first impressions after trying to make it run on windows ended up in failures:<p>Refused to execute script from &#x27;<a href="http:&#x2F;&#x2F;127.0.0.1:8080&#x2F;public&#x2F;index.js?id=932c553753eec4ef375e1c17f4b28c10&#x27;" rel="nofollow">http:&#x2F;&#x2F;127.0.0.1:8080&#x2F;public&#x2F;index.js?id=932c553753eec4ef375...</a> because its MIME type (&#x27;text&#x2F;plain&#x27;) is not executable, and strict MIME type checking is enabled.

I evaluated concourse CI and I was impressed by the concept.<p>Pros - Setup was really easy.(I didn&#x27;t use bosh). - Both server and build pipeline configuration (Yaml file) are easy to backup and recreate.<p>Cons - When I evaluated it , it had a few bugs which forced me to restart the server almost daily to keep it working. It should be more stable now.<p>-

We looked at Concourse deeply, while we didn&#x27;t go with it, it&#x27;s inspired a number of projects I know and I think has pointed out the obvious; representing how you think about a problem is how you should represent it visually, great insight that has been overlooked in CI.

I hadn&#x27;t heard of this, it looks nice! Would I be far off if I&#x27;d say that Concourse is an open source clone of Travis CI and CircleCI? Any strong pluses of the SaaS offerings over Concourse I should know about?

During my Jenkins build I am creating a file with the number of database queries per test, since I am generating the file it can be in any format.<p>How can I feed this to Jenkins and plot it over time?

Curious to know why no one actually mentions Bamboo - definitely a great alternative to Jenkins, especially for those, who are into Atlassian infrastructure.

Concourse is a really amazing piece of software, but the BOSH requirement I think will keep adoption low for small companies.

Typo on front page.<p>Rather than &quot;Rather than a myriad of checkboxes...&quot; it should be &quot;Rather than myriad checkboxes...&quot;

We&#x27;ve been happy teamcity users for years.

We&#x27;re using Buddy - <a href="https:&#x2F;&#x2F;buddy.works&#x2F;" rel="nofollow">https:&#x2F;&#x2F;buddy.works&#x2F;</a> - and it seems to be easier, faster and more reliable than most of these other CI solutions so far.