Pretty cool! We use Airflow heavily here at Instacart. Some of our teams use a managed service from google for deployment and orchestration <a href="https://cloud.google.com/composer/" rel="nofollow">https://cloud.google.com/composer/</a>
For companies wanting a standard structure of dags and self hosting their airflow deployments, your tool would be super helpful to get started.
One suggestion - would be cool to add separate deployments for the different components of airflow - webserver, workers, scheduler etc. reading through the readme it looks like you deploy the single image to the Qubole Cloud? Often times deploying code to airflow is updating the dags files in airflows file system.
What's the simpler solution analogous to airflow that I an deploy on the same ec2 instance as my webserver? I prefer to use the simpler tool until it becomes unwieldable, and my current stack is just an elastic beanstalk deployment that runs the webserver as well as a celery worker. This seems easy to make the site highly available (scale up and down instances easily), with managed RDS and a redis broker taking care of managing state. A scheduled task runner is the missing piece - turns out celery is just not designed for long-running tasks, and both celery and airflow seem to require that only one instance of their scheduler run at any given time. I'd much prefer a tool where the service is robust enough to handle multiple redundant schedulers so I can go home and sleep without having to bring in kubernetes just to deal with this.<p>Currently using some custom code to do this using a database table but totally aware of how fickle and easy to screw up this method can be, and open to suggestions.
'Airflow' as in ....?<p>It does ring a bell, but airflow gives a lot of hits on google. Would be helpful to add a link in your readme to 'your' Airflow.
<a href="https://astronomer.io" rel="nofollow">https://astronomer.io</a> offers a managed platform and installation into your cloud or data center.<p>Great product experience, fully open source. Gives your people using Airflow a sweet UI so they don't need to go CLI