This seems so overengineered. If you're using a BI tool like Looker, you can configure how often and how long you would like your "query" to be persisted. They also support dynamic triggering - when value X changes, rerun query Y. This is all 2-3 lines of LookML configuration.<p>Also, pulling this out of the dependency graph of the rest of your ETL system seems silly. Analysts now have to schedule their queries "sometime roughly when I think the data will be available" vs "immediately once the data is available".<p>It sounds like Chartio isn't really keeping up with their workflow. So assume for a second they need to ease this pain but can't justify a vendor switch right now. What's the fix? I think you make it easier for analysts to put jobs into Airflow. That way you aren't reinventing the wheel (building a completely isolated ETL scheduling tool) and you're actually attempting to solve one of th underlying problems (Airflow isn't "analyst friendly").<p>I'd love to hear other people's experience. I recognize I'm being a bit of a negative nancy.