This seems so overengineered. If you're using a BI tool like Looker, you can configure how often and how long you would like your "query" to be persisted. They also support dynamic triggering - when value X changes, rerun query Y. This is all 2-3 lines of LookML configuration.<p>Also, pulling this out of the dependency graph of the rest of your ETL system seems silly. Analysts now have to schedule their queries "sometime roughly when I think the data will be available" vs "immediately once the data is available".<p>It sounds like Chartio isn't really keeping up with their workflow. So assume for a second they need to ease this pain but can't justify a vendor switch right now. What's the fix? I think you make it easier for analysts to put jobs into Airflow. That way you aren't reinventing the wheel (building a completely isolated ETL scheduling tool) and you're actually attempting to solve one of th underlying problems (Airflow isn't "analyst friendly").<p>I'd love to hear other people's experience. I recognize I'm being a bit of a negative nancy.
Cane here to say just that. This feels wayyy over engineered.
First of all, do they really need 30 analysts pouring over viewer stats?! Isn't their primary business journalism and producing good quality content? I wonder how many analysts and data engineers they have supporting the articles and research..
Couldn't they just use data exploration tools like Looker and focus on their primary business instead?<p>Now, I go back to my naive village wondering why The Times Digital subscription costs so much...