I 100% agree with the premise of this, I certainly attest to run tracking being a critical component of my own personal model development/training/tuning pipeline. Not sure how I feel about using yet another 3rd party data science platform though - it'd be nice if the run tracking piece was just a component I could import into my notebook, and have it automagically track things for me (like... how seamless tqdm is for showing progress bars while iterating over loops).<p>Regardless of my uncertainty around trying another 3rd party platform, the premise is spot on.
This is an ad, but they're not wrong.<p>At my work, we use Azure ML Studio. I think the solution for deployment is to run some scripts to save the model information to git and automatically deploy from there. It will take a little bit of effort to set up, but I think it should work.
I totally agree with this and I built wandb (wandb.com) to solve this problem. We try to do this in as lightweight a way as possible - for example we can do keras tracking with a single line (<a href="https://www.wandb.com/articles/visualize-keras-models-with-one-line-of-code" rel="nofollow">https://www.wandb.com/articles/visualize-keras-models-with-o...</a>) and pytorch with just a couple lines (<a href="https://www.wandb.com/articles/monitor-your-pytorch-models-with-five-extra-lines-of-code" rel="nofollow">https://www.wandb.com/articles/monitor-your-pytorch-models-w...</a>). Would love any feedback on it.
I understand that for some people this might help, but frankly I find all of these "reproducibility" frameworks fall flat as soon as truly big data enters the picture. Data versioning is not sufficient, because I typically cannot roll back my datasets to a previous version (and we moved forward for a reason).<p>Also, we are deliberately not using Databricks for this to avoid vendor lockin for something that will almost certainly be open-source soon.
Seems like there's been an explosions of startups trying to win B2B dollars for this.<p>There is an excellent open source project that nails this called sacred. It's not perfect, but it works, and as far as I can tell it has won the popularity contest.<p>Please join me in using and contributing back to sacred!