How to put machine learning models into production

Too many people focus on &quot;properly&quot; putting ML into production...<p>I&#x27;d like to propose an alternative... Build a model (once) on your dev machine. Copy it to S3. Do CPU inference in some microservice. Get the production system to query your microservice, and if it doesn&#x27;t reply in some (very short) timeout, fallback to whatever behaviour your company was using before ML came along.<p>If the results of yor ML can be saved (eg. a per-customer score), save the output values for each customer and don&#x27;t even run the ML realtime at all!<p>Don&#x27;t handle retraining the model. Don&#x27;t bother with high reliability or failover. Don&#x27;t page anyone if it breaks.<p>By doing this, you get rid of 80% of the effort required to deploy an ML system, yet still get 80% of the gains. Sure, retraining the model hourly might be optimal, but for most businesses the gains simply don&#x27;t pay for the complexity and ongoing maintenance.<p>Insider knowledge says some very big companies deploy the above strategy very successfully...

Overall, a well written article.<p>If you&#x27;re interested in ML Ops, I have a shameless plug to share: on November 19th I host a free online panel, &quot;Rage Against the Machine Learning&quot;, with industry experts. [0]<p>[0]: <a href="https:&#x2F;&#x2F;cotacapital.zoom.us&#x2F;webinar&#x2F;register&#x2F;8116020076218&#x2F;WN_DIIptnvUQhi0AkSze_XhAw" rel="nofollow">https:&#x2F;&#x2F;cotacapital.zoom.us&#x2F;webinar&#x2F;register&#x2F;8116020076218&#x2F;W...</a>

There&#x27;s a great paper from Google about this &quot;Machine Learning: The High Interest Credit Card of Technical Debt&quot; [0] which discusses why you should use a framework to deploy ML models (the authors are involved in developing TFX).<p>In my experience, spending time explaining results to the business is also a very time consuming element of deploying a model too.<p>0:<a href="https:&#x2F;&#x2F;research.google&#x2F;pubs&#x2F;pub43146&#x2F;" rel="nofollow">https:&#x2F;&#x2F;research.google&#x2F;pubs&#x2F;pub43146&#x2F;</a>

I was expecting this to be more about running inference in production, though the information in the article itself was interesting on its own.<p>There does seem to be a dearth of writing on the actual topic of deploying models as prediction APIs, however. I work on an open source ML deployment platform ( <a href="https:&#x2F;&#x2F;github.com&#x2F;cortexlabs&#x2F;cortex" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;cortexlabs&#x2F;cortex</a> ) and the problems we spend the most time on&#x2F;teams struggle with the most don&#x27;t seem to be written about very often, at least in depth (e.g. How do you optimize inference costs? When should you use batch vs realtime? How do you integrate retraining, validation, and deployment into a CI&#x2F;CD pipeline for your ML service?).<p>Not taking anyway from the article of course, it is well written and interesting imo.

I&#x27;ve recently come across the MLOps community here <a href="https:&#x2F;&#x2F;mlops.community&#x2F;" rel="nofollow">https:&#x2F;&#x2F;mlops.community&#x2F;</a>.<p>The meetups are all on YouTube and have great topics like putting models into production, but also more interesting ones (to me) like ml observability and feature stores.<p>Their slack channel is great too, learned a lot about the reality of using kubeflow vs the medium article hype

As a practical detail, I&#x27;m wondering if it always makes sense to wrap your predictor in a simple if-then based predictor. If your learned model makes bad predictions in certain specific cases, you can &quot;cheat&quot; with Boolean logic. This could also be useful when the business has a special case that doesn&#x27;t follow the main patterns.<p>Any thoughts on that?

The title doesn&#x27;t really match the article in my mind. To me, it talks about everything <i>but</i> actually deploying a machine learning model in production. In particular, there are a lot of words around where training data is stored. In my experience, the training data is really more part of the the development process than the actual productionisation of the model.<p>That said, there is a piece here on TFX, which is valuable in this context. I also think the advice about going with proprietary tools that speed up the process is good. Tools like Microsoft&#x27;s AI tooling, Dataiku and H20 are good in that context.<p>I would have liked to have seen some discussion around when you should deploy a model as an API vs generating batch predictions and storing them - I&#x27;ve done both on a test bench, but I don&#x27;t really know how well the API scales.

is anyone running TFX in their companies in production ? how has the experience been ?<p>since like everyone is on K8s, im wondering if kubeflow is not the more natural fit