MLOps is a mess but that's to be expected

As a researcher applying ML in my work the OP reads like me looking from the outside at web-frameworks: there&#x27;s a LOT of noise but in practice, people run just the same 3 things.<p>For example, when I first got into ML the ADAM optimizer was the new big thing, since then hundreds of &#x27;better&#x27; optimizers have been published. This paper from August &#x27;21 shows that most of that is overblown and no optimizer consistently outperforms ADAM: <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2007.01547.pdf" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2007.01547.pdf</a><p>I&#x27;d even go so far and say that OP&#x27;s big image of MLOps is misleading. &#x27;Data Science Notebooks&#x27; shows Jupyter, binder, colab, I can&#x27;t see the other logos; but binder and colab both run Jupyter notebooks? Under ML platforms there are a ton of companies in this picture which effectively do the same thing. Some of these logos are tools (Jupyter, R), some of these logos are companies using ML in some way or other (John Deere, Siemens) - and once you go down this path you might as well put any mid-sized company in the world onto this.

I work as a Senior Software Engineer and DevOps for a AI consultancy, and I&#x27;ve been dealing with MLOps for the past 2 years.<p>I&#x27;m mostly aligned with what the article says, MLOps today is definitively in a frazzled state. However, I disagree on the following points :<p>- Google and al. are not good example to follow for ML deployment best practices. Sure their sophistication is higher, but they also have a lot more staff to handle &quot;the other side&quot; of modelling : Data, Infrastructure, Tooling. They build tools to suit their needs, as big SaaS products holding *bytes of already organized data ready to be used for data science, and it&#x27;s a really different perspective than a large retailer trying to get better sales forecasts.<p>- Vendors from all sides have really, really bad fitting products for Data Science and ML in general. &quot;Platforms&quot; are trying to profit off MLOps with a commercial product claiming to be the silver bullet to every pain your team has. Three months later, it&#x27;s just another life-sucking lock-in with a list of tickets to be addressed. We really miss a new &quot;Docker&quot; here. Have a few examples: Databricks ? It&#x27;s a Spark-as-a-Service platform with the worst APIs you could imagine. Git-for-data vendors ? They don&#x27;t understand Git nor Data : is a model data or code ? Both ?<p>Finally, ML at Reasonable Scale builds on top of regular software engineering best-practices. If you don&#x27;t usually store a shell script&#x27;s output in a repository, then you should do the same for a notebook. Same goes for Idempotency, Reproducibility (of model training), Composability (of pipeline steps), (data) Versioning, etc..

MLOps is radically overhyped.<p>I&#x27;ve seen two companies in the last year start twisting themselves into circles to worry about &#x27;productionizing&#x27; a data science project.<p>The intent is right to focus on end use case and business value, but it feels like the communication is exclusively in terms of overbuilding systems.

It’s not that big of a deal.<p>1. Collect new data<p>2. Clean data<p>3. Annotate<p>4. Train models and store versions<p>5. Analyze errors&#x2F;model metrics (and re-train as need be)<p>6. Deploy model&#x2F;s<p>7. Monitor<p>8. Repeat steps 1 - 7<p>Yes, there are many tools that can help with each the above. Use whatever suits to automate it and make your job easier.

Maybe I’m a bit naive, but I’m convinced any great traditional software engineer or devops engineer in combination with a data scientist&#x2F;ML person should be able to setup the ops pipeline for a ML project. The details and algorithms themselves may look new and exciting, but operationalizing algorithms isn’t a totally new thing.

&quot;New fundamental science advances come out every week&quot;? While there&#x27;s certainly a lot being published, I think that the word &quot;fundamental&quot; is being abused in that sentence.

WhyLabs cofounder here, so my opinions are probably biased.<p>When it comes to MLOPs, data makes it much more complex to handle. Think of it as the curse of dimensionality. Nobody wants to deal with metrics across tens, if not hundreds or thousands of features. In addition, data is often not stored in nice SQL based system with strong schema enforcement, so we see data bugs creeping up all the time. An example is when an upstream API service returns the 9 digit zip code instead of the 5 digit one. This sort of data issues can creep up at many parts of the ML system, especially when you use JSON to pass data around.<p>You can defend against some of these problems with some basic devops monitoring, but when you deal with tons of features this becomes a tedious task. DevOps tools focus on solving problems around code, deployment and systems health. They are not designed to address the curse of dimensionality above, and you sacrifice a lot by trying to reduce data problems into DevOps signals.<p>To be fair, I don’t think we need some fancy algorithms, but I think we need tools that are optimized around the user experience (I.e. removing frictions) and workflows for these data specific problems. There’s a lot to learn and apply from the DevOps world to think about data health, such as logging and collecting telemetry signals.

Operationalizing ML is hard. But it has nothing to do with the models at all. It is hard because the main use-case (besides Images and Text processing) is feature fusion, you generate a bunch of distinct features about say People and their history and the products they like etc (thinking of a Recommender-System now). However, these are things that usually live in really distinct parts of your DB &#x2F; your backend. So as ML ops you are now tasked with getting info from all of these places. In a big org, often with different responsible people, security protocols etc....

I&#x27;m not sure there is anything extra messy about MLOps: there are lots of vendors in pretty much any area that has profit potential. If you put all vendors on a chart, it will look messy, but you aren&#x27;t going to be working with ALL of them (e.g., if you pick Snowflake you likely won&#x27;t also be working with RedShift, Databricks...). The messy part I guess is the evaluation&#x2F;selection, but not the integration or learning per se, as this article seems to imply. The article looks like a good reference for what&#x27;s out there though.

My frustration so far with mlops is how unnecessarily large containers need to be to serve even the smallest models. Want to serve an mnist pytorch model? It&#x27;ll likely be a huge dockerfile compared to model size, exceeding the capacity of most free-tier hosts

There&#x27;s a nice article by Martin Fowler about this topic: <a href="https:&#x2F;&#x2F;martinfowler.com&#x2F;articles&#x2F;cd4ml.html" rel="nofollow">https:&#x2F;&#x2F;martinfowler.com&#x2F;articles&#x2F;cd4ml.html</a>

I don’t think a bigger ecosystem equates to more complexity as directly as the author implies.<p>Ie suddenly having the option of two different databases doesn’t mean using one becomes twice as hard

&gt; You remember that one VC [..] always does some annual review of what’s hot in AI today. [...] So you check out his 2021 review<p>It takes all kinds, I guess.

since the ML hype is here to stay, it&#x27;s really good in my opinion because investors are constantly pouring more money into the industry

IMO &quot;MLOps&quot; is a &quot;DevOps&quot; problem, if you break it down fundamentally MLOps&#x27;s requirements are<p>* Computing resources (CPUs, Memory, Storage, GPUs)<p>* Distributed computing in most cases w&#x2F; spark + hadoop stack<p>* Keeping state, which may be required to mutate<p>* Rapid iteration<p>The ML tooling part of it is an implementation detail, i.e, the software and dependencies required. These are hard problems even with trad deterministic computing. I don&#x27;t seem to understand why the author seems to think ML engineers or scientists need to know these Ops tooling.<p>For example in this tweet <a href="https:&#x2F;&#x2F;twitter.com&#x2F;mihail_eric&#x2F;status&#x2F;1486750600343822343" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;mihail_eric&#x2F;status&#x2F;1486750600343822343</a> the author complains that data scientists need to learn kubeflow (they don&#x27;t), and that it&#x27;s complicated. Thing is, insofar as scalable architecture diagrams along with all the other security side-requirements it&#x27;s about as complicated as one would expect, maybe a little too abstract for those that do this for a living. I mean your typical k8s-based SaaS tech stack can reach that complexity, but it&#x27;s managed complexity about as complex needed for the stakes at play.<p>I don&#x27;t know if ML folk are in the peak hype cycle arrogance where they think global ops problems can be solved for their use case, or if there&#x27;s some misunderstanding on the iceberg of a problem of managing infra is.<p>I do agree it is messy, I did some ML Ops (w&#x2F;a big data stack) as a &quot;DevOps engineer&quot; but I stuck with k8s and infra primitives, filtering out most of the list. The ML aspect was the easy part, mainly managing the install deps, jupyter notebooks state etc., the hard part was scaling to manage costs, managing a big data stack in general, and making the entire flow UX friendly to ML engineers and data scientists, since you can&#x27;t expect them to learn new cli tools and trad software dev tooling (they&#x27;re paid too much to waste time not working on ML problems). I think a lot of these problems are solved if your company has a lot of money to burn on SaaS solutions or not care about scaling down, or being able to afford your own datacenter.<p>My counterpoint to the article is that the industry has bent backwards to cater to the ML space, integrating all these tools to existing tech (spark on k8s, kubeflow), making entire pipelines jupyter-driven (<a href="https:&#x2F;&#x2F;netflixtechblog.com&#x2F;notebook-innovation-591ee3221233" rel="nofollow">https:&#x2F;&#x2F;netflixtechblog.com&#x2F;notebook-innovation-591ee3221233</a>), and generally using massive amount of resources for ML. The ROI and massive push to burn resources and time into the tooling seems work out for big tech more than anyone.