Successful machine learning models: lessons learned at Booking.com

&quot;Content Overload: Accommodations have very rich content, e.g.descriptions, pictures, reviews and ratings.&quot;<p>Laughed at that one. Booking.com is so full of dark patterns that I dread using it.

Experimental design is just a t-test? At least accord to that picture it seems that way. There are no ANOVA or interaction test?<p>Do websites usually just use t-test only? Like adding one feature at a time?

As an aside,<p>&gt; developing an organisational capability to design, build, and deploy successful machine learned models in user-facing contexts is, in my opinion, as fundamental to an organisation’s competitiveness<p>You hear that, right? In 2019 already you have to have AI and do it well to be competitive. I just wanted to point out how <i>cyberpunk</i> that is.

&gt; Model performance is not the same as business performance<p>This is interesting. Sometimes some people from business side consider that AI is the solution to all problems (as if there was just one catch-them-all AI solution) and some academic people think that the top-performance model for some classification task is the must-go, and all they forget that the goal is to earn money.

Nice to see that I am not only one hating booking.com with a passion.<p>But what really amazes me is the market failure that hotels and other accomodation providers can&#x27;t come up with a co-op booking site. I am sure there are issues that are difficult to solve from competition point of view, but are they <i>really</i> so difficult to solve that the rent seeking fees of current booking sites are justified?

&gt;&gt;<i>Booking.com go to some lengths to minimise the latency introduced by models, including horizontally scaled distributed copies of models, a in-house developed custom linear prediction engine, favouring models with fewer parameters, batching requests, and pre-computation and&#x2F;or caching.</i><p>Any idea what these are ? especially the pre-computation&#x2F;caching and batching. I&#x27;m not able to see what advantage does batching bring...or how you can really cache a prediction request

&gt; Once deployed, beyond the immediate business benefit they often go on to become a foundation for further product development.<p>This is one of the reasons I am a big believer in having a system to track model research and deployment lineage. (I personally use Domino Data Lab for this. I also work for Domino, but use it in my own modeling work and that of others I mentor.) No matter which system you use to track lineage, I&#x27;ve found it important to have a strict history of retraining, versioning, and experimentation. When models are used in downstream systems from the one they were originally intended, it becomes even more critical to able to explain and reproduce the &#x27;research&#x27; that led up to deployment.

I’m glad they highlighted inference latency. This is a big issue that I’ve started running into at scale.

Any other good resources on production level deep learning practices?