Organising labelling is a core part of productionised ML and the Domain Team has the most knowledge to do it. But are they the best suited to do it?<p>Our experience shows that because incentives differ in production and off-line labelling, the Domain Team might create biased (subjective) dataset in Human-In-The-Loop settings.<p>Adding an extra team that was trained by the Domain Team but can only influence end-to-end performance through the deployed model helped to keep the training dataset objective while improving the productionised pipeline and helping the Domain Team.<p>First time HN poster, please share your recommendations in the comments. We are working on a "Machine Learning Product Manual" ebook about best practices to productionise and operate ML from a product perspective. Follow me on twitter at <a href="https://twitter.com/xLaszlo" rel="nofollow">https://twitter.com/xLaszlo</a> for updates.