Hey HN.<p>We’re Peter, Raza and Jordan of Humanloop (<a href="https://humanloop.com" rel="nofollow">https://humanloop.com</a>) and we’re building a low code platform to annotate data, rapidly train and then deploy Natural Language Processing (NLP) models. We use active learning research to make this possible with 5-10x less labelled data.<p>We’ve worked on large machine learning products in industry (Alexa, text-to-speech systems at Google and in insurance modelling) and seen first-hand the huge efforts required to get these systems trained, deployed and working well in production. Despite huge progress in pretrained models (BERT, GPT-3), one of the biggest bottlenecks remains getting enough _good quality_ labelled data.<p>Unlike annotations for driverless cars, the data that’s being annotated for NLP often requires domain expertise that’s hard to outsource. We’ve spoken to teams using NLP for medical chat bots, legal contract analysis, cyber security monitoring and customer service, and it’s not uncommon to find teams of lawyers or doctors doing text labelling tasks. This is an expensive barrier to building and deploying NLP.<p>We aim to solve this problem by providing a text annotation platform that trains a model as your team annotates. Coupling data annotation and model training has a number of benefits:<p>1) we can use the model to select the most valuable data to annotate next – this “active learning” loop can often reduce data requirements by 10x<p>2) a tight iteration cycle between annotation and training lets you pick up on errors much sooner and correct annotation guidelines<p>3) as soon as you’ve finished the annotation cycle you have a trained model ready to be deployed.<p>Active learning is far from a new idea, but getting it to work well in practice is surprisingly challenging, especially for deep learning. Simple approaches use the ML models’ predictive uncertainty (the entropy of the softmax) to select what data to label... but in practice this often selects genuinely ambiguous or “noisy” data that both annotators and models have a hard time handling. From a usability perspective, the process needs to be cognizant of the annotation effort, and the models need to quickly update with new labelled data, otherwise it’s too frustrating to have a human-in-the-loop training session.<p>Our approach uses Bayesian deep learning to tackle these issues. Raza and Peter have worked on this in their PhDs at University College London alongside fellow cofounders David and Emine [1, 2]. With Bayesian deep learning, we’re incorporating uncertainty in the parameters of the models themselves, rather than just finding the best model. This can be used to find the data where the model is uncertain, not just where the data is noisy. And we use a rapid approximate Bayesian update to give quick feedback from small amounts of data [3]. An upside of this is that the models have well-calibrated uncertainty estimates -- to know when they don’t know -- and we’re exploring how this could be used in production settings for a human-in-the-loop fallback.<p>Since starting we’ve been working with data science teams at two large law firms to help build out an internal platform for cyber threat monitoring and data extraction. We’re now opening up the platform to train text classifiers and span-tagging models quickly and deploy them to the cloud. A common use case is for classifying support tickets or chatbot intents.<p>We came together to work on this because we kept seeing data as the bottleneck for the deployment of ML and were inspired by ideas like Andrej Karpathy’s software 2.0 [4]. We anticipate a future in which the barriers to ML deployment become sufficiently lowered that domain experts are able to automate tasks for themselves through machine teaching and we view data annotation tools as a first step along this path.<p>Thanks for reading. We love HN and we’re looking forward to any feedback, ideas or questions you may have.<p>[1] <a href="https://openreview.net/forum?id=Skdvd2xAZ" rel="nofollow">https://openreview.net/forum?id=Skdvd2xAZ</a> – a scalable approach to estimates uncertainty in deep learning models<p>[2] <a href="https://dl.acm.org/doi/10.1145/2766462.2767753" rel="nofollow">https://dl.acm.org/doi/10.1145/2766462.2767753</a> work to combine uncertainty together with representativeness when selecting examples for active learning.<p>[3] <a href="https://arxiv.org/abs/1707.05562" rel="nofollow">https://arxiv.org/abs/1707.05562</a> – a simple Bayesian approach to learn from few data<p>[4] <a href="https://medium.com/@karpathy/software-2-0-a64152b37c35" rel="nofollow">https://medium.com/@karpathy/software-2-0-a64152b37c35</a>
This looks pretty great, though the SaaS model is an absolute non-starter for my own usage unfortunately. We've been pretty prolific users of Explosion AI's (Makers of SpaCy) Prodigy [1] and actually the interfaces look very similar. What would you say the core differences are between Humanloop and Prodigy?<p>1: <a href="https://prodi.gy/" rel="nofollow">https://prodi.gy/</a>
<a href="https://jacobbuckman.com/2020-01-17-a-sober-look-at-bayesian-neural-networks/" rel="nofollow">https://jacobbuckman.com/2020-01-17-a-sober-look-at-bayesian...</a><p>"But in practice, BNNs do generalize to test points, and do seem to output reasonable uncertainty estimates. (Although it’s worth noting that simpler approaches, like ensembles, consistently outperform BNNs.)"
It looks awesome!<p>I see the snorkel logo on the website and they recently also launched snorkel flow for data annotation and model training. There isn't much detail on that, but I wonder is there any advantage humanloop has over that?<p>On the same track, prodigy also has a prodigy team version that is being ready for launch forever. So glad you guys are few steps ahead.<p>I am also building a labeling interface myself because I couldn't find the right product for my needs(I have tried tools like label studio, doccano, prodigy, dataturks and ml annotate). They just miss one thing or the other. I really wish there is one place that features like HTML support, hierarchical labels, active learning, batch labeling, project tracking, multi user management and most important the UI/UX are all well put together.
Is this something we will be able to buy and run on our servers? I don't think we're the only ones wary of working hard to develop IP for a different company.<p>Also predictions/month pricing is just really challenging and incompatible with many downstream business models. The value has to be really huge to justify that.
Neat how do you compare yourself on the annotation capabilities with Datasaur.ai which launched in the last YC batch?<p>In terms of training the models for deployment -- do we own the artifact? Can I move that into my own model repository?<p>Also how do you feel this compares to using fine tuning on a publicly available BERT family model which is already fairly fast and easy not requiring a huge corpus, speaking from experience of recently having done so?<p>Are the benefits more from the tight feedback loop and already standing infrastructure?
#1 and #2, if they work as advertised, are great features but a lot of other companies claim to do this but have failed.<p>One of the biggest problems I have is image annotation using CVAT - the tool works when the task is simple annotation but outputting the annotation data and integrating it has been a pain-point. Also CVAT has a tool is great but has a lot of missing features :/
This workflow is still so complex to get right. Really excited to see more tools for it and try it out ourselves!<p>At visitorX we're building a fairly large bank of comments and a tagging system and Humanloop looks really great for that.