Hey HN community -<p>I’m Ivan from Datasaur (<a href="https://datasaur.ai/" rel="nofollow">https://datasaur.ai/</a>) - we build software to allow humans to more efficiently label data for training natural language processing (NLP).<p>NLP algorithms are being trained in a wide variety of industries - from customer service to legal contracts, forum moderation to restaurant reviews. All these algorithms benefit from recent breakthroughs in academia and a generous open-source community. However, in order to be deployed to the real world, they require a custom set of training data to learn and understand the language unique to each industry. Therefore, people around the world are meticulously labeling data samples.<p>Example sentence: <i>London is the capital and largest city of England and of the United Kingdom.</i><p>Labels: “London” —> “capital”, “United Kingdom”<p>Labels: “London” —> “largest city”, “England”<p>In the last few years I’ve worked at companies such as Apple and Yahoo and noticed that many organizations tend to reinvent the wheel when creating labeling interfaces for their labelers. Some companies still do this work in Excel.
We saw an opportunity to create a "single interface to rule them all" - to handle all sorts of text labeling tasks.<p>We leverage existing NLP capabilities to intelligently validate the quality of labels in a document and complement human judgment. Furthermore, we already understand terms like “Starbucks” and “New York” - why spend time labeling these terms from scratch every time? We created an API so you can plug in existing models to apply a first pass on labeling the document. We also built many other extensions to help labelers optimize their time - a “find and label” extension for labeling repetitive terms, a dictionary extension for quickly looking up unfamiliar terms. We spent the past year building out the labeling solution I wish I could have used.<p>We now handle named entity recognition, parts of speech, document labeling, coreference resolution (multiple words referring to the same object/person) and dependency parsing (drawing relationships between words). A case study with one of our clients shows 70% improved labeling efficiency upon adopting the Datasaur platform, and we have much more room to improve.<p>We also spoken with 100+ AI teams globally and identified the best practices in labeling. In addition to providing an enhanced interface, we can help track labeler performance, peer disagreement scores, and detect/remove labeler bias. By incorporating and encoding these features into our software, we can not only help improve the labeling efficiency but also improve the quality of the data and therefore the resulting AI model.<p>We believe that as AI becomes ever more prevalent and ubiquitous, labeling will become an increasingly important task. AI is a garbage-in, garbage-out technology, and the quantity and quality of data can often make a critical difference in the resulting AI model. We’re really excited to open Datasaur up to the world today and hear your feedback. Have you run into similar labeling issues? What tips and tricks have you employed to keep up with AI’s voracious appetite for data? We’d love to hear how you’ve tackled data labeling at your own companies. Thanks so much in advance!<p>Ivan
It resembles an open source annotation tool that has existed for years. <a href="https://brat.nlplab.org/" rel="nofollow">https://brat.nlplab.org/</a><p>It doesn't include a ML assistant though.<p>We have built a semi-automated annotation tool for our internal use too. ML models help classify documents and extract named entities by making suggestions. Sometimes I'm thinking of spinning it off as a standalone product but not sure how big the market would be.
Looks great, Ivan. Congrats!
If I understand correctly you would use Spacy and some pre-trained models to validate human labels. Now my question is that what is the point of collecting labels for ML training if we already have a valid model for the same task that can complement human labels?
Could you please elaborate on what you mean by "intelligently validate the quality of labels in a document and complement human judgment", and discuss your methodology?<p>This seems to operate under the assumption that human labels are not actually the ground truth. I understand that they can be dirty, but most unsupervised approaches aren't producing a ground truth, either. So, are you saying it's better to have multiple pretty good sources of truth instead? Because depending on the application, that might make sense or it might be like trying to start a farm with a dead horse and a dead cow.
Looks interesting, signed up to try this out and see if it might deal with some of our labelling.<p>A few notes -<p>Some help docs would be good, or better links to them. You specify a few types of projects but don't really explain what they are - I tried searching for "constituency" type projects but I have no idea what they are still.<p>You're sending error messages to the frontend. "Cannot read property 'startsWith' of undefined" is not something that should be reaching an end user, and this is happening unreliably when I upload files.<p>If I upload a CSV file I cannot seem to do NER from "new project". NER specifically chosen supports TSV but not CSV. My TSV files just say "server error", though they load as just txt files.<p>What's a question set? What's the format you need from me (it just says "csv").<p>Autolabelling seems to do nothing. Do you have example text where that should work?<p>I can navigate the text but the hotkeys for labelling don't do anything until I've already clicked once.<p>Search & label all is interesting but doesn't seem to give me any labelling options. Also the regex search for "someword \w" just returns all two words next to each other which seems wrong to me.<p>Congrats on the launch!
This looks awesome! Waiting for my email confirmation.<p>I was looking for information about where my data has to be hosted to use this service and could not find it. Will there be some more information about how this data is handled once I get past the login? Thanks!
In the spreadsheet view, do users need to upload labels as a text file to then assign them to items? I work with Quill.org, a nonprofit edtech tool that helps students improve their writing skills, and we do a lot of labeling work now where we may need to assign say one of 20 labels to 1,000 responses at a time. I uploaded some sample data, but didn't understand how I could quickly assign labels to my content. Please let me know if I'm missing something here.
Congratulation for the launch!<p>To understand the scope of your work a little bit, if I have Prodigy with custom labeling needs set up for me, do I still benefit from switching to datasaur?
This looks wonderful, will definitely try it out. We ran into the labeling issue when doing NER a couple years ago on Reddit books dataset. If only this existed then.
Wish you good luck, the website looks clean, the product idea is good:) You request an image however which width is 3000+ pixels: <a href="https://s.datasaur.ai/static/media/homepage-hero.4917b8af.png" rel="nofollow">https://s.datasaur.ai/static/media/homepage-hero.4917b8af.pn...</a> . 1200px in width should be enough, I would resize the image, it slows down the page.
Any chance you could support HTML files? We've been using <a href="https://www.tagtog.net/" rel="nofollow">https://www.tagtog.net/</a> for some of our data labeling / annotations needs but their tool for these file types is still "experimental".
LinkedIn suggested a post from you a couple weeks ago and I remember thinking “what’s Ivan up to?” and I saw Datasaur. Congrats on YC! I know that our time at Yahoo was a brief overlap but I remember the swirl of ML, Knowledge Graph and labelling our org was at 5 years ago.<p>Good luck with Datasaur!<p>-
Julio Nobrega
Datasaur looks awesome! Can't wait to try it out. Congrats on the launch!<p>Curious about data security and privacy? How do you guarantee privacy? Is there some cryptography or secure enclaves used? Some sets of documents (and email) are super high trust.<p>Guessing the on-prem version is probably safest route
I've got a question, a lot of startup websites have a similar look to this one. It's a look I actually really like. What technologies are they all using? How would I build a site like this?<p>wappalyzer doesn't give me anything and I don't have a ton of webdev experience.
On the pricing page, the Growth box shows a checkmark for "Unlimited labels" but right below in the "Choose the right plan for you", the Growth plan says the number of labels is 10,000,000.
Great idea - and this is an odd comment: I think you're pricing it too low relative to the number of people in the market. Just my gut - could well be wrong, wrong, and wrong again.
Hey Ivan, this looks great! What are the privacy implications for my data that I want to label with your tool? I’m assuming I upload it to your servers?