This is amazing content marketing, similar to Backblaze hard drive stats.<p>It was well-written and very informative. I enjoyed reading it and learned something new and potentially useful. And it brought attention to a potential pain point (trying to train and infer models on multiple platforms) and suggested their product to help(roboflow). It got the marketing message across without being irritating or obtrusive. A win-win situation for both reader and company.<p>Again, great writeup. I wish more content marketing would be this engaging and useful.
The thing with evaluating these services is that business people or product managers don’t understand that accuracy on benchmarks doesn’t map well to accuracy on <i>your</i> problem. If you have a wacky data distribution (hint: you do), say like a collection of photos all taken inside the same building, you can’t generalize to consider benchmark stats.<p>On top of this, these services charge you for usage, not for <i>accurate</i> usage. This is a major issue that so many people overlook.<p>What does it matter if it’s cheap per request? That only helps you if accuracy per request is very high. Otherwise you’re paying for cheap garbage, or in some use cases you must grow requests much larger to overcome errors per request, like if you’re trying to get a bulk of labeled data via these services.<p>Most use cases are still better off hiring ML engineers who understand how to evaluate accuracy for the unique business use case, fine tune or train a model, and do it in house (or at least give you deep assurances of the rare cases when the big cloud services actually are cost effective to use).<p>For most use cases, you’re wasting your money trying for ML-as-a-service like this.
Biggest surprise for me was that they can’t train on COCO. Would have thought their performance would be a major part of the marketing considering how prominent that benchmark is in the research community.