From my experience "single prompt classification" isn't as simple as "type in sentence and it works" in practice. But you can use some methods to massively improve it's consistency/output.<p>I cannot recommend guidance enough. You can use shockingly small Llama models for some tasks with guidance while only actually generating a handful of tokens.<p>You should highly consider some form of guidance/logit bias for classification especially if you have a known set of classes. This will ensure you get it in the format that you want, with the correct classes that you want.<p>Keep in mind LLMs perform much better with COT. So you make it explain what the text/image is, then explain the possible classifications, then list its final decision. Again guidance can ensure it follows the correct format to do this.<p>LLM's still massively benefit from finetuning, especially if you want too classify it in a particular format. Notebook tags vs SFW/NFSW vs important subjects, etc. Existing alignment can sometimes mess with some of these classifications too which finetuning helps smooth out.
Yeah totally agree. We've found that a ton of OpenAI usage in practice is a variant of either classification or information extraction. This makes sense -- going from a human-native form of information (free text) to a computer-native form of information (structured data) is a key component of many pipelines!<p>Of course, GPT-4 is insanely expensive to use at scale, and still isn't a perfect classifier. So the next step is to take the outputs you get from GPT-4 and use them to fine-tune a smaller model that's really fast and good at your specific problem. In my experience, even <i>without</i> using any human annotations or online learning, a model fine-tuned just on GPT-4 outputs can actually <i>outperform</i> GPT-4 as a classifier! This seems really counterintuitive at first, but my guess is what's happening is that the training process is a kind of regularization, so the weird mistakes GPT-4 occasionally makes are overwhelmed in the training data by all the times when GPT-4 gets it right.<p>As a disclaimer, we're building open source tooling to ease the transition from prompt to cheaper fine-tuned model at my company OpenPipe.
Totally agree that this is under appreciated.<p>Between open source modeling tools being incredible, transfer learning allowing dirt cheap fine-tuning and now mega-models being able to instantly give you a "mostly right" data set, the cost of creating ML features has dropped to almost nothing.<p>Products that took quarters/years and required big budgets for labeling, ML specialist, GPUs etc just a few years ago can now be done in an hour or so for free (if you are scrappy). I imagine this is going to lead to a ton of great ML features that weren't worth funding in the past but are very valuable in aggregate. Similar to the mid-2000s when the cost/ease of web development came down enough that there was a lot more experimentation and fun to be had.
Agreed! I think it's going to be hard for software developers to adjust to the "data science/engineering" mindset of monitoring and iterating on the long tail of maintenance. A lot of teams already have this issue with deterministic code running in production. I think there's a big opportunity to help purely software teams to learn and adjust.
Honestly, this was one of the first things that excited me with chatgpt. I'm really eager to see a high performance inference engine that can keep up with my log data.<p>Being able to teach an AI assistant to look for specific (but not too specific) things with just a prompt would be incredibly helpful.
Another thing you can do with LLMs that I think is pretty interesting is use them to train a cheaper and faster model. Then use the faster model in your application.
We’re doing this pretty successfully to identify products from massive text content, and even more importantly, we then perform a second pass and let the models categorise the identified products, and <i>then</i> do a third pass to build a category hierarchy. This gets us a full product taxonomy with practically no sweat. It’s amazing, really.
I couldn't agree more. I talked a bit about how amazing and magical LLM-based text classification is here: <a href="https://blog.superb-owl.link/p/the-shapes-of-stories-with-chatgpt/" rel="nofollow noreferrer">https://blog.superb-owl.link/p/the-shapes-of-stories-with-ch...</a>
You can use Particlesy to create a custom GPT-4 bot trained just to classify csv row data and integrate systems.<p>One interesting use case we see is a SaaS company using the our REST API to access a Particle with custom instructions just for integration with other systems. They will provide a CSV row and the GPT-4 model will classify and map the columns into their key columns. In effect, they are able to integrate with almost any system in their vertical with an out-of-the-box integration. Albeit, it is more expensive, but it is great for the initial trial phase and the costs can be passed to the customer. <a href="https://www.particlesy.com" rel="nofollow noreferrer">https://www.particlesy.com</a>
I'm waiting for a well-optimized LLM-based system built-into a local editor like obsidian and I can ask it scan my entire local Documents folder and then it supercharge my reading/writing locally.
We've found the same. A lot of usage through our LLM Categorization endpoint. The toughest problem was actually constraining the model to only output valid categories and not hallucinate new ones. And to only return one for single-classification (or multiple if that's the mode).<p>[0] <a href="https://matt-rickard.com/categorization-and-classification-with-llms" rel="nofollow noreferrer">https://matt-rickard.com/categorization-and-classification-w...</a>
With everyone talking about LLMs being glorified autocomplete, I actually would like to see how well they perform as autocomplete. Because most built-in ones are pretty bad.
There is a software company in the UK that is performing medical-jargon-to-simple-English translation on text emitted from speech recognition.<p>They achieve this via a fairly complex set of regular expressions, which must represent a significant time investment to research and maintain.<p>The same effect can be achieved with a properly prompted GPT-4 completion request, which took about five minutes to write.
The nice thing about active learning with a classic ML model is, everytime you annotate a data point, the model learns.<p>How do you update your prompt to take the new data point into account? Or do you just add it as an example inside the prompt and let it grow?