This library is so cool. It makes spinning up a quick demo incredibly easy - I've used it in Observable notebooks a few times:<p>- CLIP in a browser: <a href="https://observablehq.com/@simonw/openai-clip-in-a-browser" rel="nofollow">https://observablehq.com/@simonw/openai-clip-in-a-browser</a><p>- Image object detection with detra-resnet-50: <a href="https://observablehq.com/@simonw/detect-objects-in-images" rel="nofollow">https://observablehq.com/@simonw/detect-objects-in-images</a><p>The size of the models feels limiting at first, but for quite a few applications telling a user with a good laptop and connection that they have to wait 30s for it to load isn't unthinkable.<p>The latest releases adds binary embedding quantization support which I'm really looking forward to trying out: <a href="https://github.com/xenova/transformers.js/releases/tag/2.17.0">https://github.com/xenova/transformers.js/releases/tag/2.17....</a>
I’m using this library to generate embeddings with gte-small (~0.07gb) and using Upstash Vector for storage.<p>It’s only 384 dimensions but it works surprisingly well with a paragraph of text! It also ranks better than text-embedding-ada-002 on the leaderboard<p><a href="https://huggingface.co/spaces/mteb/leaderboard" rel="nofollow">https://huggingface.co/spaces/mteb/leaderboard</a>
The Syntax podcast recently did an episode on Transformers.js and the developer…<p><a href="https://syntax.fm/show/740/local-ai-models-in-javascript-machine-learning-deep-dive-with-xenova" rel="nofollow">https://syntax.fm/show/740/local-ai-models-in-javascript-mac...</a>
transformers.js is such a cool library.<p>I made a small web app with it that uses it to remove backgrounds from images (with BRIA AI's RMBG1.4 model) at <a href="https://aether.nco.dev" rel="nofollow">https://aether.nco.dev</a><p>The fact you don't need to send your data to an API and this runs even on smartphones is really cool. I foresee lots of projects using this in the future, be it small vision, language or other utility models (depth estimation, background removal, etc), looks like a bright future for the web!<p>I'm already working on my next project, and it'll definitely use transformers.js again!
This is super cool, but unfortunately it also seems super impractical. Models tend to be quite large, so even if a browser can run them, getting them to the browser involves either:<p>1. Large downloads on every visit to a website.<p>2. Large downloads and high storage consumption for each website using large models. (150 websites x 800 MB models => 120 GB of storage used)<p>Both of those options seem terrible.<p>I think it might make sense for browsers to ship with some models built in and be exposed via standardized web APIs in the future, but I haven't heard of any efforts to make that happen yet.
Is training not possible? I did some stuff years ago where I create and train small NNs in the browser and I'm curious if that type of thing would work better today with a small custom transformer.
Also this opens the possibility of running these models on Node.js serverless functions no?<p>That certainly also has to open up possibilities for on-demand predictions?
can someone explain what this means I can do with it if I know vanilla JavaScript?<p>for example, I used the image upscaling playground on hugging face all the time. But I do it manually here: <a href="https://huggingface.co/spaces/bookbot/Image-Upscaling-Playground" rel="nofollow">https://huggingface.co/spaces/bookbot/Image-Upscaling-Playgr...</a><p>would transformers.js allow me to somehow executive that in my own local or online app programmatically?