Transformers.js – Run Transformers directly in the browser

This library is so cool. It makes spinning up a quick demo incredibly easy - I&#x27;ve used it in Observable notebooks a few times:<p>- CLIP in a browser: <a href="https:&#x2F;&#x2F;observablehq.com&#x2F;@simonw&#x2F;openai-clip-in-a-browser" rel="nofollow">https:&#x2F;&#x2F;observablehq.com&#x2F;@simonw&#x2F;openai-clip-in-a-browser</a><p>- Image object detection with detra-resnet-50: <a href="https:&#x2F;&#x2F;observablehq.com&#x2F;@simonw&#x2F;detect-objects-in-images" rel="nofollow">https:&#x2F;&#x2F;observablehq.com&#x2F;@simonw&#x2F;detect-objects-in-images</a><p>The size of the models feels limiting at first, but for quite a few applications telling a user with a good laptop and connection that they have to wait 30s for it to load isn&#x27;t unthinkable.<p>The latest releases adds binary embedding quantization support which I&#x27;m really looking forward to trying out: <a href="https:&#x2F;&#x2F;github.com&#x2F;xenova&#x2F;transformers.js&#x2F;releases&#x2F;tag&#x2F;2.17.0">https:&#x2F;&#x2F;github.com&#x2F;xenova&#x2F;transformers.js&#x2F;releases&#x2F;tag&#x2F;2.17....</a>

I’m using this library to generate embeddings with gte-small (~0.07gb) and using Upstash Vector for storage.<p>It’s only 384 dimensions but it works surprisingly well with a paragraph of text! It also ranks better than text-embedding-ada-002 on the leaderboard<p><a href="https:&#x2F;&#x2F;huggingface.co&#x2F;spaces&#x2F;mteb&#x2F;leaderboard" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;spaces&#x2F;mteb&#x2F;leaderboard</a>

The Syntax podcast recently did an episode on Transformers.js and the developer…<p><a href="https:&#x2F;&#x2F;syntax.fm&#x2F;show&#x2F;740&#x2F;local-ai-models-in-javascript-machine-learning-deep-dive-with-xenova" rel="nofollow">https:&#x2F;&#x2F;syntax.fm&#x2F;show&#x2F;740&#x2F;local-ai-models-in-javascript-mac...</a>

transformers.js is such a cool library.<p>I made a small web app with it that uses it to remove backgrounds from images (with BRIA AI&#x27;s RMBG1.4 model) at <a href="https:&#x2F;&#x2F;aether.nco.dev" rel="nofollow">https:&#x2F;&#x2F;aether.nco.dev</a><p>The fact you don&#x27;t need to send your data to an API and this runs even on smartphones is really cool. I foresee lots of projects using this in the future, be it small vision, language or other utility models (depth estimation, background removal, etc), looks like a bright future for the web!<p>I&#x27;m already working on my next project, and it&#x27;ll definitely use transformers.js again!

This is super cool, but unfortunately it also seems super impractical. Models tend to be quite large, so even if a browser can run them, getting them to the browser involves either:<p>1. Large downloads on every visit to a website.<p>2. Large downloads and high storage consumption for each website using large models. (150 websites x 800 MB models =&gt; 120 GB of storage used)<p>Both of those options seem terrible.<p>I think it might make sense for browsers to ship with some models built in and be exposed via standardized web APIs in the future, but I haven&#x27;t heard of any efforts to make that happen yet.

Ok so now we can make a browser plugin which will pick out all bicycles or bridges in a Google captcha, right?

Is training not possible? I did some stuff years ago where I create and train small NNs in the browser and I&#x27;m curious if that type of thing would work better today with a small custom transformer.

Also this opens the possibility of running these models on Node.js serverless functions no?<p>That certainly also has to open up possibilities for on-demand predictions?

Does it support Apple Silicon (accelerated)?

can someone explain what this means I can do with it if I know vanilla JavaScript?<p>for example, I used the image upscaling playground on hugging face all the time. But I do it manually here: <a href="https:&#x2F;&#x2F;huggingface.co&#x2F;spaces&#x2F;bookbot&#x2F;Image-Upscaling-Playground" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;spaces&#x2F;bookbot&#x2F;Image-Upscaling-Playgr...</a><p>would transformers.js allow me to somehow executive that in my own local or online app programmatically?