Show HN: YoHa – A practical hand tracking engine

The demo really sells it here [1]. It&#x27;s amazingly intuitive and easy to use, it should be a part of video-conferencing software.<p>[1] <a href="https:&#x2F;&#x2F;handtracking.io&#x2F;draw_demo&#x2F;" rel="nofollow">https:&#x2F;&#x2F;handtracking.io&#x2F;draw_demo&#x2F;</a>

This is a GREAT website, I can understand what it does with zero clicks, zero scrolls.<p>Really great, congratulations, I hope that I can find a way to apply this lesson to my SaaS.

I&#x27;ve been working on a couple of chording keyboard designs and was thinking I might be able to create a virtual keyboard using this library. It would be nice to also be able to recognize the hand from the back. A keyboard would also obviously be necessary to track two hands at a time.<p>How does the application deal with different skin-tones?

Was wondering how easy it&#x27;d be to port to native mobile, so went looking for the source code, but doesn&#x27;t appear to actually be open source. The meat is distributed as binary (WASM for &quot;backend&quot; code and a .bin for model weights).<p>Aside from being a cool hand tracker, it&#x27;s a very clever way to distribute closed source JavaScript packages.

An &quot;undo&quot; gesture seems necessary, it was a bit too easy to accidentally wipe the screen. Aside from that, this is fantastic! Love to see what WASM is enabling these days on the web.

Hi, I&#x27;m not sure if you&#x27;ve looked into this or not, but another area that is interested in this sort of thing and might be very excited is musical gesture recognition.

What would be nice is a version that can be used to paint on the screen with your fingers, such that the lines are visible on a remotely shared screen. The use-case is marking up&#x2F;highlighting on a normal desktop monitor (i.e. non-touch) while screen-sharing, which is awkward using a mouse or touchpad (think circling stuff in source code and documents, drawing arrows etc.). That would mean (a) a camera from behind (facing the screen), so that the fingers can touch (or almost touch) the screen (i.e. be co-located to the screen contents you want to markup), and (b) native integration, so that the painting is done on a transparent always-on-top OS window (so that it&#x27;s picked up by the screen-sharing software); or just as a native pointing device, since such on-screen painting&#x2F;diagramming software already exists.

This looks great! Recently I&#x27;ve been wanting to make a hand-tracking library for video editing. I&#x27;d make a hand gesture like an OK with my index and thumb to begin recording, and when I was done I&#x27;d make a thumbs up to keep the take or thumbs down to delete a bad take. That way, I could very easily record stuff while only keeping the good takes, to sort out later.<p>Hell, the library could even stitch the takes together, omitting the times when my hand started&#x2F;finished doing the gestures.

This reminds me of TAFFI [0], a pinching gesture recognition algorithm that is surprisingly easy to implement with classical computer vision techniques.<p>[0] <a href="https:&#x2F;&#x2F;www.microsoft.com&#x2F;en-us&#x2F;research&#x2F;publication&#x2F;robust-computer-vision-based-detection-pinching-one-two-handed-gesture-input&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.microsoft.com&#x2F;en-us&#x2F;research&#x2F;publication&#x2F;robust-...</a>

Bit of feedback: the home page is pretty sparse. The video is great, but it wasn&#x27;t obvious how to find the repo or where to get the package (or even what language it can be used with). I had to open the Demo, wait for it to load, and then click the Github link there, and then the readme told me it was available on NPM.<p>Otherwise looks pretty impressive! I&#x27;ve been looking for something like this and I may give it a whirl

The demo doesn&#x27;t seem to work on my chromebook. Maybe it&#x27;s too underpowered?<p>Web page doesn&#x27;t say anything after `Warming up...` and the latest message in the browser console is:<p><pre><code> Setting up wasm backend. </code></pre> I expected to see a message from my browser along the lines of &quot;Do you want to let this site use your camera&quot;, but I saw no such message.

I wish there was a nice open-source model for tracking hands and arms with multiple viewpoints (multiple cameras), similar to commercial software like this: <a href="https:&#x2F;&#x2F;www.ipisoft.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.ipisoft.com&#x2F;</a>

Awesome.<p>Just note that in the demo video, the user is &#x27;writing&#x27; everything mirrored.

BTW this would be great for spaced repetition foreign character learning (Chinese, Arabic, Japanese, Korean, etc.): if the drawn figure is similar enough to the character the student is learning mark it as studied.<p>Congrats again

My first question is whether this has the capability of being adapted to interpret&#x2F;translate American Sign Language (ASL)?

Very impressive.<p>I want something like this so I can bind hand gestures to commands.<p>For example scroll down on a page by a hand gesture.

Could it be used to translate sign language signs to written or spoken words, I wonder.

Demo keeps crashing on iOS.

Great idea which is brilliantly executed.<p>So many educational uses, well done.

Would you provide the related paper to this approach?

Great work!

Great demo.

I think these tools are super interesting, but I tools like this marginalize users with non-standard number of limbs or fingers.