I’ve been doing something similar to control racing video games while biking on an indoor exercise bike. I convert pedal speed into brake and acceleration keys (there’s some middle amount of pedaling that’s neutral) with pynput. I used OpenCV for webcam input and MediaPipe’s models to convert lean angle into left and right keyboard presses, and head tilt into up and down. It’s really fun but the slight latency in recognizing my pose is tough for fast courses. Now I’m thinking about some Arduino with a sensor or low res IR camera or something else that could detect changes a lot quicker.
Reminds me of Pigeon Sim, using a Kinect and the Google Earth browser plugin (which is sadly now discontinued).<p><a href="https://www.youtube.com/watch?v=w5ye9dfeu7c">https://www.youtube.com/watch?v=w5ye9dfeu7c</a><p><a href="https://github.com/jawj/pigeonsim">https://github.com/jawj/pigeonsim</a>
Considering his last (brilliant) project featured here, the semaphore keyboard madness, I'm not sure how serious this is but it seems there is potential for providing a useful API for AR/VR apps/games/experiences; I'm not sure the body is extensively served by Quest etc beyond hand and head tracking.
This reminded me of the full body keyboard controller someone else shared a few weeks back...<p>My ideas for body games are growing<p><a href="https://github.com/everythingishacked/Semaphore">https://github.com/everythingishacked/Semaphore</a>
That's interesting, I always wanted to try something similar but put it off. I'll try it with my phone & see if I can get it working in small space but with ulta-wide camera.