Hands-Free Coding: How I develop software using dictation and eye-tracking

Appreciate all the discussion!<p>I touch on this in the article, but I should say: I&#x27;m an edge-case when it comes to CTS. Most cases resolve spontaneously. Part of it is that my nerve dislocates when I bend my elbow, and this mobility causes a bunch of additional friction &#x2F; inflammation. This happens naturally to about 13% of the population, and mine is particularly pronounced.<p>All this to say, if you ever do start to get a burning or tingling in your elbows or wrists, or get numbness in the hands, it&#x27;ll likely go away on its own, or with conservative treatment (a physical therapist is a great first step!).<p>And, for those who also fall into my unlucky percentile, hopefully it helps to know that there are tools and a community that exist to let you keep working without needing to use your hands at all!

Not sure if this helps but I had a similar issue with my joints (arm, shoulder, wrist) where I would scream in pain when lifting a small book or anything above 3 lbs.<p>The issue was caused by my immune system acting up which would cause my tendons to become inflamed. After a year of misdiagnosis, Doctors found some heavy medication (methotrexate balanced with plaquenil) to help regulate it. One of the medicines side effects was listed as ‘death’ on the label.<p>After a year of experimenting, I found that major diet changes (lots more hearty greens, way less sugar and carbs, no caffeine or alcohol), improved sleep, reduced stress (quit my stressful job) completely alleviated my symptoms. I would still have flare ups from time to time which I reduced via physical therapy &#x2F; exercise (to strengthen muscles supporting my tendons).<p>Just sharing as I had a somewhat similar condition and was surprised that the fix didn’t have to be a pill.

Exploratory building of rich non-traditional (not necessarily handless) user interfaces is becoming increasingly accessible. For instance, here&#x27;s a web demo of Google&#x27;s MediaPipe&#x27;s face and iris tracking[1]. And hand tracking[2] with a downward-facing camera, permits enriching keyboard and touchpad&#x2F;tablets: annotating events with which finger was used, and where on a keycap it was pressed; hand and finger gestures in 2D and 3D; and positional touchless events. And speech to text... sigh.<p>But doing sensor fusion is hard. And strongly impacts system architecture. &quot;Launch the missiles&quot;... 1000 ms later... oh, nope, that was &quot;<i>Lunch is mussels</i> in butter&quot;. &quot;Spacebar keypress event&quot;... 50 ms later... &quot;btw, that was a thumb, at the 20% position&quot;. &quot;Ok, so 2000 ms ago, just before the foo, there was a 3D gesture bar&quot;. So app state needs to easily roll backward and forward, because you won&#x27;t fully know what happened now until seconds from now. Upside is traditional &quot;have to wait a bit until we know whether mumble&quot; latencies can be optimistically speculated away.<p>[1] <a href="https:&#x2F;&#x2F;viz.mediapipe.dev&#x2F;demo&#x2F;iris_tracking" rel="nofollow">https:&#x2F;&#x2F;viz.mediapipe.dev&#x2F;demo&#x2F;iris_tracking</a> (&quot;run&quot; button is top right) [2] <a href="https:&#x2F;&#x2F;viz.mediapipe.dev&#x2F;demo&#x2F;hand_tracking" rel="nofollow">https:&#x2F;&#x2F;viz.mediapipe.dev&#x2F;demo&#x2F;hand_tracking</a>

&gt; <i>I&#x27;ve heard that learning Vim can make this much more effective.</i><p>This doesn’t surprise me; Talon is a lot like Vim.<p>They’re both modal: Talon’s command mode roughly matches Vim’s normal mode, and its dictation mode roughly matches Vim’s insert mode.<p>Talon’s ordinals apply to Vim operations (normal mode) as well: operations can be preceded by a number which normally says how many times to do it. Talon’s “go left ninth” would be 9h or 9← in Vim. (h goes left one character.) “One zero third” is more cumbersome because in Vim it entails switching between modes (Talon and Vim modes don’t match precisely), but starting in normal mode, a1&lt;Esc&gt;3a0&lt;Esc&gt; works to append 1000 where the cursor is. (a to enter insert mode after the current position, 1 to type a 1, Escape to return to normal mode, 3 to say “do the next thing three times”, the next thing being appending a zero.)<p>Both Talon and Vim value this sort of consistency in being able to build bigger things from smaller pieces.<p>So I imagine Talon would work quite nicely with Vim, once you get talon to shift out of the way in some cases and let Vim handle those things. (It’s similar with browser extensions that let you interact with the browser Vim-style: when the page in consideration has extensive key bindings, especially Vim-like things like j&#x2F;k for down&#x2F;up, the extension actually gets in the way.) I expect you’d find that having used one made learning the other a good deal easier, since they’re working in just the same ways.

&gt; The biggest issue I&#x27;ve found so far is voice strain<p>My singing teacher started out as a voice therapist. He is still working with a lot of people with damaged voices that are there for therapeutic reasons as well as professional opera singers that there to level-up their game.<p>He says, the best thing you can do for your voice is to sing.<p>Singing will give you the proper &quot;support&quot; from your lungs and train the vocal muscles without wearing them out. One of the key aspects here is producing high quality vowels, and don&#x27;t let the sound be disturbed by consonants. This is opposite to how the voice is used when speaking (in most languages), where consonants are of primary importance for people to understand what you are saying. However, consonants are interruptions in the air-flow and can easily wear the voice out and cause stain.

I have suffered with thoracic outlet syndrome for almost 2 years. Countless physical therapy and deep tissue massages helped alleviate the pain but nothing seems to solve the root cause. I am 26 and the thought of not being able to use my hands to do something I have frightened the hell out of me.<p>That&#x27;s when I stumbled upon John Sarno and his book &quot;The mindbody prescription&quot;. I thought it&#x27;s pseudoscience and I read it with very low expectations. And It&#x27;s been almost four months since I have trouble using my computer.<p>In a nutshell, most of RSI injuries are not just physical problems and they are tightly coupled with your mind. The most important step is to acknowledge that subconscious thoughts related to pain and work along with it. Whenever I feel a tingling sensation, I yell at myself ( In my head ) that it&#x27;s all in my mind and I have started to feel normal. Of course, this might not work for everyone but definitely worth a try.<p>And I also do lot of strength training and climbing which really helped strengthening my hands

As a developer who also has cubital tunnel syndrome, having keyboards that places commonly use modifier keys [0] on your thumbs, such as Kinesis advantage or Ergodox, helps reducing the fatigue and symptoms immensely.<p>[0] it is called Emacs pinkies for a reason, and yes, my CTS is totally emacs’s fault as well.

I’ve been having similar issues lately, and after looking at talon, serenade, and caster, I ended up using caster [0]. The different programs all have significant differences in usability, and have clever ideas behind them, but unfortunately automatic speech recognition is still bad enough that the primary factor is which has a better ASR engine. Caster supports Dragon Naturally Speaking, which is expensive, but enough better to make it worthwhile.<p>There are moments where I think this is going to be the future of programming, since code as text is only the easiest for as long as typing is the easiest way to record things unambiguously.<p>But for the most part it’s still pretty frustrating. If ASR systems can get the sentence error rate down by an order of magnitude or two, I sincerely think this will take off not just for accessibility, but for normal use.<p>Until then, it’s a PITA that is saving my career nonetheless.<p>[0] <a href="https:&#x2F;&#x2F;caster.readthedocs.io&#x2F;en&#x2F;latest&#x2F;" rel="nofollow">https:&#x2F;&#x2F;caster.readthedocs.io&#x2F;en&#x2F;latest&#x2F;</a>

When I started my undergraduate degree, there was a PhD student everyone knew -- he looked like he ran the Linux User Group, which he did, he coded his own window manager, wrote games in Haskell etc.<p>At some point, he got some hand injury that meant he couldn&#x27;t type normally for months. He would only have been 21 or 22.<p>This scared me, so since then I&#x27;ve followed the standard advice on avoiding computer-related injuries very carefully. I use a desktop computer with an external, adjustable monitor. I use ergonomic keyboards, and use a mouse with either hand. I learned to touch-type with the Dvorak layout, I have an adjustable desk, etc.<p>I haven&#x27;t had any problems, and I don&#x27;t know which (if any) of these actions helped, but I&#x27;m surprised when I see other developers apparently happy to hunch over a laptop keyboard for 8 hours, while sitting on an awkward chair in a coffee shop. Why do this to yourself?

I don&#x27;t know why, but I&#x27;ve often thought about the situation the Josh finds himself in. IE - having to find a different way of building software if I couldn&#x27;t use a keyboard and mouse due to medical condition.<p>The solution I always imagine is paying someone or having my employer pay someone to strong style pair program with me. Perhaps a student, junior developer or even someone unfamiliar with software development entirely.<p>For those unfamiliar with strong style, this rule sums it up: &quot;For an idea to go from your head into the computer it MUST go through someone else&#x27;s hands&quot;. Like the standard driver &#x2F; navigator pair programming technique but with the navigator never touching the keyboard.<p>In the case of someone completely unfamiliar with software development I imagine that there would initially be a dramatic high &#x2F; low skill gradient between us, with the person essentially transcribing. However given the intensity of the practise I think this gradient would level out quite quickly.

The author links to this other project, which I’ve never heard of:<p><a href="https:&#x2F;&#x2F;serenade.ai&#x2F;" rel="nofollow">https:&#x2F;&#x2F;serenade.ai&#x2F;</a><p>He also references Tavis Rudd’s viral voice coding video, which is already 7 years old.<p><a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=8SkdfdXWYaI" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=8SkdfdXWYaI</a><p>The future is arriving slower than i would have guessed. I thought we’d be developing on the iPad using voice, gestures, and eye tracking by now.<p>Can I a least get “build and run” with my voice soon?

I love the eye tracking. Even without having any issues with my hands, I&#x27;d like to have quick mouse movements without having to take a hand off of the key board. Zooming in to have more precision is clever.<p>It seems the current generation systems aren&#x27;t working for Linux yet, but that gives me hope to have a way to work with it at some point in the future.

I personally have had carpal tunnel release surgery and bilateral rib resection for thoracic outlet syndrome. These surgeries fixed most of my issues but for the last couple of years I haven&#x27;t tolerated sitting well because of my back so I use a standing desk. I&#x27;ve thought about a computer interface that is the opposite of this article. Something that requires more non-repetitive physical movement so that you&#x27;re exercising while you program. Sure it would take longer to type but we all know working on a computer all day is bad for our health.

Thanks for this! Sorry it had to come about as a result of injury&#x2F;disease. Hopefully you&#x27;re not going to hurt your vocal tract -- are there precautions you take against this?<p>My interest currently is in voice-assistants and developing something like a personal assistant &#x2F; code-monkey &#x2F; grad-student.<p>I did have a stroke a few years ago, from which I&#x27;ve recovered, but it made me realize that almost all of my work, hobbies, aspirations, are tied to fairly extensive hand-eye integration. It&#x27;s good to start building a Plan B before I need it.<p>On a(nother?) side-note, my 88 year old dad still codes and his experiences have really shown me how bad we are at accessibility, as a developer community.<p>One final thing -- watching you work in this way highlights for me how low-level a lot of the stuff we do as developer is. There&#x27;s a certain amount of just text massaging that seems irrelevant to what we&#x27;re actually trying to get done (although it can be fun&#x2F;soothing&#x2F;aesthetic).<p>EDIT&gt; The last point is why I can&#x27;t give a fark about Vim&#x2F;Emacs&#x2F;whatever debates and obsessing. It seems like it might be fertile ground, but IMNSHO it&#x27;s a trap.

I have had an issue with dictation software... I want a dictation software to read me what it typed on screen. Right now the software expects you to check what it wrote and you have to necessarily keep looking at the screen.<p>Call me lazy but I wish for a method where I could lay back, speak a sentence, wait for transcribing, listen to what was written and ahead I go.

The eye tracking sounds like it could be incredibly useful as a 3rd input.<p>Very cool stuff, thanks for sharing

The eye tracking looks great, just as a regular additional way of using the mouse for things like focusing windows or tapping dialogs. I&#x27;m very tempted to get one of those.<p>Does anyone with experience of them know what the multi-monitor support is like? I have 3 screens side be side and one below, so quite a large range to cover that requires some head movement to go across all of them. Will it work with viewing over that wide of a space where I&#x27;m not always directly facing the primary screen?

If you&#x27;re using an eye-tracking mouse, you might enjoy Precision Gaze Mouse. Its a much faster way to click because you can skip the zoom in step. It uses eye tracking for large movements, and head tracking for small precision movements. I&#x27;m also a developer with an injury and I made it help myself, but several other people are using it now too.<p><a href="https:&#x2F;&#x2F;precisiongazemouse.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;precisiongazemouse.org&#x2F;</a>

I&#x27;ve been wanting something like Talon for years. Easily customized voice recognition for commands (you hear me say &#x27;blah&#x27;, you type&#x2F;start a program&#x2F;click a button labelled &#x27;blah&#x27;) is something I&#x27;ve not had since I was very young and tried to trick out LCARS x32 using only Windows XPs speech recognition software using the crappy stick mics that lived on top of every single beige CRT monitor at school.

I also bookmarked <a href="https:&#x2F;&#x2F;serenade.ai&#x2F;" rel="nofollow">https:&#x2F;&#x2F;serenade.ai&#x2F;</a> after seeing it here on HN. 10 years ago I shattered my hand and collarbone and used Dragon which sort of worked but was not specific to coding. These new tools look really neat.

Redox keyboard is amazing, never had any pain since starting to use one about 18 months ago.

Tavis Rudd, &quot;Using Python to Code by Voice&quot;, 2013: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=8SkdfdXWYaI" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=8SkdfdXWYaI</a> . Old but fun.

Since I don&#x27;t have any problems writing, I would be interested if there was some sort of system that was multi-modal in its working, or that maybe I could turn on dictation based coding with some specific rules for what I am doing at the moment.<p>For example if I am reading some code and I see<p>const updatedSortFields = Object.values(options).map(({ value: id, label }) =&gt; ({ id, label }));<p>If I could then say &quot;format line 130 multilines&quot; and it would turn it into something nicely indented over multiple lines.

Side comment, but relevant for anyone with wrist&#x2F;arm RSI reading this. I published a tactical guide that includes examples, references, and tactics to manage this injury.<p><a href="https:&#x2F;&#x2F;medium.com&#x2F;@robindji&#x2F;tactical-recovery-guide-for-wrist-arm-repetitive-strain-injuries-rsi-part-2-of-my-rsi-series-8f648bd55afe" rel="nofollow">https:&#x2F;&#x2F;medium.com&#x2F;@robindji&#x2F;tactical-recovery-guide-for-wri...</a>

The one major problem with these solutions is how expensive it all is. If you don&#x27;t find out about these programs BEFORE your finances get destroyed by health issues (both in cost of lost work time and in the medical care; most insurance will only cover SOME of the medical bills), you&#x27;ll never be able to afford the hardware and software to work around them.

&gt; In English, an ordinal number is one used to describe order, like &quot;fifth&quot; or &quot;ninth&quot; or &quot;three hundredth&quot;. In Talon, they&#x27;re used to repeat commands. If I wanted to go left by 9 spaces, I would say go left ninth.<p>Right! Like 9l in vim, but swapped.<p>I&#x27;ve always thought vim-like interfaces would be a good way to structure vocal input and I&#x27;m glad there are solutions operating in that space.<p>Edit: beaten to the punch :)

An alternative to eye tracking, that uses your webcam (move your head to move the mouse, smile to click):<p><a href="https:&#x2F;&#x2F;smylemouse.com" rel="nofollow">https:&#x2F;&#x2F;smylemouse.com</a><p>Demo: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=fg6Q3r2p_yE" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=fg6Q3r2p_yE</a>

This might be able to be a bit faster if you use a virtual QWERTY keyboard displayed with eye tracking acting as a cursor, with select being a consonant, preferably &#x2F;d&#x2F;. You can map certain phonemes for other shortcuts, including one for click&#x2F;right-click&#x2F;vim-commands.<p>Articulating words takes too long, even subvocalization is, which has the advantage of only using non-voiced phonemes.

This is really, really interesting -- I&#x27;ve often wondered about how the accessibility space is meeting the needs of disabled programmers, especially after I briefly lost my vision due to an eye injury.<p>The eye-tracking setup in particular looks really nice. Even if I stuck to a regular keyboard flow I feel like that would be incredibly useful in reducing strain.

The demo is incredible and heartening. I did not know the state of the art had improved so much. Thank you for sharing this, Josh.

as a dev with severe hand pain I would love more tools and help like this

Is there any way to write code on iPhone? Would a similar approach work or there is something much better?

Related:<p><a href="https:&#x2F;&#x2F;www.reddit.com&#x2F;r&#x2F;emacs&#x2F;comments&#x2F;42mn6y&#x2F;using_emacs_and_python_to_program_by_voice_awesome&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.reddit.com&#x2F;r&#x2F;emacs&#x2F;comments&#x2F;42mn6y&#x2F;using_emacs_a...</a>

&gt; Honestly, it&#x27;s just been such a relief to discover that my hands aren&#x27;t needed for me to do my work.<p>This describes my reaction exactly. I&#x27;m noticing my rsi getting better, which I&#x27;m grateful for, but the gift of not being so scared feels like the bigger deal.

Has anyone with joint pain tried accupuncture? Friend swears by it when she went to Tokyo, Japan to get it done a couple years back, 2018 if I recall correctly.<p>She recovered without taking any medication!<p>Edit: updated typo and extra info about not taking any medication.

For RSI folks, my experience is that RSI can “move” to other parts of the body pretty easily, including eyes and vocal chords. Somehow doing “work” with a part of the body can strain it, even when talking all day long socially wouldn’t.

Reminds me of the Figma GPT-3 Plugin<p><a href="https:&#x2F;&#x2F;mobile.twitter.com&#x2F;edleonklinger&#x2F;status&#x2F;1284251419172909057" rel="nofollow">https:&#x2F;&#x2F;mobile.twitter.com&#x2F;edleonklinger&#x2F;status&#x2F;128425141917...</a>

hands free and perl out loud here <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=Mz3JeYfBTcY" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=Mz3JeYfBTcY</a>

Seems like it would be a useful feature in voice recognition to have a &quot;whisper&quot; mode to prevent strain.

Reminds me of this story I encountered the other day. I have been on the same track, successfully too, for other types of pain (neck, shoulder and back pain). I post this here because this is an evidence based technique and think people should at least know that this is a possibility.<p>10&#x2F;23&#x2F;2008<p>&quot;Hi all,<p>I have been battling computer related RSI that runs from elbows to fingers for 10 years. It started 2 weeks into my first software programming job out of college, and has been a constant disability since. I went through ergonomics, biofeedback, drugs, acupuncture, physical therapy, 5 jobs, Carpal tunnel surgery, a year of rest, voice Software, foot pedals, hands free mice, you name it.<p>I finally gave up software and changed careers 2 years ago to a more Sales focused job and still couldn&#x27;t keep up with the typing. The only thing that has worked is having a full time computer assistant who sits with me and does what I say paired with a tablet PC that does handwriting recognition. Its dreadfully embarrassing in the workplace, slow, frustrating and just plain no fun. I got a rhythm though, got back on my feet and all went well for the last 2 years until a few months ago my throat began hurting (i talk all day now) and I was diagnosed with acid reflux.<p>With no arms and no voice, my future looks bleak and it has already been bad for almost a decade now.<p>I have officially given up on western medicine (my parents are doctors) and begun to explore alternatives. I stumbled on to Sarno and this newsgroup on Monday, read Sarno&#x27;s book today. Does it really work? I mean, I have had non-stop, permanent, 24&#x2F;7, life altering pain for 10 years. It has been my constant nemesis, battle, and focus.<p>It seems too good to be true.<p>I definitely have the personality traits, the onset of pain coincides with a rough first job out of college and my history of back spasms, shin splints and now reflux seems to fit the symptom imperative. I want to believe but can someone recover after this long of continuous computer pain?<p>-Dan&quot;<p>source: <a href="http:&#x2F;&#x2F;www.tmshelp.com&#x2F;forum&#x2F;topic.asp?TOPIC_ID=5187" rel="nofollow">http:&#x2F;&#x2F;www.tmshelp.com&#x2F;forum&#x2F;topic.asp?TOPIC_ID=5187</a><p>=====================<p>12&#x2F;12&#x2F;2008 (8 weeks later)<p>&quot;Hi,<p>I had 10 years of 24 hour a day, 7 days a week chronic pain in my fingers, wrists, forearms, arms, elbows from typing, mousing, and using a computer up until 12&#x2F;12&#x2F;2008 as a computer science student and as a software engineer. I believed I would live with it the rest of my life. It took me 8 weeks to fully recover.<p>I am an interesting case because I don&#x27;t have any major trauma&#x27;s in my childhood or daily life that led to TMS but just a generally stressful family and a high achieving, perfectionist attitude.<p>I grew up with a solid family unit, a very close identical twin brother, I was good at sports, school, had high self esteem. I went to college and was the captain of the track team. I found computer science and loved it. Things were on the up and up.<p>My parents got divorced when I was in college but it didn&#x27;t seem to bother me and transitioning to the working world. I was in a cubicle typing away and I didn&#x27;t like it but I &quot;sucked it up&quot; and did my job the best I could. I started getting pain while typing&#x2F;mousing and nothing ergonomic, or medical seemed to help. I plowed through the pain and kept working. I tried everything for 10 years. It was awful, bad, horrific and depressing all wrapped up in one. I finally gave up and left computers behind to be a project manager, and then a headhunter. But the pain was permanent and it still followed me no matter what job I did and how little I used a computer. My quality of life was pretty low. I found TMS 6-7 weeks ago and the recovery began.<p>I wrote an entry the day I found the TMS diagnosis and checked in along the way so the best way to learn my story is to read it below:<p><a href="http:&#x2F;&#x2F;www.tmshelp.com&#x2F;forum&#x2F;topic.asp?TOPIC_ID=5187" rel="nofollow">http:&#x2F;&#x2F;www.tmshelp.com&#x2F;forum&#x2F;topic.asp?TOPIC_ID=5187</a><p>I think I have a personality conducive to this type of pain defense mechanism. I am high-achieving, I have an intense and high achieving family, but I have a sensitivity to what others are feeling&#x2F;thinking that governs much of my decision making.<p>I think the key is reading enough information (posts, books, etc) that you are 90% sure that this is possible and could be your diagnosis and then get ready to do some serious crying. Its a personal journey that ends really well and it starts with writing down or talking about all the crap that has ever happened in your life. I started with when my twin brother got surgery when we were 4 years old - my earliest memory - and ended with my boss telling me that the phone system I installed sucks. And it had 1500 items in between. I feel flushed out now and clean and ready to tackle the world. This newsgroup has been helpful and the process is hard but I honestly believed I would have to live with this my whole life, and now I don&#x27;t. My phone number is at the end of my thread listed above. Call me, I would be happy to tell you about it live.<p>- dan&quot;<p>source: <a href="http:&#x2F;&#x2F;www.tmshelp.com&#x2F;forum&#x2F;topic.asp?TOPIC_ID=5301" rel="nofollow">http:&#x2F;&#x2F;www.tmshelp.com&#x2F;forum&#x2F;topic.asp?TOPIC_ID=5301</a>