AI headphones let wearer listen to a single person in a crowd by looking at them

If the size could shrink to the size of a small earplug, I&#x27;d love to use this as a person who is not hearing-impaired <i>(at least they couldn&#x27;t diagnose me with it, so now I&#x27;m not sure if their diagnostics sucks, or I&#x27;m just a normal person and others pretend better that they hear everything well)</i>.<p>In groups and with friends, it&#x27;s inevitable that you end up in a busy restaurant or a bar, and it always frustrates me that I don&#x27;t hear something, I ask the person to repeat only to not hear it again, usually because they repeat it at the same low level (considering the circumstances). Missing jokes and throwaway comments is even worse (&quot;hey what are you all laughing about, I didn&#x27;t hear it, could you repeat it for me like three times until I hear it&quot;).

One thing that the HN crowd should appreciate is just how expensive and shit hearing aids are.<p>go and look the up the price, they are deeply expensive, even for basic &quot;make it louder&quot; type aids.<p>Worse still, because they interfere with your ear, you tend to loose the ability to &quot;steer&quot; your hearing. This means that you can&#x27;t tune out other conversations&#x2F;noises or stuff.<p>The one good side effect of facebook spending billions on its (probably) futile search for practical and popular AR is <a href="https:&#x2F;&#x2F;www.projectaria.com&#x2F;glasses&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.projectaria.com&#x2F;glasses&#x2F;</a><p>Which is a (cheap) platform to do experimentation for AR type actions.<p>However it has eye tracking, microphone array and front facing cameras, so it can be fairly easily modified into being a steerable microphone.

As somebody who is hearing impaired, a feature like this would be a Godsend for me! This feature should be integrated into hearing-aids ASAP! Shut up - no, actually - keep talking and take my money!

This but more advanced would quite nicely help with my tinnitus. I hear fine when one person is speaking (even softly and at a distance), but multiple or with music, I hear nothing.

Code: <a href="https:&#x2F;&#x2F;github.com&#x2F;vb000&#x2F;LookOnceToHear">https:&#x2F;&#x2F;github.com&#x2F;vb000&#x2F;LookOnceToHear</a>

I&#x27;ll bet they achieve commercial success with the reverse application. Imagine being able to mute that one obnoxiously loud person with an annoying voice at a party!

I used to work at Sonos, long before their current app update debacle and headphone debut.<p>During the first aborted product effort to develop headphones, we were looking at a conceptual feature similar to this - selectively allowing people’s voices through the ANC chipset.<p>I don’t recall the exact approach the DSP folks were using (I was closer to the hardware for ANC) but they were really only able to figure out how to isolate the wearer’s voice by virtue of that signal having more power than all the others.<p>This is terribly cool. I wonder what other kinds of fun you could have with headphones. ANC chipsets are incredibly powerful and I’d wager their capabilities are not even close to fully tapped.

The open source code is at <a href="https:&#x2F;&#x2F;github.com&#x2F;vb000&#x2F;LookOnceToHear">https:&#x2F;&#x2F;github.com&#x2F;vb000&#x2F;LookOnceToHear</a> and the research paper is at <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2405.06289" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2405.06289</a><p>So perhaps this is not as out of reach as many pop-science articles. I’d love to hear if anyone is able to get this working independently.

This could actually be really helpful to me, as I have trouble hearing someone speaking in a busy room because my mind is trying to pick up everything (I think this is because of my ADHD). Having a way to significantly quiet out other noises aside for the voice of the person I&#x27;m speaking with would be amazing.

A potential feature I didn&#x27;t know I needed. Have headphones with ANC on around home all the time, would be really useful if it auto passthrough my partners voice.

They couldn&#x27;t use this to listen to me. They would just get &quot;I am just a large language model, I can&#x27;t help you with that.&quot;<p>I use a lot of curse words. ;)

Imagine it helping people with Autism and ADHD! ADHD people have hard time listening to 1 person because part of the brain tries to listen to all other conversations going around.

This remembers me of NVIDIA RTX Voice [0]. Although not made to isolate single persons, this is quite impressive. I hope that this single person isolation will find it&#x27;s way to consumer noise-cancelling headphones<p>[0] <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=uWUHkCgslNE" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=uWUHkCgslNE</a>

I used to think of building something related to let a mic pick up a single person to handle questions from the audience, during presentations. Will save the hassle of passing around mics.<p>This looks like it could do just that with the headphones feeding directly into the mixer and behaving like a focused mic.

When I was a youngling, I dreamed of having headphones with the opposite power -- muting specific people. For me, it&#x27;s not the hubbub of a crowd that&#x27;s distracting, it&#x27;s usually one or two offending specimens - like in the video example, the inconsiderate vermin using a speakerphone in public.<p>I wonder if the problem maps easily from &quot;select this source&quot; to &quot;select everything but that source&quot;

How much is the AI necessary for this? At least for the targeting of sounds in the line of sight, that should be fairly easy to do without AI, but I don’t know about the human voice identification.

<a href="https:&#x2F;&#x2F;en.m.wikipedia.org&#x2F;wiki&#x2F;Cocktail_party_effect" rel="nofollow">https:&#x2F;&#x2F;en.m.wikipedia.org&#x2F;wiki&#x2F;Cocktail_party_effect</a>

The “cocktail party effect” externalized. Extremely cool.<p><a href="https:&#x2F;&#x2F;en.m.wikipedia.org&#x2F;wiki&#x2F;Cocktail_party_effect" rel="nofollow">https:&#x2F;&#x2F;en.m.wikipedia.org&#x2F;wiki&#x2F;Cocktail_party_effect</a>

The concept of headphones in the last 60 years, with the exception of sound quality, comfort, and eventually being cordless.... they have not changed much in terms of style and appearance.<p>However, I think in the next 50 years, headphones will disappear or.. should I say evolve as part of the human anatomy. Same thing for screen monitors, mouse&#x2F;keyboard, smartphones, etc.<p>Think about it. The way things are going, along with &quot;AI&quot; (sure buzzword in a number of ways but something that will change our way of living) many of things we use will be replaced and, likely, be simple extensions or, dare I say, be implanted.<p>Hard drives will be a thing of the past. Everything will be (as we call today) &quot;cloud-based&quot; and we will be more cybernetic than we think. Of course, someone today will fear such an idea. As we slowly accept the little changes.. in 50 years we will look back and think &quot;how did they cope without it&quot;... a bit like how someone today look back and think &quot;how did they cope without the internet&quot;<p>Many fear what they dont understand. It is the unknown. AI is a fear factor for many. For me, I accept it for what it is and the changes it will impact our lives and our careers.<p>All I will say is -- strap yourselves in.. it will be a bumpy ride. I hope we make it through without destroying ourselves. Once we past it, the world could (finally) be at peace and to quote a famous TV show --- &quot;to bondly go where no man has gone before!&quot;

This reminds me a lot of <a href="https:&#x2F;&#x2F;github.com&#x2F;xiph&#x2F;rnnoise">https:&#x2F;&#x2F;github.com&#x2F;xiph&#x2F;rnnoise</a> and my use of it locally. It zeroes in on voice via RNN which seems to beat most other noise detection filters I&#x27;ve tried. Unfortunately, I mostly disable it these days since it&#x27;s a bit harder to tune than I&#x27;m up for, but it&#x27;s by far the most promising local noise reduction I&#x27;ve used.

In my experience, most people don&#x27;t seem to understand the concept of noise cancelling headphones and will still try to talk to people who clearly can&#x27;t hear them. I can&#x27;t imagine it&#x27;d be any different for these AI headphones in practical use. Probably worse because the person you&#x27;re actually trying to talk to might think you can&#x27;t hear them.

Bit annoying that they added ambient music over the demo youtube video, spoiling the one thing you want to demonstrate.

Pretty cool what they are working on. However, I wished there would be more funding for restoring hair cells which are the root cause for most people with hearing loss.<p>Researchers are getting closer. Dr. Chen from Harvard was able to regenerate hair cells in mature mice last year.<p>The problem is also becoming more widespread. 30 Mio people in the US and 400 Mio people worldwide have disabling hearing loss. Regenerating hair cells and the synapses around them would also cure Tinnitus. 30 Mio x $5k for a treatment = $60B market (probably way bigger with aging population)<p>I think we probably need more rich tech billionaires to get affected to attract large funding.<p>What billionaires that you know are affected besides:<p>- Brad Jacobs<p>- Ryan from Flexport&#x2F;Founders Fund

Sounds like a great way to spy on all people and extract all conversations. I can&#x27;t wait for judges to declare that all conversation at your office must be recorded like some of them have for chat. This tech is a step to enable such a thing.

I hope it&#x27;s not just a prototype press release, will help people with hearing loss..

Presumably the tv ad would feature Gene Hackman. Edit: an AI simulation of Gene Hackman.

Before getting all excited that your ML model runs on your brand new 2024 macbook, before you run off to create earbuds &#x2F; hearing aids with it, please try to run it on-target and see whether your model runs within your runtime budget &#x2F; power budget &#x2F; device size budget &#x2F; battery life budget.<p>And make sure if you&#x27;re going to do bluetooth + wireless, remember that both bluetooth and wifi transmit on 2.4 GHz, and need to coordinate in order to coexist in the same IoT device. There are interconnects and wire protocols to connect the bluetooth and wifi chips together - or, preferably, you buy a chip that does both.

Presumably this could be used to block out specific voices&#x2F;sounds.<p>There&#x27;s an episode of the sci-fi show Black Mirror (White Christmas), where a person is convicted of some hideous crime and permanently blocked&#x2F;made invisible and inaudible from everyone (the entire population has embedded audio&#x2F;video processing enhancements by then).<p>You can imagine future headphones where you could block out the guy in your office with the annoying laugh our download &#x27;blocks&#x27; from the headphone appstore - no more Rick Astley or the politician you don&#x27;t like etc.

Their paper is quoting an end-to-end latency of under 20ms… so impressive!

Curious what sort of processing power or chipsets the &#x27;onboard embedded computer&#x27; needs. Could this be an iPhone app? Or is this going to require new, specialized hardware to commoditize?

&gt; To use the system, a person wearing off-the-shelf headphones fitted with microphones taps a button while directing their head at someone talking. The sound waves from that speaker’s voice then should reach the microphones on both sides of the headset simultaneously; there’s a 16-degree margin of error.<p>Perhaps the accuracy of identifying the correct voice could be vastly increased by adding video input. The AI can then try to match the various voices with the lip movements in the center of the video, basically lip reading.

No need to stand close to the Big Brother&#x27;s telescreen anymore

I think someone made something similar in the 80s by using blind source separation techniques like ICA<p>But this is very useful for people like me who don&#x27;t hear well in the high frequencies.

This is pretty amazing -- and a practical application of a solution to a notoriously tricky problem called the &quot;cocktail party problem.&quot;[1] For a small subset of researchers, writing an algorithm to isolate a voice in a crowd is on par with e.g. writing an AI to play Go.<p>[1] <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Cocktail_party_effect" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Cocktail_party_effect</a>

&gt;A University of Washington team<p>Oh, so it barely works and its a proof of concept.<p>What is the interesting thing here? We all know how sound waves work. Pretty sure this technology is old. Until there is a product here, it just sounds like you are rehashing noise cancellation.<p>Academia has dug this grave of skepticism. I just have 0 faith this will get to market through University of Washington. Maybe it will be patented and used even less!

While not exactly the same, I came across an app called Tunity the other day. It allows you to use your phone camera to catch the live audio feed of the television that you are attempting to watch, whether the audio is muted or if it&#x27;s in a loud, crowded location, like a sports bar or airport. I haven&#x27;t used it, but it&#x27;s an interesting concept.

Next can we have them identify ambient noises that need amplification for safety reasons, like the nearly-silent electric car about to run me over, or the bike I&#x27;m about to accidentally step in front of? As someone who spends a bit too much of my time walking around on calls, I think selective amplification of ambient sounds for safety would be amazing!

I opened an issue with this. Maybe someone here knows.<p>I see a Python script I can run on my computer, I haven&#x27;t tried it yet, but I think I could connect a microphone and process real-time audio and output it in real time, but I don’t know how to detect the user looking at someone. Could you tell me how that works?

I am in the market but “ The system is not commercially available”. This is a perfect opportunity for Apple.

When ANC headphones came out, my friends thought about something like filtering certain sounds away. I bet many people have also had this kind of idea, but nevertheless, haven&#x27;t actually built it. This looks intriguing, and with open-source POC code, it seems promising.

I want to filter out all non-nature sounds. I dream of walking through the airport or the park in peace. AI seems the way to go with that since you have to predict the sound to counteract it. Good to see we are finally making progress.

This could easily hold a library of voices that you interact with (e.g at a bigger table of friends and family) and let you toggle in and out voices that are relevant. Apple please include this feature for your Airpods, thanks! :)

My daughter has an auditory impairment which she describes as &quot;brain deaf&quot;.<p>Basically, her hearing is perfect but her brain struggles to process sound in a noisy environment; she can&#x27;t single out what she is listening to.<p>This sounds perfect for her!

How does it solve the problem of humans being able to detect that someone&#x27;s looking at us? We tend to stop talking when we sense someone&#x27;s staring at us.

A useful tool for when you need to surveil a shady multinational called Quantum while they discuss their evil plan during a performance of Tosca.

I don&#x27;t technically have a hearing problem, sometimes when there&#x27;s a lot of noises occurring at the same time I hear it as one jumble.

This is an actual thing that could work: AI&#x27;s ability to &quot;stem&quot; voices and instruments is really impressive.

By looking at them AND pressing a button. But they might be able to get rid of the button with some sensors and AI.

Feels like what AI <i>should</i> be used for... &quot;filtering out the noise&quot; rather than creating it.

What about the privacy concerns? So basically I can just look at a couple of people talking and eavesdrop?

I would like AI headphones that let me pinpoint the source of noises, such as inside a car engine.

Oh my god, these would be absolutely amazing as someone with auditory processing issues.

This is stuff for spy movies.

You could even make a black list of people you don&#x27;t want to hear!

Curious if it will also help to find a missing person in the crowd.

Does this use Computer Vision camera or how does it work?

This is the equivalent of staring but for ears.

Honestly AI speech recognition still sucks so bad I&#x27;m basically convinced it will fall on its face in many daily use cases.<p>I realize this is slightly tangential, but please don&#x27;t replace customer support with chatbots or whatever you want to call them. It&#x27;s a freaking horrible experience.

I think this is an awesome invention

This will appeal to eavesdroppers.

Is it April 1st already?

get two of these for e2e communication (peer -&gt; ear)

cant wait to see people using earphones at parties

This is not AI.

Ok

Fry_Shut_Up_And_Take_My_Money.gif

Occasional bartender here. Okay!

cue 2024 gene hackman!

aka beam-forming. No AI needed, just good mics.

Now do the same thing with video.<p>Turn anything into a mirror, or something like that.

How is this AI? Not just some form of a parabolic microphone

I love this.<p>I know this is just the beginning and the tech and UX will mature a lot - but being able to consciously choose what we allow into our sensory world would be a great superpower to have.<p>In the distant future this will all be embedded inside a cochlear (neural?) implant.<p>You can &quot;save&quot; known voices, prioritize them, identify various scenes&#x2F;modes automatically like meetings&#x2F;parties&#x2F;concerts&#x2F;driving&#x2F;walking etc, know when to allow external sounds in (alarms, honks, someone calling your attention, etc)<p>And with great power yada yada.<p>I can already imagine a few ways this can be misused &#x2F; abused &#x2F; create non-existent challenges and problems too. But I am (cautiously) optimistic that we as human race will collectively figure out how to steer these new technology applications into net positive territory.<p>2040: iAudio and xSmell blamed for people losing connect with nature&#x27;s sounds (like bird chirps and flowing streams) and smells (petrichor) - things that inspire us, make us creative, make life worthwile, and make us humans.