WhisperNER: Unified Open Named Entity and Speech Recognition

The title is dense and the paper is short. But the demo is outstanding: (<a href="https:&#x2F;&#x2F;huggingface.co&#x2F;spaces&#x2F;aiola&#x2F;whisper-ner-v1" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;spaces&#x2F;aiola&#x2F;whisper-ner-v1</a>). The sample audio is submitted with &quot;entity labels&quot; set to &quot;football-club, football-player, referee&quot; and WhisperNER returns tags Arsenal and Juventus for the football-club tag. They suggest &quot;personal information&quot; as a tag to try on audio.<p>Impressive, very impressive. I wonder if it could listen for credit cards or passwords.

GitHub repo: <a href="https:&#x2F;&#x2F;github.com&#x2F;aiola-lab&#x2F;whisper-ner">https:&#x2F;&#x2F;github.com&#x2F;aiola-lab&#x2F;whisper-ner</a><p>Hugging Face Demo: <a href="https:&#x2F;&#x2F;huggingface.co&#x2F;spaces&#x2F;aiola&#x2F;whisper-ner-v1" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;spaces&#x2F;aiola&#x2F;whisper-ner-v1</a><p>Pretty good article that focuses on the privacy&#x2F;security aspect of this — having a single model that does ASR and NER:<p><a href="https:&#x2F;&#x2F;venturebeat.com&#x2F;ai&#x2F;aiola-unveils-open-source-ai-audio-transcription-model-that-obscures-sensitive-info-in-realtime&#x2F;" rel="nofollow">https:&#x2F;&#x2F;venturebeat.com&#x2F;ai&#x2F;aiola-unveils-open-source-ai-audi...</a>

Is there any reason why this would work better or is needed compared to taking audio and 1. doing ASR with whisper for instance 2. applying an NER model to the transcribed text?<p>There are open source NER models that can identify any specified entity type (<a href="https:&#x2F;&#x2F;universal-ner.github.io&#x2F;" rel="nofollow">https:&#x2F;&#x2F;universal-ner.github.io&#x2F;</a>, <a href="https:&#x2F;&#x2F;github.com&#x2F;urchade&#x2F;GLiNER">https:&#x2F;&#x2F;github.com&#x2F;urchade&#x2F;GLiNER</a>). I don&#x27;t see why this WhisperNER approach would be any better than doing ASR with whisper and then applying one of these NER models.

On a similar note, I&#x27;ve a request for the HN community. Can anyone recommend a low-latency NER model&#x2F;service.<p>I&#x27;m building an assistant that gives information on local medical providers that match your criteria. I&#x27;m struggling with query expansion and entity recognition. For any incoming query, I would want to NER for medical terms (which are limited in scope and pre-determined), and subsequently where I would do Query rewriting and expansion.

It&#x27;s so great to see that we finally move away from the thirty year old triple categorization of people, organizations and locations.<p>This of course means that we now have to think about all the irreconcilable problems of taxonomy, but I&#x27;ll take that any day over the old version :)

&quot;The model processes audio files and simultaneously applies NER to tag or mask specific types of sensitive information directly within the transcription pipeline. Unlike traditional multi-step systems, which leave data exposed during intermediary processing stages, Whisper-NER eliminates the need for separate ASR and NER tools, reducing vulnerability to breaches.&quot;