Show HN: Using GPT-3 and Whisper to save doctors’ time

I don&#x27;t spend much time worrying about AI ethics, but bringing AI close to patient interactions and record-keeping in healthcare seems grossly premature and irresponsible. There are countless brilliant applications of AI and the last one we need is a mediocre automated transcriptionist that distances a doctor from their responsibility to engage with the patient and can bear no accountability for error.<p>This is a task that perhaps can be supported with AI some day, but there are fields that deserve the application of a mature technology, not the gold rush rush game of integrating today&#x27;s hottest thing.

As a patient, this is an application that I would be extremely wary of, personally. I don&#x27;t want the details of my conversations with my doctor to be sent to a third party in the first place, and I wouldn&#x27;t trust the results of the transcription to be correct.<p>My doctor could vet it for accuracy, I suppose, but why? He&#x27;s already putting his notes in my records anyway.

I like the idea, but I would like it even more if it was on-prem. The doctors (at least in EU) will be very wary of having their client meeting essentially recorded by a third party. With this as a cloud SaaS, patient confidentiality is essentially broken since the raw data is available to you while you transcribe it. I understand that you compete with &quot;google speech-to-text&quot; but this is not a feature meant to be used by doctors (even if they &quot;illegally&quot; do).<p>Obviously the business model is harder with on-prem, but cloud-first for doctor notes is in the long run much harder.

My worry is verification of facts. What if the model summarized incorrect facts, and it gets added to the medical history? How can doctors easily verify and fix things?

I feel uncomfortable knowing doctor would use anything like this.<p>How are 2 party consent states handled?<p>Is this HIPPA compliant?

This is the scariest real life attempt to use the bullshit machine I’ve seen so far.<p>Nothing like “subtly wrong” clinical notes to affect positive medical outcomes. And time pressed medical professionals are certainly well suited to vigilance tasks like correcting machine generated errors. &#x2F;s.<p>I sincerely hope this never sees the light of day.

Technically interesting but solving the wrong problem.<p>Doctor&#x27;s already today spent too little time with their patients to understand diseases holistically enough.<p>Adding technically between these 2 will make treatments in most of the cases worse, not better.

Last time I went to the doctor, he charged me $300 to google symptoms on a computer. Now you&#x27;re saying I get to pay $300 for him to ask ChatGPT? No thanks. We all know that is where this is going - doctor decision support system, i.e. the AI has access to information locked behind expensive journals that the patient does not.

The &quot;feed it to a heavily trained GPT-3&quot; part is backwards. The transcribed notes should be stored and retained. The LLM should be applied later, in real time, when someone is referring back to the notes.<p>That way you still get the benefit of summaries, without polluting the medical records with hallucinated AI nonsense.

Ignore the nay-sayers in this thread. I doubt the ones commenting have spent many days personally observing or personally doing the problem scenario you are solving for.<p>This is spectacular for telemedicine in the post-covid regulation environment.<p>The new model is an onsite medical assistant taking patient vitals and setting up the Zoom, the patient then talks to an APRN (Advanced Practice Registered Nurse) in the patient&#x27;s field (e.g. subacute care, gerontology, etc.).<p>The APRNs spend 30 minutes reading the file, tele-consult with a patient for a half hour. Medicare pays for the half hour. Then the APRN has to spend ~60 minutes turning that session into the EHR notes. Many just don&#x27;t, they make an ultra short note, and hope Medicare doesn&#x27;t challenge it. Patient loses. If too strapped for time, they also may not pre-read, because they only get paid for the video conf time. Patient loses.<p>Downside I see with this is the APRNs doing telemedicine are making about 1&#x2F;4 of the rate they should be, and can&#x27;t afford an expensive tool. The telehealth provider is not going to pay for it, they get paid for the 1&#x2F;2 hour, the wasted time is not out of their pocket, so they won&#x27;t spend to make it up. This is going to be on the nurse, who likely can&#x27;t afford it.<p>On the plus side: time saved with this tool would go directly to ability to pre-read, or ability to see a patient an hour instead of one per two or three hours. It would also permit a higher caliber of expertise nurse to do tele-medicine visits in off days, because the time saved could bring comp back more in line with the expertise.<p>Go for it!

&gt; This allows doctors to be fully focused on their consultation<p>Here&#x27;s the thing: rigorous medical documentation is not a burden on the practitioner, but rather an integral part of treating someone. You need detailed historical data to diagnose some tricky conditions and treat them correctly.<p>What&#x27;s the tool doing to prevent a practitioner from simply blindly copy-pasting the output without verifying it?<p>&gt; and saves them a lot time.<p>Which means more billable. Have you considered charging per medical act performed?

Is Whisper really up to this? When I&#x27;ve used it, it&#x27;s...fine. But it also randomly dreams up weird transcriptions that have nothing to do with what&#x27;s actually being said, and with perfect audio too. Like the audio will be a presentation at a tech conference, and Whisper will predict &quot;check out our site at randomlink.com&quot;, when no one said anything close to that.<p>I&#x27;d be pretty concerned if my doctor relied on this to be honest.

I&#x27;d prefer to have 0 AI interpreting my session with a doctor. Will there be any way for me to identify which physicians use this service so that I can avoid using them?

I did some work at a startup that tried to solve a similar problem and currently finishing medical school. Nobody here is discussing the paperwork and burnout problem that exists in medicine that will make these tools valuable. Fortunately, language models used for summarization based on a transcript are much less likely to hallucinate compared to upscaling&#x2F;denoising image algorithms. Recent LMs are scoring top marks on LSAT, so they are maintaining accuracy as they get larger. If doctors can save time, this will be useful for the patient in the long-run.<p>On-prem stuff is more tricky, but useful for privacy. Nvidia is supporting the space with their health AGX series.

I used to work in a hospital and one thing that was already tough was when doctors would copy massive (multiple page) notes that were mostly boilerplate, probably stored as some template in the EHR. After seeing the same note a few times, you started to learn which page to scroll to in order to view the relevant section for the patient. I would hope this system only generates clinically-relevant text for that patient, usually only 2-3 paragraphs for a standard consultation.

This is really interesting work, congratulations! A lot of negativity in the comments seems to stem from cynicism around doctors and their billing, but in a country like the UK I know removing essentially &quot;admin tasks&quot; from the GP would be very welcome and give time back to the clinician, allowing them to spend more time focussing on the patient and outcomes.

&gt; We then transcribe the consultation using a fine-tuned version of Whisper. We&#x27;ve trained Whisper with tens of thousands of hours of medical consultation and medical terms recordings, and we have now reached an error rate which is 3× lower than Google&#x27;s Speech-To-Text.<p>I think the general idea is interesting but this is the wrong benchmark to convince people imo. What is the actual error rate of Google&#x27;s Speech-to-Text on this your data? How convinced are you that it&#x27;s properly labeled? That it&#x27;s 3x lower isn&#x27;t necessarily impressive. Then a follow up question... What I as a patient would want to know is something else, how does it compare against medical secretaries that usually transcribe documents like these?

I&#x27;m a medical student and one of my main takeaways from spending time in clinical environments is that a lot of doctors and other staff seem to either not realise or not care about how much time is wasted due to slow, inefficient processes and technology. I was wondering to myself about what it might take to train a model to recognise medical jargon to help doctors rapidly transcribe audio recordings of their consults&#x2F;spoken recounts of their notes. Not surprised that somebody is looking to make a commercial product of it, there&#x27;s definitely space for it.

It looks pretty neat! Just wondering, what other languages are supported? English seems pretty natural as it is where GPT-3 thrives, but have you been able to support other languages as well?

How tied are you to the name, it&#x27;s almost Nambla.

What is OpenAI&#x27;s privacy policy? How do you ensure the transcription doesn&#x27;t get stored by OpenAI or used in training their future LLM versions?<p>I&#x27;d recommend looking at a fully self-hosted LLM alternative or ensuring the GPT-3 endpoint you&#x27;re using is compliant (I&#x27;ve heard Azure has a GPT-3 API that is supposedly more &quot;compliant&quot;).

Maybe the future of the first line of healthcare will start to use AI with the assistance of a specialized nurse to follow your health overtime and only fall back to a real doctor when a real problem occurs.<p>This could elevate a lot of problems we have with our public healthcare(Canada).

Love this and have had discussions w&#x2F; doctors and practitioners about this exact problem. I think extracting to FHIR is key, along with making it searchable as a longitudinal data store for a more robust personal record. Clear need. Would love to help.

This is really cool! What surprised you guys about fine-tuning whisper + gpt-3?

Side-note: your email field to sign up for your newsletter is very wonky with both Safari and Firefox (at least for me). It only lets me type one letter at a time, for some reason.

Just read the blog post, very interesting. I thought sections about changing the temperature and repetition penalty were particularly neat, thanks for including examples.

One thing that a lot of folks still don&#x27;t get is that some problems are bad, while others are not-so bad.

Pretext: My wife is a psychiatrist. Via her, I&#x27;m am close friends with many doctors. Also, I know it&#x27;s super easy to be critical, but I&#x27;m going to save you an insane amount of effort and heartache.<p>This is actually a solved problem in medicine, already. Maybe not solved, but absolutely &quot;good enough&quot;. M-Modal, Dragon, and Telephone services (yes, call someone and have them translate) all exist. They all translate notes well enough. For most doctors, it&#x27;s a huge improvement over typing.<p>The problem is primarily access. Most institutions simply do not want to pay for this. They don&#x27;t view it as necessary. Further complicating the issue. Unless a physician is in private practice, they have little to no control over their ability to use a tool like this. HIPAA is far too high of a barrier.<p>You&#x27;re trying to implement a technical solution to what&#x27;s ultimately a politics&#x2F;human problem.<p>----<p>I&#x27;ve spent a bunch of time in the space, both as a MedSpouse and the CTO of a HealthTech company. I wish you the best of luck as I&#x27;d love to see life be easier for my wife, but this is _extremely_ far from being MVP viable.<p>* HIPAA - You absolutely, 100%, without a doubt need to be HIPAA compliant. You likely want to be HITRUST compliant as well. You&#x27;re very, very clearly a Business Associate (HIPAA term) and processing PHI. Without that, you&#x27;re dead in the water.<p>* Chrome extension - This is not going to cut it. At baseline, it requires copy and pasting between Chrome and the EHR system. Bad UX.<p><pre><code> * More tactically, most EHR access occurs via remote access, like Citrix. Often from a thin-terminal on the hospital floor or at a med station. These are locked down environments that have a high likely hood of not having Chrome installed _and_ not being allow to install _any_ software (including Chrome extensions). </code></pre> * Doctors need real-time feedback and the ability to correct. Real-time feedback is critical because carrying a patient takes mental load. Part of the purpose of notes is to create a written record that &quot;closes&quot; a patient for the time being. An async service isn&#x27;t really helpful since it generates an interrupt after the physician has already moved on.<p>* Accuracy is important. Accuracy is even more important with _extremely_ similar and _extremely_ uncommon words. For example, can this system differential between prednisone and prednilisone? In fact, can it even identify those words? Can it do it with an accent?<p>* FHIR is only as much of a standard as tables are to relational databases. The hard part is the thousands of custom fields that are different between health systems.<p>* Structured data really isn&#x27;t used or useful in medicine. You might have a medication list that can be structured as data, but actual notes include a lot of prose. I believe part of this is necessary to capture the nuance&#x2F;minutiae of some healthcare. Part of it stems from the lack of any standard.<p>* Again, this is all possible already. Tele-notes have been done since the introduction of EHR systems.<p>* Nobody is going to use a Chrome extension.

it sounds like you&#x27;re removing a checklist from doctors.

Will you sign a BAA for HIPAA covered entities?

Looks very interesting! Are you recruiting ?

Congrats on the launch Nabla!

That&#x27;s super cool!

Amazing idea!!

how can this be compliant with HIPAA?

this is amazing. great great

amazing. you&#x27;re ROCKKKKKKKKKK