Welcome back to our Generative AI Playground series. In the past months, we’ve shown you some of our demos illustrating how to use generative AI in a broader business setting. For this article, our focus has shifted to an essential area of our society—healthcare. Nurses and doctors, the backbone of patient care, often find themselves burdened with extensive documentation requirements that add a considerable workload to their primary care duties. Our latest experiment in our playground aims to lighten their burden through the power of AI, enhancing both efficiency and patient interaction.

New to our Generative AI Playground? We’ve already discussed report generation, function calling, multi-agent GPT systems, visual search and AI avatars in our previous parts.

Healthcare documentation: a never-ending challenge

We have all heard of nurses and doctors working long hours under intense pressure while trying their best to offer the best possible care. In addition to taking care of patients, monitoring their condition, and administering medications, nurses must keep patient records up-to-date.

The same applies to doctors: each interaction, diagnosis, or treatment must be noted somewhere in Electronic Health Records (EHR). The traditional method of documenting patient data is simply doing it manually—a process that is time-consuming and susceptible to errors. The need for an innovative solution to streamline this process was evident, and we wanted to see if generative AI could help lighten this burden.

Introducing AI-driven speech-to-text documentation

Combining SST and LLMs

To tackle this challenge, we developed a prototype using the latest speech-to-text technology, powered by OpenAI’s Whisper, combined with a highly customized Large Language Model (LLM). The system is designed to understand and process natural language spoken by nurses or doctors during their rounds, converting these oral reports into structured, accurate EHR entries.

How it works: The healthcare professional speaks naturally, discussing everything from vital signs like blood pressure and body temperature to subjective assessments of patient progress. The system captures this audio in real-time and processes it through our state-of-the-art speech-to-text model. The healthcare professional has the flexibility to provide information all at once or in separate segments, and can freely switch topics or revise previously mentioned details, all in fluent natural language. The LLM then interprets the content to identify and classify relevant medical information. Importantly, the AI can accurately handle and structure the information in any language.

As mentioned, we customized our LLM to recognize a wide array of medical terms and their synonyms, ensuring high accuracy in data capture. Moreover, we have integrated a verification step that prompts the nurse or doctor to verify any ambiguous data before final submission, adding an extra layer of reliability. This is important as the final responsibility always lies in the hands of the healthcare professional using the tool.

‍

‍

Privacy and compliance

Patient confidentiality is one of the most important requirements when developing IT systems in medicine. That’s why, for this use case, we run our models on secure Azure data centers in Europe, meaning our system complies with GDPR standards. This ensures that patient data is handled with the utmost privacy and security. Additionally, our AI is programmed to automatically redact any personal identifiers from the documentation, thereby maintaining patient confidentiality without compromising the quality of medical records.

Beyond the medical field: endless possibilities

Of course, this playground experiment goes beyond the medical field—for us, it highlights the power of generative AI in completing administrative tasks without having to type a single letter, be it in sales, customer support, human resources, or any field overwhelmed with manual data entry. To show the possibilities, we translated it into four different use cases.

• Sales and CRM: After client meetings, sales representatives can simply speak to update their CRM entries, dramatically reducing administrative overhead and improving data accuracy. Even in their car on the way to the next meeting!
• Meeting summarization: During corporate meetings, generative AI can provide real-time summaries and action points, facilitating better meeting outcomes and record-keeping. Microsoft and Google are already implementing this use case as we speak!
• IT support enhancement: By structuring support tickets from verbal descriptions of technical issues, the AI system can improve response times in IT support services.
• Recruitment: After a job interview, a recruiter could simply explain what they learned about a person, and the information could be automatically entered in their applicant tracking software.

Of course, this is a non-exhaustive list of what we could do with this technology. Have an idea in mind? Let’s discuss!

What’s next?

The deployment of conversational AI in medicine is only one of the practical applications we have explored over the last few months. This specific use not only streamlines administrative tasks but also allows healthcare professionals to devote more time and attention to patient care. And if you’re wondering what we’re experimenting with right now, don’t worry. We’ll be back soon with our seventh (!) experiment of our playground.

‍