prescribing drugs with a voice assistant

Drugs can be prescribed to help patients heal from disease or stay in balance. However, some of these prescription tasks are part of routine jobs in case the patient has a well-known or high prevalence disease with a standard treatment. In other cases, there can be a ‘batch’ of prescriptions to perform to acquire a certain treatment protocol. During the prescription process, variables like weight, kidney function and liver function can all have an influence on which drugs are prescribed and in which dose.

In this Python tutorial, we are going to see how drugs can be prescribed with a voice assistant. This might sound scary because of the possibility of the system making mistakes in understanding you as a prescriber, but we are gonna use closed-loop communication to ensure that the voice assistant does exactly what we want.

Closed-loop communication is a way of communication that’s most popular for its use in the military and in aviation. This communication style entails the reporting back of the interpretation of verbal instructions to confirm that the message has been received accurately.

Let’s start with our workflow to get an overview of what we are going to create. In our case closed-loop communication consists out of the reporting back of the whole medication list:

creating a CONVERSATION

To enable verbal conversation with the voice assistant we need to install the following Python modules:

pip install SpeechRecognition (speech to text module)

pip install pyttsx3 (Python text to speech module)

Import the modules in your script in the following way:

import speech_recognition as sr
import pyttsx3

We need a collection of sentences the computer can say in our conversations. Instead of creating a separate string for the interaction in the execution of different procedural steps we are gonna make a list of all expressions we need and save it as a list of strings. We can use the {} to assign places were we want to insert a variable. Feel free to add your own sentences to the list or add your own name within the sentences to personalize the voice-assistant! You can’t break something over here as long as you also change the references to this list later on.  

conversation_list = [
"How can I help you?...",
"You want to start with {medication}. \
The standard dosing is {numb_day} times a day {dosing}mg. Is this what you want to prescribe?",
"Based on a kidneyfunction of {GFR} and a weight of {weight} kg, \
I calculated a needed dosing of {numb_day} times a day {dosing}mg. Is this what you want to prescribe?",
"Do you want to change the dose or cancel prescription?",
"How much milligram do you want to prescribe?",
"How many times per day do you want the patient to take this?",
"Ok, do you want to change the dosing to {dosing}mg {numb_day} times per day?",
"Ok, let's start over again",
"Ok, I will prescribe {medication} in a dosis of {dosing}",
"The patient medication list now consists of {medication_list}",
"I didn't hear you correctly, or the drug is not in my drug dictionary, please tell me again which drug you want \
to prescribe"]

Ok, let’s now connect the list of strings to the text to speech module. Because we are gonna do this action repetitively we are making a function to do this and try the function out by running it:

def text_speech(text_line):
engine = pyttsx3.init() # import the pyttsx3 module functions under the name engine
engine.say(text_line) # prime the say function with the variable text_line
engine.runAndWait() # run the tts function, without calling this function the voice assistant doesnt start talking

text_speech(conversation_list[0])

You should be able to hear the voice assistant speak for the first time. That’s cool right? And all of that with just 3 lines of code.

To make a conversation we need to be able to use our speech as input. This takes the steps of capturing audio from the microphone, analysing the speech using google speech tot text and returning the result of google speech tot text analysis as a string like this:

def speech_text():
r = sr.Recognizer()
# start capturing audio from microphone
with sr.Microphone() as source:
print("Listening...")
audio = r.listen(source)
# analyse speech using google speech tot text
try:
string = r.recognize_google(audio)
print(string)
except Exception as e:
print(e)
string = "error occured during NLP of audio"
# return result as string
return string

extracting THE DRUGNAME

Next, we need to extract the medication we want to prescribe from our human voice input. We can do this by comparing whether the input resembles a medication in a dummy medication dictionary. We can make a sample dictionary of medication with standard doses. The key of the dictionary item is a drug, and the value is a list with two numbers. The first number is the number of times a drug has to be administered and the second on is the number of milligrams. To keep it simple we are not going to define the route of administration:

medication_dictionary = {
"Morphine": [6,5],
"Midazolam": [6,5],
"Metoprolol": [1,50],
"Ciproxine": [2,500],
"Augmentin": [3,625],
"Amoxicillin": [3,500],
}

Now we compare the input string of our conversation with the fuzzywuzzy module with the keys in the dictionary. This is the same module we used as in our finding a record tutorial. In short, the Levenshtein distance is calculated by finding the easiest way to transform one string into another.

We first import the fuzzywuzzy function and then make a function in which we define the voice input as string1, and the drugs in the medication dictionary as string 2 with the help of a for loop. Within the for loop we do a comparison with fuzzywuzzy between string1 and string2 and if the comparison yields a resemblance (partial ratio, calculated with the Levenshtein distance) of more then 80 we get the standard dose from the medication dictionary. If there is no partial ratio > 80 we go back into the loop by recalling the function we’re in:

# get standard dose function
def get_standard_dose(voice_input):
# define human voice input as string1
string1 = voice_input
# make drug_name a global variable because we need to use them outside the function
global drug_name
# start a for loop to parse the medication dictionary
for drug_name in medication_dictionary:
# define drug in dictionary as string2
string2 = drug_name
# calculat Levenshtein distance between the drug name and the different words in the input string
partial_ratio = fuzz.partial_ratio(string1.lower(), string2.lower())
if partial_ratio > 80:
print("found medication in list: ", drug_name, ", partial ratio = ", partial_ratio)
# make numb (number of times per day and dose global variable, we need to use them outside the function
global numb, dose
# define numb as first item of list in the value of dictionary key (drug)
numb = medication_dictionary.get(string2)[0]
# define numb as second item of list in the value of dictionary key (drug)
dose = medication_dictionary.get(string2)[1]
return
# let voice assistant say there was an error if drug was not found
text_speech(conversation_list[-1])
# let user give new input in case of error
voice_input = speech_text()
# go into loop (start function again) in the case of no match
get_standard_dose(voice_input)

We can try out what we have until now with the following three lines. If everything is working correctly you should be able to have a short conversation in which you instruct the voice assistant to prescribe a drug (e.g. “I want to prescribe morphine”) and the voice assistant tells you the standard dose of that drug. The condensed version of what we did until now looks like this:  

text_speech(conversation_list[0])
voice_input = speech_text()
get_standard_dose(voice_input)

ADJUSTING DOSE BASED ON PATIENT WEIGHT AND KIDNEY FUNCTION 

We skipped one important step in our workflow, taking into account kidney function and weight as an input to adjust the dosing. To do that we are gonna open a browser and scrape the lab results and weight from our online dummy patient file with selenium:

# import scraping modules
from selenium import webdriver
import pandas as pd

# open browser
driver = webdriver.Chrome()

# scrape lab results
driver.get("https://medicalprogress.dev/patient_file2/lab_results.html")
html = driver.page_source
data = pd.read_html(html)

# clean the data
data = data[0]
df = pd.DataFrame(data)

# extract GFR
GFR = df.iloc[2,1]

# scrape weight
driver.get("https://medicalprogress.dev/patient_file2/weight.html")
html = driver.page_source
data = pd.read_html(html)

# clean the data
data = data[0]
df = pd.DataFrame(data)

# extract weight
weight = df.iloc[3,1]

Now we calculate the adjusted dosing based on kidney function and weight. You can make adjustments for all different drugs in the medication dictionary, but in this tutorial we are going to focus on the most frequently prescribed antibiotic in the Netherlands augmentin. The Dutch drug dictionary for augmentin says the following in respect to weight and kidney function:

  1. The dose has to be adjusted in the case of a weight <40kg. Nothing is said about extreme high body weights.
  2. The dose has to be adjust based on kidney function: GFR 10–30 ml/min -> 2 times a day and GFR <10 ml/min -> 1 time a day.

This results in the following lines of code in a function:

# Adjust augmentin dose function
def adjust_dose_augm(numb):
print(weight,GFR)
if weight < 40:
raise Exception(text_speech("I received an error, the weight is lower than 40kg"))
if 30 > GFR > 10:
numb = 2
if GFR < 10:
numb = 1
return numb

We incorporate this in our get_standard_dose function with an if statement that compares string2 to the string “Augmentin” and adjusts the dosing if needed. The voice assistant is asked to speak out loud the adjusted dosis:

# get standard dose function
def get_standard_dose(voice_input):
string1 = voice_input # define human voice input as string1
global drug_name # make drug_name a global variable because we need to use them outside the function
for drug_name in medication_dictionary: # start a for loop to parse the medication dictionary
string2 = drug_name # define drug in dictionary as string2
# calculat Levenshtein distance between the drug name and the different words in the input string
partial_ratio = fuzz.partial_ratio(string1.lower(), string2.lower())
if partial_ratio > 80:
print("found medication in list: ", drug_name, ", partial ratio = ", partial_ratio)
global numb, dose # make numb (number of times per day and dose global variable, we need to use them outside the function
numb = medication_dictionary.get(string2)[0] # define dose as dose (first value) within the dictionary key (drug)
dose = medication_dictionary.get(string2)[1] # define dose as dose (second value) within the dictionary key (drug)
text_speech(conversation_list[1].format(medication=drug_name, numb_day=numb, dosing=dose)) # speak!
if string2 == "Augmentin":
numb = adjust_dose_augm(numb)
text_speech(
conversation_list[2].format(medication=drug_name, numb_day=numb, dosing=dose, GFR=GFR,
weight=weight)) # speak!
return
else:
text_speech(conversation_list[1].format(medication=drug_name, numb_day=numb, dosing=dose)) # speak!
return
text_speech(conversation_list[-1]) # let voice assistant say there was an error
voice_input = speech_text() # let user give new input
get_standard_dose(voice_input) # go into loop (start function again) in the case of no match

disAPPROVAL and CHANGING THE DOSE  

From here, we make a couple of branches in our workflow as shown below. Disapproval can lead to choosing the cancel option and restart the module or choosing the change option and creating a new dose from the new user input. Approval leads to the voice assistant prescribing the drug.


To keep it simple we are going to do a restart of the algorithm by restarting the whole script in the case of cancelling the prescription. To achieve this we need to import the os and sys module. We can express the above workflow as the following function (Note the closed-loop communication loop before the 4th if statement):

import os
import sys

def decision(voice_input):
# approval
if str.lower(voice_input) == "yes":
return
# change
if str.lower(voice_input) == "no":
text_speech(conversation_list[3])
voice_input = speech_text()
# confirm change
if "change" in str.lower(voice_input):
text_speech(conversation_list[4])
# input number of mg
global dose
dose = speech_text()
print(dose)
text_speech(conversation_list[5])
# input times per day
global numb
numb = speech_text()
print(numb)
# closed loop communication
text_speech(conversation_list[6].format(dosing=dose, numb_day=numb))
voice_input = speech_text()
# if new dosing correct continue
if str.lower(voice_input) == "yes":
return
# new dosing incorrect restart function
if str.lower(voice_input) == "no":
decision(voice_input)
# cancel
if str.lower(voice_input) == "cancel":
# restart script
os.execv(sys.executable, ['python'] + sys.argv)

We can do a test run with the following lines of code. Note the conversation like structure of the code: speech-to-text -> input string -> processing -> text-to-speech -> speech-to-text, etc.

text_speech(conversation_list[0])
voice_input = speech_text()
get_standard_dose(voice_input)
voice_input = speech_text()
decision(voice_input)

updating the patient medication list

The next objective is to update our medication list in the patient file. We scrape the medication list from our dummy patient file. The medication list is a javascript list so we can’t just use pandas, we have to use the BeautifulSoup web scraping module in this case. Install BeautifulSoup, but to make BeautifulSoup work we also need the html5lib module:

– pip install html5lib

– pip install beautifulsoup4

We use the following lines of code to scrape the ‘soup’:

from bs4 import BeautifulSoup

driver.get("https://medicalprogress.dev/patient_file2/medication_list.html")
html = driver.page_source
soup = BeautifulSoup(html, 'lxml')

The scraped text consists of the HTML file but in a kind-of-soup-style. We need to extract the list of medications and clean it. we can do this with a for loop:

download_medication_list = []
for li in soup.findAll('li'):
download_medication_list.append(li.getText())

We check if the drug we prescribe is in the downloaded list and if not we prescribe the drug given by the user with the Selenium web automation. When doing this, the script can be too fast for the web browser. This is a common case when using Selenium in python. The web browser can’t keep up and the script will then skip certain instructions and let your application crash. We therefore import the time module and let the script sleep for a second between instructions. You might want to prolong this period if you have a slow computer or a lot of software running in the background of your computer:

import time

# check if drug in medication list
if drug_name in download_medication_list:
text_speech("The drug is already in the medication list.")
else:
# closed loop communication
text_speech(conversation_list[9].format(medication_list=download_medication_list, new_drug=drug_name,
numb_day=numb, dosing=dose))
voice_input = speech_text()
if "yes" in voice_input.lower():
pass
else:
text_speech(conversation_list[7])
os.execv(sys.executable, ['python'] + sys.argv)
# prescribe new medication
time.sleep(1)
input_field = driver.find_element_by_xpath("/html/body/input[1]")
input_field.send_keys(drug_name + " " + str(dose) + "mg " + str(numb) + " times per day") # fill in info in inputfield
time.sleep(1)
driver.find_element_by_xpath("/html/body/input[2]").click() # click on the add button
time.sleep(1)

Closed loop communication

We already used a small closed-loop communication loop, but now is the time to reevaluate if the system understood what we said and is going to do the right thing.  Inside the else block of the previous code we implement communication feedback of the medication list that we downloaded.  If the user gives approval, we continue with prescribing. 

# check if drug in medication list
if drug_name in download_medication_list:
text_speech("The drug is already in the medication list.")
else:
# closed loop communication
text_speech(conversation_list[9].format(medication_list=download_medication_list, new_drug=drug_name, \
numb_day=numb, dosing=dose))
voice_input = speech_text()
if "yes" in voice_input.lower():
pass
else:
text_speech(conversation_list[7])
os.execv(sys.executable, ['python'] + sys.argv)
# prescribe new medication
time.sleep(1)
input_field = driver.find_element_by_xpath("/html/body/input[1]")
input_field.send_keys(drug_name + " " + str(dose) + "mg " + str(numb) + " times per day") # fill in info in inputfield
time.sleep(1)
driver.find_element_by_xpath("/html/body/input[2]").click() # click on the add button
time.sleep(1)

 

We reload the medication list and return it in printed form to the user and after that close the browser:

# redownload adjusted medication list
time.sleep(2)
html = driver.page_source
soup = BeautifulSoup(html, 'lxml')

# get drugs from adjusted medication list
adjusted_medication_list = []
for li in soup.findAll('li'):
adjusted_medication_list.append(li.getText())

# print list
for item in adjusted_medication_list:
print(item)

# close browser
driver.close()

the backbone

So, now we have completed a basic tutorial version of a drug prescription voice assistant that can take on the task of prescribing drugs in a dummy patient file in a dummy electronic health records system. In the case of augmentin it will adjust the dose automatically based on kidney function and weight. This script is far from useful in day-to-day work. There is a whole lot more to do. However,  the backbone structure is there and you can start to tinker with it.   

A nice ideas to start with is to organise the script. Functions, lists and import statements are scattered throughout the script for the purpose of making a logical step-wise tutorial, but for maintenance purposes, it is often better to make blocks of code. It would also be a good idea to place the functions and lists in a separate file and call them from there into a main.py file. Other ideas to try out are creating an evaluation function for allergies, letting other variables (such as liver function or disease indication) play a role in the decision proces of a dosing or connecting the backend of the machine to an official lookup table of drug names.

You can find the full code on my github page. Before you use the code in production please check the readme and the MIT license.