Opportunity
Dr. Jeff Nagge, Associate Clinical Professor in the School of Pharmacy at the University of Waterloo, faced a dilemma. He developed an online continuing education course for pharmacists that required an in-person clinical experience with patients under his mentorship. The course was in demand across Canada, but the clinical component required pharmacists to travel to his Kitchener, Ontario, clinic three times to fulfill course requirements. This made the course out-of-reach for all but those pharmacists willing to travel to the clinic.
After researching various possibilities for creating an online clinical experience to make the course more accessible, he discovered an innovative solution. He partnered with Ametros Learning and Waterloo’s Centre for Extended Learning to produce an artificial intelligence (AI)-based tool to simulate real life clinic visits by patients. Eight pharmacists took part in a pilot in early 2018, and within a year, some 70 professionals took the course and the opportunity was extended to 4th year pharmacy students.
Innovation
The online course, which is called MOAT (Management of Oral Anticoagulation Therapy), develops learners’ knowledge and skills in managing patients taking blood thinner medications. It consists of a 7-week self-paced learning module and a 5-week series of clinical simulations. The learning module is comprised of introductory materials, required readings, a video of a clinical visit, an assignment, an online test, and a short reflective survey. The most innovative aspect of MOAT, however, is the clinical simulation component.
Through the clinical simulations, learners interact with eight artificially intelligent patients through a text-based chat window. Each patient has a unique health record, medication history, and personality representative of the spectrum of patients a pharmacist is likely to encounter in a real clinic. Learners are expected to consult with each of the patients eight times, resulting in a total of 64 consultations. Consultations typically take 15 to 20 minutes, and no two consultation sessions with the same patient are identical. No more than one consultation session of eight patients per clinic day is permitted so participants have the opportunity to reflect on each day’s experiences and receive feedback.
Through natural language processing, the tool interprets questions asked by learners and responds with answers consistent with the simulated patient’s pre-defined characteristics. Patients are also programmed to ask questions. Learners are supported upon their initial use of the tool, so they know what to expect from the simulated patients and what the patients expect from them. The supports include an introductory video, simulated chat transcripts, and frequent tips as learners conduct their first consultation. The system alerts learners if they fail to ask patients essential questions, such as whether they missed taking their medication. Over time, the system improves its understanding of the questions learners ask. Below is a screenshot of a chat between a project consultant role playing as a pharmacist, Marie Lippens, and a simulated patient, Bruce Gilderschmit.
At the end of each consultation session, learners complete an online form summarizing their advice to the patients about the appropriate dosage of medication, when they should return for their next visit, when to have their blood tested, and other pertinent considerations. The tool provides automatic feedback if their answers are within an acceptable margin of standard patient care. If answers are outside of this range, the instructor receives an e-mail notification to review the learners’ answers and provide appropriate feedback.
Benefits and Outcomes
The most obvious benefit is the virtual clinic allows the MOAT course to be taken anytime, anywhere without having to travel to Dr. Nagge’s physical clinic. Thus, the initial motivation for the undertaking was fulfilled. Moreover, the virtual clinic ensures learners are exposed to a full range of patients having a variety of medical conditions and histories, unlike in an actual clinic where a learner may encounter, by chance, only patients who are not experiencing any abnormalities.
By automating the feedback to learners who respond correctly to patients, the tool frees up the instructor to concentrate on those who need specific help in managing patients. This feature allows for more learners to enrol in the course without significantly increasing the instructor’s workload, although there is a cap of 35 learners per offering to allow enough time for personalized feedback.
Learners are often somewhat skeptical when they first engage with the tool, more so for the practicing pharmacists than the 4th year students. According to Dr. Nagge, any initial doubt quickly gives way to very positive feelings about the experience. Below are typical comments from learners.
“The simulations were by far the most important part of the course for me. They provided simulations of real-world scenarios, which was extremely valuable. The standard feedback as well as the personalized feedback from Jeff were very helpful.”
“I didn't feel confident in my abilities before the simulations, but now I do. I feel prepared to tackle the anticoagulation clinic.”
“I feel like the simulations were the most important aspect of the course to make me feel comfortable with managing patients on warfarin.”
“The simulations were amazing and helped to instill the confidence and practical real world knowledge required to make informed clinical decisions.”
“The simulations allowed me to apply my learning and feel independent. I cannot believe what an amazing learning tool this has been and the confidence it has given me.”
“I feel like the simulations are what really cemented my learning - even if I got a consultation wrong I felt like I learned from it a lot.”
In 2019, the MOAT program was honoured by the Canadian Association for Continuing Education for exemplary work in the category “Non-credit Programming under 48 hours.”
Challenges and Enhancements
Dr. Nagge monitors how the tool responds to learners’ questions and continues to work with Ametros Learning to tweak its performance. For example, the tool initially did not understand a pharmacist’s advice to “hold the dose today,” so this was corrected. Dr. Nagge is also working on a list of frequently asked questions (FAQs) to answer questions learners may have about the tool. Additionally, he wants to improve the quality of the feedback the tool provides to learners.
Although experiences so far are very positive, Dr. Nagge wants to run randomized controlled studies to see how the learning outcomes of the simulated clinic compare to those obtained from an actual clinic. He is also interested in knowing if a less costly “chose your own adventure” design of the tool is as effective as the more elaborate AI-based system currently used.
Potential
According to Dr. Nagge, the topic of anticoagulation management is well suited to a simulated clinical experience because most, but not all, decisions about medication follow commonly accepted guidelines. Adaptation of the tool to other medical topics, such as hypertension medication management, is possible as the recommendations for dosage are relatively straightforward. Indeed, the tool could be adapted to almost any field requiring the development of knowledge and skills to work with people provided there is a clear set of procedures to be learned.
Further Information
Jeff Nagge, PharmD, ACPR
Associate Clinical Professor, School of Pharmacy, University of Waterloo
Pharmacy Lead, Michael G Degroote School of Medicine, Waterloo Regional Campus, McMaster University
Kitchener, Ontario
Canada
[email protected]
