Opportunity
At the Michael G. De Groote School of Medicine at McMaster University in Hamilton, Ontario, Canada, senior medical students rotate through 63 weeks of core placements during their workplace learning phase. The students, referred to as Clinical Clerks, may be in a hospital, an out-patient clinic and other settings across the geographic area served by McMaster’s three Ontario campuses for undergraduate medicine in Hamilton, Niagara and Waterloo.
The MD Program wants to ensure students complete a broad range of essential experiences with actual patients, as well as receive direct observation and feedback from faculty and other observers in all the various rotations, such as Emergency Medicine, General Surgery and Pediatrics. These are also requirements for program accreditation.
In the past, tracking of senior medical students during their clinical placements was done on paper, a different format used for each rotation, such as carbon copy forms, booklets and checklists. The forms were not easy to review or compile on a student or rotation basis.
Dr. Anthony Levinson, Director of the Division of e-Learning Innovation, participated in many provincial and national medical groups looking at the possibility of using technology to facilitate recording of essential experiences and feedback during student placements. He carefully monitored their first attempts at integrating technology. Using desktop or laptop computers, students were required to complete detailed reports on each encounter. Due to time constraints, most students delayed completing the forms and completed them in batches. Improvements were made, but there were still issues.
With the introduction and ubiquitous ownership of tablets and smartphones, Dr. Levinson saw McMaster’s opportunity to develop tools Clinical Clerks could use to quickly and effectively track and report on their essential experiences during each rotation and to record direct observation and feedback on their performance. Two tools were developed to respond to these needs: The Essential Clinical Experiences Tracker and the McMaster Direct Observation Tool.
Innovation
Essential Clinical Experiences Tracker (ECE Tracker): Each Clinical Clerk completes 10 clerkship rotations of varying lengths during their 15-month work placements. For example, the rotation in Emergency Medicine is 2 weeks, while General Surgery is 2 months. In each placement, there are essential clinical experiences students must have.
The first step in developing the ECE Tracker tool was reaching consensus among all the various medical disciplines involved in clerkship rotations on how the input is completed and exactly what is recorded. It is essential the logging experience for the student be the same for every rotation.
ECE Tracker was initially developed using responsive web forms and later as a native app, listing each essential experience under each of the rotations, defining patient type and specifying the role of the Clinical Clerk. As shown on the sample screen below, students can easily indicate completion of an experience.
Sample screen from ECE Tracker
Student dashboards let them know which experiences they completed, and which experiences they still need. The same information is available to administrators who can see students falling behind and help them to access the experiences they need.
The ECE Tracker is an essential tool during mid- and post-rotation reviews with students.
ECE Tracker is available on the School of Medicine web portal and through the app stores for Android and Apple smartphones and tablets. Some components of apps, such as lists of the essential experiences, can be freely accessed without setting up an account.
https://play.google.com/store/apps/details?id=ca.medportal.ecetracker
https://itunes.apple.com/us/app/ece-tracker/id1244494664
McMaster Direct Observation Tool (MacDOT): The MacDOT tool for formative feedback was developed to authenticate direct observations by faculty and other observers of Clinical Clerks during specified encounters, as well as provide feedback on student performance. Each rotation has a number of observations that must be done in order for that rotation to be complete. The Clinical Clerks’ encounters with patients may include history taking, physical examination, procedures and other professional competencies such as communication and critical appraisal skills.
As with the Essential Clinical Experiences, these processes were recorded on paper, with different content and styles by each of the rotations. The first requirement was for a consistent approach for the tool. Academic Leads from different disciplines worked with the Clerkship Director, Dr. Jill Rudkowski, and Dr. Levinson’s team to determine the format and content of MacDOT.
A second requirement was a tool to facilitate quick and useful completion by observers, respecting the multiple calls on their time in clinical settings. To facilitate this, the use of the MacDOT tool is student initiated and driven:
- The student initiates the encounter, selecting the clerkship rotation and type of clinical encounter to be observed and setting this up on their device;
- The student selects the observer from a list on the app or they can add a new observer, using the physician’s license number;
- After completing the clinical encounter, the student hands the device to the observer who provides feedback through voice, typing or choosing comments from pick lists, to outline what the student is ‘doing well’ and ‘needs to work on’;
- The observer returns the device to the student;
- Based on the encounter, the clinical observer and student discuss a learning plan with one or two areas of improvement;
- The student inputs the points of the learning plan;
- The student rates helpfulness of encounter using a 5-point scale, with optional comment;
- The student submits the record of the observed encounter;
- The Supervisor and Clinical Clerk both get e-mails summarizing the encounter to ensure validity; and
- In the e-mail, students and observers are offered the opportunity to assess the app on a 5-point scale, with optional comment.
Dashboards show students which encounters they completed and what they need to complete in each rotation.
Sample screen from MacDOT
The Class of 2019 is the first cohort to use MacDOT throughout their clinical placements.
MacDOT is available through the School of Medicine web portal and app stores for Android and Apple smartphones and tablets.
https://play.google.com/store/apps/details?id=ca.medportal.macdot
https://itunes.apple.com/us/app/macdot/id1249758503
Benefits and Outcomes
The ECE Tracker serves a dual purpose: for learners and for the program to ensure all students get all the required experiences. This is key information for program accreditation.
The Tracker is not a test for students but a tool to ensure good, broad-based experiential learning.
Faculty monitor students needing essential experiences and can direct them to appropriate settings to complete their lists. This is true for both apps.
MacDOT has thousand of encounters logged and lists 2,500 observers across all rotations in clinical settings. This systematic gathering of data not only supports learning, but also is useful for program accreditation.
Ratings of benefits of feedback are consistently positive. On almost 90% of over 18,000 logged observation reports, students rated the feedback as helpful to their learning.
MacDOT is not a pass-fail rating but information for the student’s learning, as well as tracking of completion of required observations for each student in a clinical setting.
Use of the apps is a driver of change, with one example being greatly increased faculty and observer development, not only on use of the app but on the importance of directly observing medical student skills, and best practices for providing feedback.
Challenges and Enhancements
Reaching consensus on the structure and style of both tools was challenging as the different rotations used different approaches to collection and analysis.
Students requested inquiry-based questions be added to the ECE Tracker relating to specific experiences. The questions challenge them to better understand and reflect on what they did, apply their prior knowledge, and consider important clinical variations related to the specific encounter.
Feedback on the apps is ongoing, but there is room for improvement. While almost 60% of responding students and observers, based on 138 ratings, agreed they had a good experience using the MacDOT app, about a quarter of users were neutral or disagreed with that statement. Concerns included the lack of specificity in the pick list, general evaluation fatigue and lack of observer time to log input. Ongoing improvements to the apps are periodically implemented and regularly released to respond to these and other issues.
Questions arise about the extension of the apps to other uses such as post-graduate medical education contexts. However, the existing apps were developed rapidly, and it is challenging to develop more ‘generic’ types of apps, that might work across multiple situations and educational environments. For now, these apps remain ‘purpose-built’ for the McMaster School of Medicine MD Program clerkship context.
Potential
The next step is to amalgamate the ECE Tracker and MacDOT applications into a unified clerkship app, to simplify the number of apps that the students need to use.
To capture the nuances of every encounter type and individualized feedback, more specific and varied pick lists will be developed.
Greater evaluation of the impact of the apps on both student learning and observer teaching are planned.
For Further Information
Anthony J. Levinson, MD, FRCPC, MSc
Director, Division of e-Learning Innovation, medportal and machealth
John R. Evans Chair in Health Sciences Educational Research and Instructional Development
Associate Professor, Faculty of Health Sciences
McMaster University
Hamilton, Ontario
[email protected]
