Improving Student Exam Results with ProfBot: An Individualized Approach to AI Tutoring

Challenge

Sean Wise, a Professor of Entrepreneurship and Innovation at Toronto Metropolitan University in Toronto, Ontario, teaches courses on introductory entrepreneurship for non-business students and start-up creation. Each class enrolls between 100 and 300 students. Professor Wise was concerned that many were unengaged, not actively present in the class and/or engrossed with their phones.

He determined to meet the students where they were – by creating a Chatbot tutor that they could use on their phones, offering asynchronous support for exam preparation that would be available 24 hours a day in the period preceding each exam. A Chatbot is a Generative Artificial Intelligence (GenAI) driven tool that provides answers to users’ questions in real time.

Among the concerns about GenAI, the possibility of students finding and using inaccurate and offensive material is paramount. Professor Wise developed ProfBot so that students could benefit from individualized, tailored tutoring based strictly on select course materials provided by the professor.

Experimentation

Early AI systems could assess student responses to quizzes but were limited as the student response had to replicate the expected answer. Earlier systems could not identify that words like ‘house’ and ‘home’ could mean the same thing. By 2021, advancements allowed AI to recognize semantic similarities between words and accept more varied expressions of ideas, making it far more effective in comparing student-generated responses with expected ones.

Building on these developments, Professor Wise developed ProfBot, a GenAI-based tutoring tool students can choose to use to practice answering questions to prepare for exams. He secured technical and ethical approvals from Toronto Metropolitan University and received a grant to fund the project.

ProfBot offers students access to questions and responses from previous exams, uploaded by Professor Wise. To ensure focus and integrity, Professor Wise designed ProfBot within ‘guard rails’; students are limited to the content provided by him - they can not access external sources, nor can they submit self-generated questions. All content is professor-approved, either self-created or sourced from trusted sources such as textbooks.

Upon logging in, students are asked five questions, and they submit their responses sequentially. ProfBot then compares the student input to the answers and grading rubric provided by Professor Wise and provides immediate feedback that he describes as “polite, encouraging and kind”. The system highlights strengths, identifies areas for improvement and suggests rephrasing or elaboration. Students may also be referred to specific sections of course content for review. They can re-submit revised answers.

The app evaluates content using the benchmarks answers and assessment guidelines provided and does not assess spelling or grammar, emphasizing meaningful engagement with learning.

Results

ProfBot offers personalized tutoring based on the course content to students on demand, prior to exams using their phones or other technology. Its accessibility is limited to those enrolled in the associated class, using university email addresses as access points.

Students have access to ProfBot 10 days before each exam, 24 hours a day when the professor and teaching assistant are unavailable. Between 70% and 80% of the students In Professor Wise’s courses used it more than once and many returned multiple times. More than 70% of the usage was in the last 72 hours before the exam, with more than half in the last 48 hours. Thousands of quizzes were completed before each exam. Limited data is collected anonymously to preserve student privacy.

Professor Wise monitors aggregate trends, such as common areas of difficulty, and integrates advice on course content to be consulted. This advice is integrated into ProfBot, not directed to students having issues as this specific information is never accessible.

Students were reluctant to use ProfBot until they were reassured that their individual answers and activity were not reported or in any way accessible to professors or teaching assistants. To alleviate concerns about surveillance, all personal information concerning logins and responses is erased immediately after use. Only generalized outcomes for each question and overall usage numbers are recorded anonymously. Professors never have access to individual student participation or results.

Students can take screenshots of their results before deletion.

Overall exam results improved with the introduction of ProfBot by an average of 5%. Some student marks increased as much as 27%. Anonymous student assessments of ProfBot found that 80% considered it ‘valuable’, ‘very valuable’ or ‘extremely valuable’.

Next Improvement Steps

Professor Wise points out that ProfBot supports two distinct types of questions:

Explicit questions: Straightforward prompts like “What are the four key characteristics of an elevator pitch?”
Tacit questions: Open-ended prompts such as “Write a sample elevator pitch for Netflix.”

Initially, ProfBot offered only explicit questions. The ability to answer tacit questions, which requires evaluation of nuanced and creative responses, was added in a second development phase. For these, Professor Wise provides criteria on which to judge the student response, including an example of an answer and the essential content to be included.

Additional Phase 2 enhancements included providing marks alongside feedback and allowing sequential feedback for each question, rather than all five at once. These improvements were in response to student requests.

Professors have numerous concerns about GenAI and its current and potential impacts on education, including complying with institutional policies, student use of GenAI for assignment generation and the false, harmful and offensive content available. Consequently, many instructors are reluctant to adopt GenAI.

ProfBot is designed to comply with strict institutional policies as it is a “very gated app” that cannot make things up, source offensive material or hallucinate answers. It is a tightly controlled, professor-driven tool for enhancing student success.

ProfBot is not a shortcut for students unwilling to engage with their coursework as it is not an easy fix for lack of attention and studying. It is most effective for learners seeking focused exam preparation.

Potential

ProfBot is available free of charge for all educators to use by logging in at ProfBot: Your Personal Digital Tutor. It is designed to be aligned with college and university institutional policies on ethics, privacy and other GenAI concerns.

Professor Wise has several strategies for improving ProfBot. Its capacities can expand to include opinion-based question and collaborative content development among educators in the same subject field. He does not characterize ProfBot as a “Wow” GenAI application but as a straightforward, professor-controlled tool that can increase student success in universities, colleges, adult education and corporate training.