Challenge
In November 2023, Jennifer Lucas, a professor at the SKEMA Business School, an international business school in France, was asked to take on the role of Multicampus Course Manager for Academic and Professional Writing, including directing the compulsory course in English-language writing skills. The course is offered at SKEMA campuses in China, Brazil, United States and three campuses in France and is part of a Master’s in Management program.
To modify aspects of the course, Professor Lucas worked with Irina Otmanine, an Instructional Designer in the Pedagogical Innovation Service with expertise in course design as well as research. The revised course was to be launched in mid-January 2024, requiring rapid design and implementation of changes. The management and staff of SKEMA provided ongoing support for the initiative.
The main modification was to introduce Generative Artificial Intelligence (GenAI) as a component of feedback given to students on their writing assignments as a complement to the comments provided by faculty and peer feedback. Accompanying this change would be research to assess student skill improvement and reactions to the feedback, including that provided by GenAI.
The introduction of GenAI had multiple purposes:
- To provide students with experience using GenAI for professional and learning purposes by developing skills in prompting and assessing the quality of GenAI’s contribution in writing, speaking and other aspects of language use.
- To model how students can use GenAI for its strengths while maintaining their personalized writing styles rather than the sometimes-generic styles offered by GenAI. Professor Lucas characterized this as recognizing “what the human adds to the writing style”.
- To train students early in their program on how to effectively use GenAI so they can develop the skills that enable them to apply it throughout their degree programs, as well as their professional and personal lives.
- To move students from being accumulators and consumers of multiple apps to “digital natives” able to use GenAI for not only content production but also for all aspects of learning through the development of GenAI literacy and critical thinking skills.
Experimentation
In January 2024, the newly revised compulsory course in English language writing skills was offered to 670 students on six international campuses by 16 instructors in face-to-face classroom settings.
In preparation for this, Professor Lucas and Ms. Otmanine introduced faculty at each campus to the new feedback structure and its role in teaching and learning. Class time was re-structured to include GenAI.
The course is taught in 12 sessions of 1 1/2 hours each. The classes and accompanying online exercises focus on aspects of writing such as clarity, grammar, structure, vocabulary, synthesis, and referencing.
In 10 of the twelve sessions, the class began with the students receiving GenAI prompts that outlined a topic, accompanied by support materials. Students wrote for fifteen minutes and reviewed their papers for five minutes. Topics included issues such as Diversity, Equity, and Inclusion and Ethics in Decision-Making. Students were encouraged to use the content resources offered through GenAI in addition to including their own experiences and opinions.
Each student submitted eight writing exercises. One submission was assessed by faculty; one was peer assessed and a third was assessed by GenAI. This multi-layered feedback approach offered students diverse perspectives on their work.
The software ChallengeMe - Learning and assessment solution facilitated the evaluations by providing each student with anonymous feedback from peers and faculty.
To create the GenAI feedback function, Professor Lucas adopted the prompts by Escalante et al (2023) to outline to GenAI what she was looking for, including the provision of feedback, not a grade, and the creation of a grid for each student showing errors and examples of how they might be corrected.
Results
Three questionnaires were made available on ChallengeMe for voluntary completion – at the beginning, mid-point, and end of the course. Completion rates were 70% of the 670 students for the first one and 72% for the third. Students were asked to evaluate their writing skills on aspects such as structure, provision of examples, concluding sentence use, professional style and tone, spelling, grammar, punctuation, and vocabulary. Students reported increased capacity for each skill, with professional style and tone having the greatest improvement.
The questionnaires also included questions on how helpful and constructive the students found the faculty, peer, and GenAI feedback, as well as self-assessment. As shown in the chart below, GenAI had the lowest rating of 6.9 on a 10-point scale at the beginning of the course but rose to an 8.1 rating in the final questionnaire, indicating that students became more aware of its benefits. The blue lines indicate the results from the questionnaire at the beginning of the course, the red from the mid-point and the green from the final questionnaire.
Students also provided written input on GenAI’s strengths and limitations. They commented on how they were now able to use GenAI to make their writing more coherent and appreciated how it supported their capacity to learn on their own and offered responses to numerous and detailed questions.
By the end of the course, students had enhanced appreciation of the value of GenAI feedback but stressed that it should be accompanied by feedback from peers and instructors.
The data collection and analysis for this project were done within the framework of the Scholarship of Teaching and Learning (SoTL) approach developed and supported by the Pedagogical Innovation Service and SKEMA Business School since 2021. SoTL encourages SKEMA faculty to use research methods to analyze and enhance student learning.
Next Improvement Steps
Students described some of the limitations and challenges they found with GenAI, indicating an enhanced critical awareness of its capacities. They found that GenAI would label correct statements or word usages as errors, suggest deletion of repetition used for emphasis, and offer vocabulary changes for words chosen by students that fit their style and story. Students became aware that they could assess and then modify or reject feedback from “quality” sources.
Students reported that GenAI feedback was often too general, lacking details and examples. Text generated or corrected by GenAI “is not natural anymore so it loses its authenticity.” One student commented that “AI tools would write all my work in a different way and I would not recognize my writing style anymore.”
Potential
As the course was being taught in multiple locations, it was not possible to make any alterations during its delivery. Some of the changes being considered for the course to be offered in Spring 2025 include a reduction in the number of writing exercises with an increased focus on the feedback. More GenAI feedback will be introduced gradually. Students may be offered enhanced training in GenAI skills, especially prompting, so that the tools are useful to them beyond the writing course.
Professor Lucas was awarded the SKEMA Business School Faculty Innovation Prize, voted on by faculty, for the re-design of the Writing Skills course, as well as the Scholarship of Teaching and Learning Award. GenAI is seen as a crucial tool for SKEMA to use for teaching and learning to ensure graduates are well equipped for workplace demands and possibilities.
For Further Information
Jennifer Lucas
English Lecturer
Multicampus Course Manager (Academic and Professional Writing)
SKEMA Business School, France
[email protected]
Irina Otmanine
Instructional Designer
Pedagogical Innovation Service
Innovation and Learner Experience Department
SKEMA Business School, France
[email protected]
Sources
Escalante, J., Pack, A., & Barrett, A. (2023). AI-generated feedback on writing: insights into efficacy and ENL student preference. International Journal of Educational Technology in Higher Education 20(1), 57. https://doi.org/10.1186/s41239-023-00425-2
