Opportunity
The University of Toronto has been asking professors to look at teaching methods that emphasize active learning and transferrable skills. Professor Steve Joordens in the Psychology department at the Scarborough Campus wanted to provide his first-year students with written assignments rather than strictly multiple-choice assessments. However, having 1,800 students, either in-class or online, made the marking of written assignments prohibitive. Working with Dwayne Paré, a graduate student in Psychology, he developed peerScholar as a peer-assessment tool which provides input to grades, as well as contributing to the development of critical thinking skills and clear communication.
Innovation
Extensive research has informed the development of peerScholar. Initially, the reliability of peer assessment was examined – and studies showed that grades from peer assessment are fair and comparable to those given by teaching assistants and professors.
Students using peerScholar in Dr. Joordens’ introductory psychology class go through a number of steps:
- Students submit a short (one to two page) paper on the assigned topic, typically one that asks students to either think critically or creatively;
- Students receive papers from a random set of six students (anonymously) and provide feedback, commenting on the best part of paper, most significant area for improvement, and any other points they wish to make. Students are provided with clear direction, a code of conduct, and a rubric to guide development of their feedback;
- Students receive feedback on their paper from six students and rate the feedback for its usefulness;
- Students revise their papers, assessing the quality of the feedback and choosing which aspects to incorporate;
- Students self-assess, considering how their paper has improved through the peer assessment process;
- Teaching assistants (who can be sent the papers anonymously as well) assign grades on the final papers, as well as the quality, critical thought, and clarity of communication in the feedback given to peers and the appropriateness of the revisions the students chose to make. With the short papers, this final step takes about 10 minutes per paper.
This final step can also be done by the students as they assess the quality of the feedback they are given and explain their choices regarding the feedback. The students providing the feedback can also provide marks, which are then averaged to determine the final mark. Students who think their final mark is inappropriate can appeal to the professor; about 2% of students have appealed their marks.
In a statistics course, students are given data for a case study in peerScholar and asked to describe the analysis, questions, and tests they would use to delve more deeply into the data. Their work is then evaluated by peer assessment. Students are exposed to different ways of approaching the data through their review of their classmates’ work and also receive advice on strengthening their own work through the feedback.
Using peerScholar, students have also generated test questions for practice exams. Students submit one potential multiple-choice question and, using a rubric, assess six other questions from their peers. The most highly rated questions are reviewed for grammar and clarity and then assembled into a practice exam. To get access to the practice exam, a student has to have submitted a possible question. As encouragement, students are told that a few of these practice questions will be on the final exam. Currently the question review is performed by a graduate student; this soon will be automated.
Numerous faculties at the University of Toronto have adopted peerScholar, including economics, astrophysics, sociology, music, English as a second language, and medicine.
Outcomes and Benefits
Students benefit from three types of assessment in a peerScholar exercise, all of which have strong empirical support in terms of supporting effective learning:
- the experience of reviewing and assessing the work of their peers,
- the formative assessment of receiving comments from their peers and revising their work, and
- self assessment in which they consider improvements in their paper, their work compared to that of their peers, and how the process has made a difference to their work.
The introduction of the possibility for revision based on the feedback received from peers has been the most important enhancement to the system, as this particularly engages critical thinking to assess the feedback and communication skills to clarify and strengthen arguments.
Research on the application of peerScholar has looked at three factors – achievement of learning outcomes, peerScholar usage experience, and change in attitudes through the assignment. Post-assignment questionnaires have shown a strong positive student response in terms of their learning, with between 80% and 90% of students agreeing that they were able to improve their final paper through peer feedback they received and through considering their own paper in comparison to the ones they were assessing. The rating of the usefulness of their own and others’ comments made them more thoughtful about their tasks.
Student usage experience has also been positive, with broad agreement that peerScholar was simple to use, clear, efficient, quick, and effective for the tasks.
The third aspect of the research investigated changes in opinions on the topics assigned for the papers, such as eating meat and the war in Afghanistan, that have resulted from completing the assignments and peer assessments. Overall students showed a clear shift in their attitudes on these topics after considering them in the context of a peerScholar assignment, and this suggests that deep and relevant thought was occurring.
Challenges and Enhancements
There was early student pushback to the idea of that assessment and grading by other students would be part of final grades, both in terms of concerns for grade and the sense of students doing professors’ work. The benefits for learning have to be stressed, especially the formative evaluation that allows students to revise their papers based on the feedback. As students become more familiar with how peerScholar works, they recognize the value.
Introducing a new system always presents challenges, especially in terms of system support strategies. One current development is a library system for assignments to be stored in the cloud. Another is the development of a community of learners/users, with the developers initially being available on site at specific times to respond to questions and suggestions.
System simplicity, with a balance of functionality and ease of use, is maintained by putting in features that research has shown to be effective, but providing options. For example, research indicates that six is the optimum number of peer reviews for both the reviewers to do and the writer to receive; the system default is set at six but users can change this to between one and ten.
Potential
PeerScholar is used in numerous faculties at the University of Toronto and is being marketed throughout North American by Pearson. A version for schools, Cogneeto, is also available.
In November 2012, an upgraded version of peerScholar will be released with such features as:
- Rich comments provided by teaching assistants can be put into Google Docs and made accessible to other TAs. Students will be told that this may result in repeat comments on assignments and how they can use this for their learning.
- The reporting tool will include not only grades but also records of who assessed whom, what were comments made and other details. This is currently available from various locations online but it will now be presented in a report.
- Infographics will improve the presentation of data for both students and professors.
Created to help develop students’ writing skills, peerScholar is now being used for peer assessment of presentation skills. Students learning to make live presentations create videos using their cell phones, often doing six to ten versions before submitting the final product. These videos then go through the peer assessment process, providing a much less threatening environment for feedback and improvement than addressing a large, live audience. Other variations include peer reviews of PowerPoint slides and of thought papers on readings so that both student presenters and participants are better prepared for seminar classes.
Steve Joordens and Dwayne Paré continue to develop and improve peerScholar and welcome opportunities to talk with colleagues about expectations and suggestions for peer assessment.
For Further Information
Steve Joordens
Professor
Coordinator of Pedagogical Development
Psychology
University of Toronto, Scarborough
[email protected]
Dwayne Paré
PhD Candidate
Psychology
University of Toronto, Scarborough
[email protected]
PDF: Peer Assessment and More with peerScholar
