As a Faculty Member, What Do I Need To Know About Quality in Online Learning?

Quality in online learning is fundamentally about:

1. The quality of the learning resources (materials, activities, readings and other resources) offered to learners;
2. The quality of the learners experience of teaching and learning, which depends in part on who the students are and who is teaching them; and
3. The quality of learning outcomes.

A variety of quality frameworks and standards have been used to establish a basis for quality assurance in online learning including:

• Quality Matters^TM (QM) has been adopted by a great many colleges and universities in Canada and in the United States as a basic quality assurance process, which includes the peer review of learning materials and subsequent interactions between learners, learners and teachers and learners and the learning materials. 
• In Ontario, the Postsecondary Education Quality Assessment Board (PEQAB) developed standards to assess the institutional capacity to offer blended and online courses with the colleges and the universities (here) each having their own individual quality services. The intention, as with many other such systems, is to have quality “designed in” to all of the processes within the institution as part of a quality framework.
• Similar standards were developed in Alberta (here, especially at pages 90-94) and British Columbia.
• Other jurisdictions have also developed significant quality assurance frameworks for online learning, including Europe and California (here).

Most of these standards focus on the materials used for learning and on the nature of assessment. Increasingly, these standards have started to require an assessment of student engagement and interaction.

The Quality Standards Framework

The way in which quality assurance functions in colleges and universities is through what might be termed “inquiry”. Inquiry is the process of critical questions about a program of studies or course being answered to the satisfaction of learners, instructors, and those reviewing the experience and outcomes of the program or course. The answers to critical inquiry questions vary by course, by discipline and for different student groups. The quality assurance process requires that the questions have been asked and answered with rigour.

The standards for quality from the various quality assurance bodies, which looked at quality for online learning (see Appendix One), generally focus on these key categories and questions:

• Course Overview and Introduction – Do the learners know what to expect once they open a course? Do they understand the intent of the course, what is expected of them, what assignments and activities they will engage in, what readings and project work they will have to undertake and what the assessments rubrics look like?
• Learning Objectives – When the learners complete the course, what competencies, skill, understanding and knowledge will they have and be able to demonstrate? Does the ways in which the learner experiences assessment ensure that they can demonstrate these competencies and skills? Will the student retain the knowledge and skills developed?
• Assessment and Measurement – How will assessment and evaluation (measurement) be undertaken, by whom and against what criteria? What exactly is it that the learner has to demonstrate through the process of assessment so as to ensure success? Are the rubrics and criteria for assessment made clear to learners?
• Instructional Materials – How well designed, from an instructional perspective, are the materials the learner is asked to work with? Do they provide sufficient basis for learning, given the assessments and expectations? Are there “remedial routes” for learners who struggle with concepts, skills and competencies? How do the various technologies/software used in the course link to best convey specific skills or content? How varied are the materials – do they accommodate the different learning styles of learners? Are good and appropriate use made of open education resources and third party materials? Have all intellectual property rights been secured and used appropriately?
• Learner Interaction and Engagement – Are learners able to actively engage with the materials through simulations, problem-solving, games or other activities? Are learners able to engage with other learners in small group or project work? Are learners able to engage in dialogue with the instructor and peers?
• Course Technology – Are the technologies used in the program or course suitable for the stated purpose to foster the learning outcomes of the program or course? If preparation is required for the appropriate use of the technology (e.g. preparation for a simulation or project), is this preparation available? How reliable is the technology? Does the technology support different locations for learners – multiple sites or home based learners? Is the technology current?
• Learner Support - If a learner is struggling with concepts or skills, how is help and support provided (how fast, how frequent and how effective)? If a learner is struggling with the technology, is help readily available when the learner needs it? If a learner is “fast tracking” through a course, are they able to do so without being held up by the schedule or technology?
• Accessibility – Are appropriate provisions available for learners with disabilities? What support is provided to learners both for their study of the course and for assessment and examinations?

When you review the above quality categories, you will see that several refer to the nature of instructional design (e.g. the first five of the above). Quality is fundamentally about design and intention with respect to achieving learner outcomes: quality is designed into programs and courses; not inspected in once they are available to the public. This is why peer review of courses before they are offered to learners – a core component of the Quality Matters™ (QM) program – is such an important component of how quality has been assured for many online programs and courses. Some programs and courses are sometimes piloted with small groups of learners for credit before being offered more widely.

Differences between Quality Assurance for Face-to-Face Instruction and Online Learning Quality Assurance

Quality assurance in face-to-face instruction focuses primarily on three things:

1. The integrity and relevance of the course outline as reviewed by peers, expert learners, advisory committees and others;
2. The level and appropriateness of the qualifications of faculty teaching the course; and
3. The satisfaction students demonstrate with their experience of the course as shown in end of course evaluations. Increasingly, samples of student work are reviewed to look at the quality of student work and marking standards.

In online learning quality assurance, much more of a course is “visible” to scrutiny by third parties. The course outline remains important, but so too are all of the course materials and how they are used. Since many components of a course are “captured” for subsequent use – e.g. videoconferences and audioconferences with learners are often stored so that students can review them – these too become available for peer review. Student and faculty interaction, as well as student and student interaction, are also captured for review.

Transitioning from Face-to-Face to Online in Terms of Quality

Many involved in online learning would claim that quality review and assurance in online learning is much more comprehensive than traditional face-to-face instruction. Some faculty have seen the level of scrutiny, which is possible for their online courses, as “intrusive”, while others welcome the opportunity for peer review to improve quality of the design and development of their courses.

The key, from a faculty member’s point of view, is to become increasingly curious about “what works” best for the learning materials and type of student who registers for the course and be welcoming of the involvement of others in design and development. It is also important to think about “modalities” of learning – the different kinds of learning experiences which a student will encounter during a program or course. This is particularly important given that you may have an increasingly diverse cohort of students, bringing different experiences and expectations to the classroom.  

These modalities can include (but are not limited to) team projects, self-study, active research, games and simulations, student presentation, challenges and case studies. In classrooms, many of these can be offered “on the fly” in response to issues or concerns raised by students. In online learning, they need to be anticipated and planned for.

The key here is simple: different kinds of learning outcomes require different learning experiences to achieve them. Quality guidelines focus on outcomes and engagement. The faculty member needs to determine how outcomes and engagement will be achieved through different modalities of learning. 

Colleagues can help shape the opportunities for the use of different modalities within a course. A research study (available here) by Chao, Saj and Hamilton (2010) noted that most faculty appreciated the collaboration which quality assurance guidelines and processes encouraged.

While more time is needed for such collaboration, the end result is a high quality learning experience for the student which makes use of reliable technology, quality third party learning resources and skillfully designed activities. Rather than working alone, the faculty member can enhance their teaching and learning activities through collaboration.

Six Key Quality Assurance Issues

The current focus for quality assurance in many colleges and universities is on these critical aspects of the list above:

1. Program and Curriculum Design
2. The Delivery System (including the learning management system) and Related Pedagogy
3. Preparation and Skills of the Faculty Member
4. Assessment and Authentication
5. Student Engagement
6. The Use of Open Education Resources

While other issues arise from time to time, these six issues dominate current quality assurance for online learning discussions.

1. Program and Curriculum Design
    
   When a program quality assessment is undertaken by a body such as Postsecondary Education Quality Assessment Board in Ontario, the review team will focus on the learning outcomes intended from the program and whether or not the pedagogy adopted is likely to achieve these outcomes, given who the students are and the experience and skills of the faculty. They will also compare the program with similar programs offered elsewhere and look in depth at how outcomes are to be assessed. The design of the program must demonstrate the link between the learning process and learning outcomes.

   Increasingly, review teams are also asked to look at the ways in which technology is being used, whether the program is classroom-based or fully online. The question is: “does this program make best use of the available resources for learning”, given who the students taking this course are?

   In a recent review, for example, significant concern was expressed that the course materials made available for an in-class program relied too heavily on a single textbook and made too little use of the rich online resources available to support the learning objectives of the program.

2. The Delivery System (including the learning management system) and Related Pedagogy

   Decisions about which learning management system (LMS) a college or university uses are generally made at an institutional level – the faculty member is then asked to leverage the system in place to meet the learning needs of the course and the students taking it. This is not the place to review the learning management system available (there are over 200 of them) or their functionalities, except to observe that all of the leading LMS systems offer similar functionalities (for a review see here).

   The challenge for the faculty member is whether what they would like to do, from a teaching and learning point of view, is possible within the limits of the LMS available to them. In practice, most LMS systems now enable most of the activities faculty members use most often – the challenge comes when the faculty member wants to do something “new” or different. Then they may need to link to a web-based activity rather than embed the work within the LMS. For example, a simulation or animation is easier to link to than to “build in”.

   How students achieve the learning outcomes intended varies by experience, levels of literacy and numeracy, motivation and knowledge. The challenge for a faculty member working online is that they know little about their learners. They have to anticipate different kinds of learners in their course and design the course so that the learning needs and styles of different learners are represented in the modalities of learning and they have to ensure that these different modalities – e.g. a simulation versus a research task, a video lesson versus a group project – are the best way of achieving the learning outcomes intended. This is not an easy task.

   What many faculty members do is to develop a learning modalities grid. This grid should:

   1. List the learning outcomes intended;
   2. For each outcome, identify the most appropriate way(s) in which students will learn and achieve that outcome;
   3. Identify what resources the learner needs within the course to enable that learning; and
   4. Identify how this will be assessed. This helps with the design of the course and ensuring the best use of the functionalities of the available LMS system.
       

3. Preparation and Skills of the Faculty Member

   Some institutions, such as the University of Phoenix, require all faculty who teach online to have successfully completed a “How to Teach Online” course before they begin this work. The Ontario Undergraduate Student Alliance sees this as an essential requirement for quality assurance[1].

   Some institutions offer voluntary professional development and there are significant resources available to support the development of professional knowledge and skills for faculty members (see here for such resources).

   For professional development, a faculty member should consider an annual update of a teaching philosophy statement. The University of Toronto offers some guidelines as to how to develop such a statement, which they use as part of a teaching dossier required for tenure review and assessment. Many institutions have teaching toolkits, which help faculty members develop their knowledge and skills and provide access to best practices.

   Unless professional development is mandatory, each faculty member should consider their professional obligation for development as an opportunity to enhance their skills, knowledge and understanding. There are a growing number of post-graduate qualifications in online learning which may also be attractive to some faculty members.

4. Assessment and Authentication

   There are a number of issues with assessment for online learning which concern faculty. Here we look at three:

   1. Authentication;
   2. Participation; and
   3. The design of assessment.
       

   Authentication

   The first is simple: in an online environment, how do we ensure that the person submitting an assignment or examination is in fact the learner registered for the course? What steps need to be taken to ensure the integrity of the assessment process?

   A variety of measures can be taken to ensure the authenticity of the learner submitting assessments. These include:

   • Biometrics and Web Video Recording: Unique typing style, signature, voice or fingerprint plus targeted recording of a student in exam via webcam to verify the learner. More recently, software has been developed called BioSig-ID, which measures the unique behavioral characteristics of students using devices like a mouse or touchpad.
   • Challenge Questions: Challenge questions asked of an individual learner with a requirement for immediate or rapid response. For example, Strayer University is one institution testing the use of software that “randomly poses two multiple choice challenge questions” to students when they login to their courses. The questions are created from information found in a  database of publicly-available records, and might include things like verifying a prior street address, a telephone number, a student identification number or postal code.
   • Face-to-Face Proctored Exam: Face-to-face with government or institution issued identification with supervision by a faculty member or senior administrator.
   • Web Video Conference Proctor: Audio and videoconference proctoring via webcam. Screen monitoring service with live, certified proctors.
      

   The use of many of these tools for authentication is usually a matter of institutional policy. When a faculty member is concerned about authenticity, they should consult their administrators. What is important to remember is that authenticity issues occur in face-to-face teaching environments, especially when proctored examinations are not used.

   Participation

   A second issue raised by faculty concerns the use of “participation” in a marking scheme for a course. What should the “participation” grade be based on and how much weight should it carry for the final grade for the course? Where there is lab work, field work or project work? How will this be assessed for participation when the course is online?

   Practices vary between programs, courses and institutions. Some general guidelines are offered here:

   • If the intention of a participation grade is to require learners to engage with others and with the faculty member in an active way throughout the course, then the grade needs to be such that it secures this behaviour. Several studies show that a low participation grade (5-10%) does not lead to significant participation. The grade needs to be in the 20-30% range to ensure a high degree of participation.
   • Some programs and courses have made securing 60% or higher on the participation grade a requirement of successful completion of a course. For example, a student who scores straight As on their written assignments and tests could still fail a course if they did not secure a satisfactory participation grade.
   • What constitutes participation varies by program and course – for example, participation in a math problem solving online class might look and be different from participation in a course of psychology of motivation. In looking at grading participation, most faculty members look beyond simple numbers (how many posts did learner X post versus learner Y?) and look instead to the quality of their input, the extent to which they build on the contributions of others and the “value” they add to the conversation online. While this can be difficult to assess, several faculty have developed rubrics for doing so (for example, see here).
      

   The Design of Assessment

   Both at the program level and the course level, faculty need to give consideration to the overall “architecture” of assessment. For example, over the course of a program of studies in a specific discipline, what kinds of assessment will the learner be asked to complete and are there a sufficient variety of assessment processes to assess learning outcomes appropriately? If every course in a program uses the same form of assessment (e.g. multiple choice or the long-form essay), are we seeing a sufficient range of abilities from the learner?

   There are no “standard” answers to this question, but it is one which faculty needs to consider – quality assurance reviews at the program level will look at this question. Some suggestions for faculty working on a course within a program are:

   • Within a course and across courses within a program there should be a variety of assessment approaches.
   • Assessment should enable the assessment of both what students have learned and how they can apply that learning.
   • Assessment should not just focus on learning outcomes, but also on the learning process – learners need to demonstrate that they know how to learn in relation to the discipline they are studying so that they can maintain their learning as new knowledge and understanding emerges in that discipline.
   • Assessment rubrics and processes should be transparent and fair.
   • Assessment is of two kinds – formative (helping the learner see where they are in their learning through feedback) and summative (a grade which counts towards the final mark for the course). In online learning environments, both are important and learners expect feedback from a formative assessment before they face a summative assessment.
   • Assessment can look both at what an individual is able to demonstrate in terms of knowledge, skill and understanding or what a team can demonstrate – assessing team performance is also possible using simple assessment tools.

   Faculty members should review the assessments used in related courses to the ones they are developing so as to ensure that learners experience both variety of forms of assessment and consistency in terms of the rigour of assessment. Quality assurance reviews of assessment will look at the rubrics, forms of assessment and rigour by sampling learner work and comparing that work with the rubrics established for the course.

   Developments in Online Assessment

   Increasing use is being made of machine intelligent assessment systems, which permit not only rapid development of complex assignments (not just multiple choice, but short answer questions, essay type questions and other assessment forms) with automated marking, linking to remedial course materials. A learner who secures a low grade on one or more parts of the assessment is directed to appropriate remedial learning materials. It is called “adaptive assessment”.

   Adaptive assessment can go further: based on learner responses, the computer program adjusts the difficulty of questions throughout the assessment. For example, a learner who answers a question correctly will receive a more challenging item, while an incorrect answer generates an easier question. By adapting to the student as the assessment is taking place, these assessments present an individually-tailored set of questions to each student and can quickly identify which skills students have mastered.

   Emerging quality standards see such assessment practices as increasingly necessary, given the range of learner abilities present in our colleges and universities. Such adaptive assessment provides both grades and feedback, but also links the learner to appropriate online materials for remediation and additional study. Any alternative assessment methods need to demonstrate this same functionality.

5. Student Engagement

   Student engagement is regarded, given the research on learning outcomes, as the best predictor of student performance and success. This appears to be the case whether the learning is classroom-based, hybrid, blended or online.

   The National Survey of Student Engagement, which many Ontario colleges and universities are participating in, measures learner engagement by focusing on these key characteristics:

   • Academic Challenge – Does the learner demonstrate, through their participation and assignments, higher order learning, reflective learning, an ability to integrate ideas concepts of skills from within this and other courses, quantitative reasoning and conscious use of different learning challenges?
   • Learning with Peers – Does the learner engage with other learners in collaborative and reflective learning and can they initiate and engage in discussions with diverse others?
   • Experience with Faculty – Does the learner engage, both on request and on their own initiative, with faculty? Does the faculty member create opportunities through their online instruction for such interactions?
   • Campus Environment – In the online and campus environments, are learning interactions (formal and informal) encouraged and enabled? Is there a supportive environment for engaged learning at the institution?

   A range of best practice examples and supportive learning materials are available to faculty in support of this framework for engagement (see here for these resources). Also available are comparative data showing how a variety of institutions and programs are doing against the same criteria, though not all of these data relates to online learning.

   To ensure engagement, a systematic review of the course before it is offered, and after the first offering, is undertaken of the key points in the learning experience students are expected to be engaged, the variety of kinds of engagement opportunities afforded and the “value” of these engagement activities, given the intentions the faculty member has for the course.  

   Once the course is offered, a systematic approach to evaluating whether these “points of engagement” worked in the way intended is usually undertaken and adjustments made to continuously improve the effectiveness of learner engagement as designed into the course.

6. Use of Open Education Resources

   One of the remarkable developments over the last six to eight years has been the growth and development of open education resources (OERs).

   These are course materials – entire courses, course components, simulations, games, activities, animations, learning designs – which are available for use under an “open source” licence. Faculty members developing a course are able to use these materials as part of their own course or as the course which they then support. 

   You can find catalogues of these resources – individual course components or substantial parts of courses – at Merlot II (here), Open University (here), MIT (here), the OER Commons (here) or look through a compendium of such resource sites at the Commonwealth of Learning (here).

   The quality assurance question which needs to be asked about OERs, and other third party materials, is how well they are integrated into a designed experience for the learner: just pointing to the URL for the materials and saying “look at these” is not a way of ensuring the effective use of OERS or third party materials. Learners need to know why they are reviewing or using them, what they are supposed to do when they use them and what learning is intended to be associated with them.  

   There are two “streams” of quality-related considerations which faculty need to consider before using OER materials in their course design and development.

   The first stream refers to basic quality. These questions apply:

   • Accuracy and validity – Are the materials correct, accurate and valid? Just because they are freely available does not mean to say that they are “valid”. For example, are they accurate given recent developments in the discipline? Do they reflect current understanding and evidence?
   • Reputation of author/institution – Do the materials come from a reputable source and a credible institution?
   • Standard of technical production - Are the materials or resources well-produced and will they integrate well with learning management systems and technology being used in the institution?
   • Accessibility – How easily accessible are the resources? Are passwords needed, firewall adjustments required or other technical considerations? Will these materials be appropriate for learners with disabilities?
   • Fitness for purpose – Do these materials fit with the design and intention for the course? Are they are 100% fit with the design or a partial fit? Are they suitable for individuals with different levels of literacy, numeracy and technical skills?
      
     These are very practical questions, but need to be addressed before the faculty member turns to the second stream of quality questions, which are:
   • How will the OER resource you are looking to use be used by learners? What guidance and support do they need to make the best use of this resource?
   • How will you assess the use of the OER resource by learners? In addition to looking at what the learners gained in terms of knowledge, skill and understanding, will you also look at how the OER materials enabled these gains?
   • How will you ensure that the OER materials in the course remain current? A video of the CEO of Microsoft talking in December 2013 will look dated in March 2014 after a month of the new CEO’s “reign”.
     
     These questions focus more on integration of the OER materials into a course design and on sustaining their value over time.

Shifting Quality Assumptions

Those who develop policy and practice for quality assurance in post-secondary education are engaged in a shift from an “input” view of quality (“it is about the design of the course”) to an “input and process” view of quality (“it is about the course and the experience of learning and the competencies it produces”).

This shift is adding a stronger focus for faculty on student assessment and engagement, both as part of the design of the program or course, and as part of the practice of teaching and learning, which is why assessment and engagement have been a strong focus in this introductory material.

Design of the Learning Process and Assessment is the Key to Quality

The key message for faculty is that design is the key to quality. How a course is designed - both in terms of the knowledge and skills it seeks to secure for learners, but also how it engages those learners – will determine whether or not the course is a “quality course”.

To ensure the course does what it says it will do in terms of knowledge and skills, peer review is increasingly seen as the approach needed. A working draft of the finished course is sent to independent reviewers for comment and feedback on both the content and the learning design. Student feedback from past versions of the course or from a review of the draft in preparation by select students can be most valuable.

Quality is not an abstract construct – it is a pragmatic one. Quality in online learning is about the design, the experience and the outcomes.

References

Chao, I.T., Saj, T. and Hamilton, D (2010) Using Collaborative Course Development to Achieve Online Course Quality Standards. International Review of Research in Open and Distance Learning, Volume 11 (3) available at http://www.irrodl.org/index.php/irrodl/article/view/912/1644 (Recovered January 22, 2014).

Appendix One: Quality Standards, Frameworks and Resources