Opportunity
Many educational institutions in Europe develop and offer open education and, in particular, Massive Open Online Courses (MOOCs) as strategies for extending their reach, enticing students to enrol in the university, offering learning opportunities in their national languages to international participants, providing specific training opportunities, and as components of degree as well as open courses.
A group of academics and researchers from institutions across Europe, including Greece, Portugal, and France, with coordination by the Welten Institute of the Open University of the Netherlands (OUNL), formed MOOQ, the European Alliance for the Quality of Massive Open Online Courses to build a tool to support quality improvement in MOOCs. Their investigation found most MOOCs have similar and limited designs and structures, lacking in opportunities for online coaching, collaboration, personalization and feedback.
To support the development of a guide supporting quality in MOOCs, a proposal was submitted to the Erasmus+ Programme of the European Commission for funding to construct a MOOC Quality Reference Framework (QRF). Funding was received for a three-year research project which was recently completed.
Innovation
The MOOQ Alliance, under the leadership of Dr. Christian M. Stracke, ICDE Chair in OER and Associate Professor for Open Education and Innovation at OUNL, is committed to improving MOOCs, specifically through the development and integration of quality approaches, innovative pedagogies and organizational mechanisms, with a strong focus on the learning processes, methodologies and learner support.
The first objective of MOOQ was the development of the Quality Reference Framework that progressed through four significant phases:
- An extensive literature review, analyzing results of research studies on MOOCs, whether case studies or theoretical considerations.
- Three online surveys distributed to learners, designers and facilitators (such as tutors and course mentors) on their experiences with MOOCs, including strengths, weaknesses, and potential. More than 250 completed questionnaires were analyzed as the basis for the draft Quality Reference Framework.
- Eight workshops at international conferences with hundreds of participants who reviewed the initial QRF, offering feedback on the contents related to five components of MOOC design and delivery – analysis and planning, design, implementation, realization and delivery, and evaluation.
- Interviews with 36 designers, facilitators and providers of MOOCs. The focus was on the strategic and business perspectives of the providers, primarily open education providers, especially universities offering their own MOOCs using Moodle or another open access platform. The detailed interviews ranged from 1 to 3 ½ hours.
This extensive research and analysis resulted in the first publication of MOOQ: The Quality Reference Framework (QRF) for the Quality of MOOCs.
The Quality Reference Framework consists of an extensive chart integrating and outlining three dimensions:
- Phases and processes: detailed descriptions of the five phases of MOOC design and delivery – analysis, design, implementation, realization and evaluation – and of each of the processes involved in each phase and each of tasks necessary for the completion of each process;
- Perspectives: notations indicating the perspective or focus of each process, whether Pedagogic, Technical or Strategic; and
- Roles: indications of whether the designer, facilitator, and/or provider has a role in each phase and process, specifying who has key responsibility and who has a supporting role.
For example, under the first phase, Analysis, one of the processes listed is Stakeholder Identification, which is classified as involving both Pedagogical and Strategic perspectives. Three tasks are identified:
- Identify the internal and external stakeholders;
- Ensure each of the key stakeholders (e.g., content provider, designer, pedagogical and technical facilitator) is represented in MOOC development and design team; and
- Identify target learners and groups in relation to content, IT competency, prior experience in online and e-learning.
For each of these tasks, the provider is identified as having key responsibility, with the designer having a supporting role.
Overall, the chart outlines close to 140 tasks that can be undertaken to ensure quality in the design and development of a MOOC. However, it is assumed designers, facilitators and providers will select the phases, processes and tasks most appropriate to their institutional objectives, learners and context. Dr. Stracke describes the purpose of the QRF as: “a list of possibilities for application; an inventory of potential to highlight all dimensions to consider in the design and development of high-quality MOOCs”.
The second key component of the publication is the QRF Quality Checklist, which asks critical questions related to the phases, processes, perspectives and roles in the design and development of MOOCs.
An example from the Analysis phase focuses on the process of Definition of Objectives, which offers the key questions:
- What are the general pedagogical objectives (e.g. higher quality, large scale, learner orientation, thematic focus, collaboration)?
- What are the general technological objectives (e.g. new functions, inhouse, outsourcing)?
- What are the general economic objectives (e.g. profit, cost-covering, charitable)?
- What are the general strategic objectives (e.g. learner support, future learners, marketing)?
The Quality Checklist differs from the QFR as it recommends all questions be answered, as providing essential input to quality design and development.
The third key component of the publication is the list of QRF Key Quality Criteria which provides all potential quality criteria and aspects to be addressed. As the QRF Quality Checklist is an instrument for beginners, the QRF Key Quality Criteria address advanced MOOC designers, facilitators and providers to support evaluation and improvement of their design, implementation and delivery of MOOCs.
The Quality Reference Framework was submitted to the International Standardization Organization (ISO) and the European Standardization Organization CEN with a common proposal to become the first quality standard in the field of MOOCs. The QRF is built using the reference model from the unique ISO quality standard for e-learning (ISO/IEC 40180), first developed in 2005 and revised in 2017.
During the development process, the draft QRF was applied to two MOOCs at the Hellenic Open University in Greece. The MOOCs were on software engineering and IT programming, providing basic skills for the Computer Science degree program and used as introductory courses for new entrants. The course developers, one a member of the MOOQ research team and one with no familiarity with the QRF, provided feedback which was then supplemented by a much more extensive review of the QRF at a MOOC Conference in Athens in July 2018.
The European Commission designated the Quality Reference Framework as a “most innovative project” funded through Erasmus+.
Benefits and Outcomes
For learners, the QRF gives them the information they need to determine which MOOCs fit their needs, as it points out all the key features for MOOC quality. Learners can assess potential MOOCs according to their own learning strategies and needs, considering factors such as availability of feedback and collaboration; access to certification or credit; all-at-once content delivery or linear, timed release of modules over the week of the course; and pedagogical design features.
Designers, especially those new to MOOCs or online learning, can use the Quality Checklist to ensure they address all important issues of design and refer to the QRF to determine key quality criteria to be applied.
The indicators for all phases, processes and tasks clarify the differing roles and responsibilities of designers, facilitators and providers. The PDF version of the publication indicates all roles and responsibilities. The online version can be sorted by Role so designers, facilitators and providers are presented with charts specifying only those tasks for which they have key responsibility or a supporting role.
Challenges and Enhancements
The key challenge in the development of the tools was ensuring contact with and detailed input from the key groups of learners, designers, facilitators and providers. The workshops at international conferences were particularly helpful with this task. As well, the project groups worked in collaboration with the United Nations organisations of UNESCO, International Labour Organization (ILO) and Food and Agriculture Organization (FAO) and received support from the International Council for Open and Distance Education, European Distance Education Network, European Association of Distance Teaching Universities and other organizations.
Potential
With the publication of The Quality Framework (QRF) for the Quality of MOOCs by MOOQ, an online community called MOOC Quality Network encourages participants to discuss, offer opinions and suggestions on the Framework and connect with others concerned with issues of quality in MOOCs. All MOOQ research results and publications are available online with a free and open license for sharing, re-using and adapting for individual purposes and situations.
For Further Information
Dr. Christian M. Stracke
ICDE Chair in OER
Associate Professor for Open Education and Innovation
Welten Institute - Research Centre for Learning, Teaching and Technology
Open University of the Netherlands (OUNL) - www.ou.nl
Heerlen, The Netherlands
[email protected]
