Introduction

Doubts remain regarding the quality of online courses despite the continued growth in online higher education (Allen and Seaman 2016). Within the academic community, only 29.1% of academic leaders “believe their faculty accept the value and legitimacy of online education” (Allen and Seaman 2016, p. 6). This statistic is disconcerting, particularly considering that online course offerings represent the fastest growing sector of higher education, accounting for three-quarters of all enrollment increases (Haynie 2015; Means et al. 2010). The American Federation of Teachers issued a report indicating the importance of high standards of good practice; they stated that a lack of quality controls could jeopardize an institution’s effort to implement a successful online education program (Feldman et al. 2000).

We could not identify a clear set of “best practices” for online courses through our literature review. Allen and Seaman (2016) note, “It is always hard to judge the quality of something where there is no universally agreed upon metric” (p. 29). Research indicates instructional design quality guidelines are valued by instructors (Chao et al. 2010). Researchers described institutions implementing quality programs: Heaton et al. (2002) performed a case study that described the implementation of a quality control program for Marshall University’s online graduate program. Herron et al. (2012) described Embry-Riddle Aeronautical University-Worldwide’s focus on development, delivery, and evaluation to promote quality for their 36,000 online students. While these efforts are individualized for each institution, both sets of researchers suggested the importance of a course review and approval process, as did Chua and Lam (2007) in their description of the quality assurance process at Universitas 21 Global. Other literature supports the use of an evaluation instrument to promote quality in online courses (Chao et al. 2006; Little 2009; McGahan et al. 2015). Furthermore, research indicated a link between applying quality standards to online courses and student learning outcomes (Parscale et al. 2015).

Designers of online courses (e.g., instructors and instructional designers) may rely upon course evaluation instruments to design and assess quality (Kleen and Soule 2010). In our review of literature, several of the papers discuss Quality Matters (QM), a “faculty-centered, peer review process that is designed to certify the quality of online courses” (MarylandOnline 2016). Legon (2015), a former director of QM, indicated that the adoption of the QM Rubric can encourage continuous improvement in online courses by encouraging course design consistency and fostering dialogue about what constitutes quality practices. While QM is frequently cited, there are other evaluation instruments used to promote quality online courses. In California, for example, the community college system, California State University system, and University of California system each have their own method of evaluating online courses. The learning management system, Blackboard, offers the Blackboard Exemplary Course Program with “the goal of identifying and disseminating best practices for designing high quality courses” (Blackboard 2016). Recently, the State University of New York (SUNY) Center for Online Teaching Excellence (COTE) developed an instructional design rubric to promote quality and accessibility in online courses. However, an analysis of these instruments was not found in our review of literature. In this research, we review national and statewide course evaluation instruments used in higher education. The following research questions guided our study:

  1. 1.

    What are the characteristics of the national and statewide evaluation instruments for online courses?

  2. 2.

    What do national and statewide evaluation instruments for online courses identify as common standards to guide design of online courses?

  3. 3.

    What are the unique features of the identified national and statewide evaluation instruments for online courses?

Method

Data Sources

We conducted a search for national and statewide higher education online course evaluation instruments. We used search terms (“quality assurance” “course design rubric” “course design checklist” “instructional design rubric” “instructional design checklist” “course design standards” “instructional design rubric” and “higher education online course design”) on the Google search engine to look for instruments. During our search, some evaluation instruments pointed us to other instruments. For example, the California State University (CSU) Quality Online Learning and Teaching Rubric (QOLT) listed several rubrics that helped to shape its development (e.g., Quality Matters and the Quality Online Course Initiative) (California State University 2015c). We only examined instruments that were publicly available. We reviewed the instruments and included in the final analysis the evaluation instruments that met the following criteria: 1) evaluate design of higher education online courses; 2) support student success; 3) have national or statewide influence; 4) were published after 2006; and 5) are currently in use. This process yielded a total of six evaluation instruments for review and analysis:

  • Blackboard’s Exemplary Course Program Rubric (2012)

  • California Community Colleges’ Online Education Initiative (OEI) Course Design Rubric (2016)

  • The Open SUNY Course Quality Review Rubric (OSCQR) (2016)

  • Quality Matters (QM) Higher Education Rubric (2014)

  • Illinois Online Network’s Quality Online Course Initiative (QOCI) (2015a)

  • California State University Quality Online Learning and Teaching (QOLT) (2015b)

Data Analysis

Initially, we evaluated the OEI Rubric, since it was the shortest instrument of the six evaluation instruments. Each standard was noted, and then additional instruments were coded against these standards by comparing phrases used in the instruments. While writing this paper, a new version of the OEI Rubric was released (OEI post-Nov. 2016). The new OEI Rubric was then coded and compared to the other state and national instruments.

Findings

Characteristics of National and Statewide Evaluation Instruments for Online Courses

We first examined the background information of the six identified national and statewide evaluation instruments. We then compiled a list of characteristics about the instruments based on information gleaned from the Internet (Table 1):

Table 1 Characteristics of Evaluation Instruments
Full size table

The Blackboard Exemplary Course Program Rubric (Blackboard)

The Blackboard Exemplary Course Program Rubric focuses on course design, interaction and collaboration, assessment, and learner support (Blackboard 2012). The Blackboard Exemplary Course Program Rubric is available on the Internet for instructors to review their own courses at no charge. It was developed by the learning management system company Blackboard as a way to acknowledge high quality courses. The Blackboard Exemplary Course Program Rubric is typically used to evaluate existing courses. The Blackboard evaluation instrument can be used in a peer review context, and as a way for users of the Blackboard learning management system to demonstrate superior achievement in online course design. A community is available online for educators, developers, and instructional designers to discuss quality course design, rooted in the Blackboard Exemplary Course Program (Blackboard 2016).

This instrument was established “with the goal of identifying and disseminating best practices for designing high quality courses” (Blackboard 2016). Blackboard (2012) encourages the sharing and remixing of its rubric, provided attribution is given, for non-commercial purposes.

The California Community Colleges Course Design Rubric for the Online Education Initiative (OEI)

The OEI Rubric focuses on establishing standards for “course design, interaction and collaboration, assessment, learner support, and accessibility” (California Community Colleges Chancellor’s Office 2016b). It was developed through a partnership between the Foothill-De Anza Community College District and the Butte-Glenn Community College District, as the result of a $56.9-million grant from the California Community College Chancellor’s Office (California Community Colleges Chancellor’s Office 2016a). The OEI Rubric was designed to help California community colleges promote quality learning and conform to state regulations (Lorenzo 2014). Individual community college campuses, within the 113-college system, are urged to develop online course design standards for their campus (California Community Colleges Chancellor’s Office 2016b), or adopt the OEI Rubric. The OEI Rubric is available for public use under Creative Commons licensing. Workshops are offered to California community college instructors to understand the components of the rubric and apply it to their online courses. Instructors wishing to become peer reviewers must complete peer online course review training. The OEI Rubric is used by trained peer reviewers to approve courses for deployment in the online course exchange system.

The California Community College Online Education Initiative supports the Canvas LMS as a common platform for California community colleges and provides online learner readiness tutorials for students. In addition, a separate rubric is available that provides quality standards for instruction of online courses (California Community Colleges Chancellor’s Office 2016b).

The Open SUNY Course Quality Review Rubric (OSCQR)

The OSCQR standards focus on course design from the perspective of the Community of Inquiry model. OSCQR concentrates on helping instructors and instructional designers improve the quality and accessibility of online courses (Online Learning Consortium 2015). It was developed by the Open SUNY Center for Online Teaching Excellence (COTE). It is available for public use as a Google form, a Google spreadsheet, or as an interactive self-assessment rubric. OSCQR can be used during course design and after the course has been deployed. OSCQR is supported by video tutorials. SUNY also provides a website with explanations and examples for using the rubric (see https://bbsupport.sln.suny.edu/bbcswebdav/institution/OSCQR/OSCQR-3.0-Explanations-And-Examples.html).

OSCQR emphasizes continuous improvement. Each OSCQR rubric indicates the estimated time needed for revisions (based on the degree of revision necessary), and generates an action plan. Each standard on OSCQR includes a link that provides an explanation and additional resources. In addition, OSCQR offers a dashboard to institutions that displays data to support faculty development and provides “tracking of course design issues and trends” (State University of New York 2016). The OSCQR dashboard, like the OSCQR rubric, can be customized. It is the only evaluation instrument we reviewed that invites suggestions and contributions for future iterations (State University of New York 2016).

Quality Matters Rubric (QM)

The QM Rubric focuses on the alignment of learning objectives, assessments, learner interaction, and course technology (Maryland Online, Inc. 2014). It was originally developed through a Department of Education Fund for the Improvement of Post-Secondary Education (FIPSE) grant awarded to MarylandOnline consortium (Shattuck et al. 2014). The QM Rubric is part of a national subscription-based program that involves professional training. The QM organization reports “825 subscribing educational institutions and 160 individual subscribers” (Shattuck et al. 2014, p. 26). While the instrument is available for download from the Internet, the QM Rubric “cannot be used without the explicit written permission of MarylandOnline, Inc.” (MarylandOnline 2016). The QM organization offers professional development to train reviewers to focus on course design, course improvement, and achieve certification. Certified (i.e., trained) course evaluators use this instrument to determine how closely an online course meets the QM’s standards of quality.

MarylandOnline, Inc. (2016) provides an online research library that offers support for the QM standards. This research was used in the development of the rubric, along with input from course developers and instructors. To keep current, the QM Rubric is examined and updated, typically on a three-year cycle (Quality Matters 2016). The use of the QM Rubric and program is intended to encourage conversations about quality among colleagues within institutions and encourage consistency in online course design (Legon 2015). The QM Rubric is frequently linked to quality in higher education online course literature (e.g., Legon 2015; Lowenthal and Hodges 2015; Shattuck 2010).

Illinois Online Network’s Quality Online Course Initiative (QOCI)

QOCI’s emphasis is on helping faculty develop quality online courses and identify best practices (Illinois Online Network 2015b). It was developed by the Illinois Online Network (a statewide faculty development initiative) and the Illinois Virtual Campus for colleges and universities in the state of Illinois. It is available as a rubric (with checkboxes for evaluation and room for comments) or as a checklist under a Creative Commons license. Designers can use QOCI when designing, redesigning, or evaluating an online course (Illinois Online Network 2015b). QOCI is also used to recognize faculty, programs, and institutions that are creating quality courses (Illinois Online Network 2015b).

California State University Quality Online Learning and Teaching (QOLT)

QOLT focuses on helping faculty and instructional designers “more effectively design and deliver online-blended courses” (California State University 2015c) by identifying exemplary practices. It was developed at San Francisco State University for the California State University (CSU) system. It is available as a viewable/printable Word document for self and peer-evaluation under a Creative Commons license. A version of the instrument is also available for students to anonymously evaluate courses (California State University 2015b). CSU faculty are encouraged to join the QOLT community, take part in QOLT workshops and training, and engage in self review and course revision using the QOLT rubric. QOLT provides a website with resources for training and support (see http://qolt.sfsu.edu/content/resources). Faculty can apply to have their course reviewed by anonymous peers, potentially earning certification and recognition (San Francisco State University 2016). Campus coordinators from each CSU campus may select one course per year to be recognized as a QOLT exemplary course (California State University 2015a). QOLT is part of an initiative to support faculty in online and blended course design and teaching strategies (San Francisco State University 2016). In addition, CSU has a system-wide agreement with QM (California State University 2015d).

Common Standards to Guide Design of Online Courses

We also examined the physical characteristics of the six instruments and reviewed the breakdown of each instrument, including the number of sections, the section names, and sub-sections (Table 2).

Table 2 A List of Components of Evaluation Instruments
Full size table

The evaluation instruments contained, on average, over six sections, and fifty-nine criteria. All of the instruments evaluate the following 12 criteria (Table 3); examples from the instruments are provided:

Table 3 Criteria Found in All Evaluation Instruments
Full size table

The following nine criteria are included in five (out of six) of the instruments (Table 4); examples from the instruments are provided:

Table 4 Criteria Found in Five Out of Six Evaluation Instruments
Full size table

The following criterion is included in four (out of six) of the instruments and examples from the instruments are provided (Table 5):

Table 5 Criterion Found in Four Out of Six Evaluation Instruments
Full size table

Overall, the instruments display similarity. This is not surprising since the evaluation instruments frequently report being shaped by other online course evaluation instruments. For instance, the OEI Rubric was developed by professionals based on a review of other online course rubrics (California Community College Online Education Initiative 2016). The QOLT Rubric was created after a survey of related research and literature, and careful consideration of existing models for assessing effective online teaching and learning (California State University 2015c).

The evaluation instruments have been created to address a particular need or context, but the general focus is on “best practices” and promoting quality in online courses. While there is not a universally agreed upon metric, these instruments demonstrate that effort is being made to help facilitate quality course design online.

Unique Features of the Instruments

Despite the similarity of the evaluation instruments, each instrument also displayed unique features.

Blackboard

The Blackboard Exemplary Course Rubric pays particular attention to interaction and communication. The instrument checks if guidance is provided for learners to work with content in meaningful ways and that instructors have a plan to use communication tools in the course. The Blackboard Exemplary Course Rubric checks that “asynchronous communication strategies promote critical reflection or other higher order thinking aligned with learning objectives.” It also checks that “synchronous communication activities benefit from real-time interactions and facilitate ‘rapid response’ communication (e.g., students gain practice discussing course content extemporaneously without looking up basic, declarative information).” The Blackboard Exemplary Course Rubric and the OEI Rubric confirm that individualized learning opportunities (remedial and advanced activities) are provided. In addition, the Blackboard Exemplary Course Rubric checks to see if a tutorial is provided to help students navigate through the learning management system, as do the OEI Rubric and QOCI.

OEI

The OEI Rubric has extensive information to ensure accessibility for all students, as required by Section 508 of the Rehabilitation Act of 1973 (California Community College Online Education Initiative 2016). Items include content page accessibility (e.g., making sure that heading styles are consistent and images use descriptive alternative text), file and document accessibility (e.g., “Reading order is correctly set so that content is presented in the proper sequence to screen readers and other assistive technologies”), and multimedia accessibility (e.g., “Live broadcasts include a means for displaying synchronized captions”). In addition, the OEI Rubric checks that “An Instructional Material Inventory of each inherently inaccessible learning object is provided with a plan that has been reviewed and approved by the DSPS office for accommodating students with disabilities” (California Community College Online Education Initiative 2016).

Unique to the OEI rubric is explaining the instructors’ role in supporting course technology, and checking that links to technology support are provided. Also, the OEI Rubric includes criteria to confirm that “The design and facilitation of communication activities are responsive to the variety of cultures and communication styles in the learning community.”

OSCQR

OSCQR provides users with clear information on design and layout requirements. OSCQR checks that “a logical, consistent, and uncluttered layout is established” by the course designer. OSCQR includes items to help readability. For example, the use of a sans-serif font of at least 12 points, titles and headings to improve the structure of the course, and a consistent color scheme and icon layout are advocated. OSCQR also encourages linear formats instead of tables. OSCQR promotes ease of use for learners, checking that “students have access to a well-designed and up-to-date gradebook” (State University of New York 2016) and that the syllabus is provided for learners to use in a clear and navigable way. OSCQR is the only instrument that checks to see if the course is free of grammatical and spelling errors.

OSCQR is the only instrument that does not check to see how course activity enables students to meet objectives. Also, unlike three out of the six instruments, OSCQR does not check if the instructor’s role within the course is explained. This is likely a result of OSCQR focusing on course design, not instruction.

QM

The QM Rubric is unique in ensuring that course technology and instructional materials are current, and that the distinction between required and optional materials is clearly explained. The QM Rubric endeavors to ensure that each of the components (e.g., objectives, assessment, instructional materials, course activities, and technology) coordinate to provide desired learning outcomes (Quality Matters 2016). A unique feature of the QM Rubric is that it checks to ensure prerequisite knowledge or competencies in the discipline are stated.

The QM Rubric is the only instrument that does not check if a course provides an opportunity for learners to provide anonymous feedback, if collaborative activities in the course reinforce course content and learning activities, or if the course includes guidelines for participation. Further, it does not check to make sure content is chunked into learning modules or units, a standard on four out of the six instruments reviewed.

QOCI

QOCI checks courses for specific instructional strategies, such as having learners demonstrate knowledge in a variety of ways, and ensuring that the “selected tool for each activity is appropriate for effective delivery of the content” (Illinois Online Network 2015b).

QOCI is written in a more colloquial manner than the other instruments. For instance, feedback is divided into three standards: by “when,” “what,” and “how.” While QOCI does not check that instructions are written clearly within the course, like five out of six of the instruments, it does have a section on organization and management. The organization and management section checks to make sure the amount of time allocated for each assessment is provided, availability dates and deadlines are clearly stated, and information is provided on whether retaking assessments is permitted. QOCI also makes sure students are provided with a list of supplies (e.g., textbooks), and the number of credit hours earned for successful completion of the course. QOCI also checks to make sure a grading scale and information about penalties and extra credit is provided.

In addition, QOCI has several Web design standards. The instrument checks that scrolling is minimized, there is a consistent design layout, the font is legible, the use of pop-up windows is appropriate, and additional frames are minimized. While other instruments also specify design standards, QOCI is the only instrument that specifies that images should be clear, optimized for efficient loading, and animated images are limited to those that contribute to the learning experience. In addition, QOCI checks that a tool or reporting mechanism is provided to help determine a student’s readiness for the course.

QOLT

QOLT is the only instrument that addresses course design and delivery (i.e., instruction). For instance, QOLT checks if the instructor was helpful in identifying areas of agreement and disagreement on course topics, how the instructor helps students understand importance of course topics and related outcomes, and if the instructor helps keep course participants engaged and participating in productive dialogues (California State University 2015b).

QOLT is also the only instrument that does not require an expected response time from the instructor (instead it states, “the instructor provides feedback in a timely fashion”). QOLT is the only instrument that includes mobile platform readiness. Moreover, it is the only instrument that encourages designers to reduce the number of steps a student must take in order to reach primary content and to limit the amount of course content that does not contribute directly to student learning outcomes.

Conclusion

Building a course without quality standards is like building a house without safety and building codes. Such construction would leave homes (and online courses) a jumble of ad hoc choices, lacking potentially important elements. Evaluation of online course design may provide validation and continual improvement for stakeholders. High quality courses are more likely to maximize user satisfaction and encourage better learning outcomes. Researchers indicate standards and peer review are helpful in promoting quality (Chao et al. 2006; Feldman et al. 2000; Little 2009; McGahan et al. 2015). The evaluation instruments reviewed offer standards with the intention to promote best practices in online course design. They offer the online course designer an organized way of viewing and/or reviewing standards in online courses. While these instruments differ in length and format—and even in the time of use —the use of an evaluation instrument may help direct attention to standards that can encourage quality in online course design.

This study provides an overview of six national and statewide online course design evaluation instruments and their unique features. This information can help inform instructional designers and instructors of online courses, and administrator of distance education in terms of selecting and using evaluation instruments for improving online course quality. This study also identifies a set of essential standards based on their presence in all six national and statewide evaluation instruments. Sharing these standards as guidance for best practices can be a start for establishing quality standards for all online courses. These standards are:

  • Objectives are available.

  • Navigation is intuitive.

  • Technology is used to promote learner engagement/facilitate learning.

  • Student-to-student interaction is supported.

  • Communication and activities are used to build community.

  • Instructor contact information is stated.

  • Expectations regarding quality of communication/participation are provided.

  • Assessment rubrics for graded assignments are provided.

  • Assessments align with objectives.

  • Links to institutional services are provided.

  • Course has accommodations for disabilities.

  • Course policies are stated for behavior expectations.

Future research can explore how frequently these evaluation instruments are currently used and the perceived value of these instruments. Research can also evaluate the cost/benefit of using different evaluation instruments. It would be interesting to research this topic from a variety of viewpoints to understand the perspectives of practitioners who currently use evaluation instruments to review online course quality and practitioners who do not use evaluation instruments to review online courses. It would also be interesting to research the perceived difference in course quality with courses that comply with the standards identified by all of the instruments in this study to other online courses. The perceived effectiveness could be measured in studies with instructional designers, instructors who design online courses, instructors who teach online courses, and online students. The more effort applied to understanding quality standards for online courses, the greater opportunity there is to provide students a better learning environment.