Introduction

A massive open online course (MOOC) is an online learning environment that learners have open access and can register for free or with low cost (McAuley, Stewart, Siemens, & Cormier 2010). The differences between traditional online course and MOOCs are that MOOCs are open access for all potential learners and do not typically charge registration fees if learners do not have the intention to obtain a certificate. As the number of annual MOOC participants continues to rise, public interest and attention to MOOCs has coincidently increased (Gašević, Kovanović, Joksimović, & Siemens 2014; Pappano 2012). Not surprisingly, empirical research on MOOCs has increased rapidly in the past few years to explore their acceptance, benefits, designs, implementations, impacts, and outcomes (Deng & Benckendorff 2017; Veletsianos & Shepherdson 2016).

To better understand the trends, topics, and research techniques employed in MOOC research studies, a systematic analysis and comprehensive synthesis of MOOC studies to date is necessary (Reich 2015). Such an in-depth analysis of MOOC research can assist governments and policy makers to strategically plan for the educational opportunities that MOOCs and other forms of open education present. They can also help MOOC researchers grasp the pivotal gaps in existing MOOC research as well as common methods employed to understand them. Finally, a thorough systematic review of the research on MOOCs could help educators understand the terminology that underpins this new and emerging field of distance education as well as issues, challenges, and boundaries within it.

Given the proliferation of MOOC research in just a little over a decade of existence, it is not surprising that there have been assorted previous systematic reviews of MOOC research conducted by other researchers, including Liyanagunawardena, Adam, and Williams (2013), Ebben and Murphy (2014), Hew and Cheung (2014), Kennedy (2014), Raffaghelli, Cucchiara, and Persico (2015), Veletsianos and Shepherdson (2015), Veletsianos and Shepherdson (2016), Deng and Benckendorff (2017), Paton, Fluck, and Scanlan (2018), Sanchez-Gordon and Luján-Mora (2018), van de Oudeweetering and Agirdag (2018), Lee, Watson, and Watson (2019), Zhu, Sari, and Lee (2018) (see Table 1). However, each of those literature reviews provide limited and narrow perspectives by covering a limited time period or targeting one specific topic of MOOC research. For example, the study that was conducted by Lee et al. (2019) focused solely on self-regulated learning in MOOCs. Additionally, those review studies indicated that the research topics and trends are constantly changing (Veletsianos & Shepherdson 2016). For instance, most MOOC studies initially were primarily conceptual studies (Kennedy 2014); in contrast, more a plethora of empirical studies have emerged since 2014 (Veletsianos & Shepherdson 2016).

Table 1 Previous MOOC research literature reviews
Full size table

Given that all previous review studies exploring the empirical literature on MOOCs were limited in terms of the research methods adopted and topics of that research, a systematic review of the methodological approaches adopted by empirical MOOC studies as well as the topics that researchers examined to date is vitally needed. To address the gaps in the literature, this study extends previous reviews of MOOC research by including all the empirical MOOC studies from the beginning of MOOC phenomenon in 2008 to present research in 2019. Given MOOC offerings constantly increase (The Chronicle of Higher Education Almanac 2017; Shah 2016b), current data about the research topics, methods, focus, dissemination outlets, researchers’ geographic distributions, etc., on MOOCs are needed to build upon these approaches in future MOOC research. The purpose of this study, therefore, is to offer a comprehensive systematic review of the research on MOOCs to help MOOC researchers better understand the research topics, trends, and typical research methods and to provide some insights and rationale for future MOOC research.

In this paper, 541 empirical MOOC studies published between 2009 and 2019 were reviewed. The present research study substantially expands on a previous review of MOOC research from October 2014 to November 2016 (Zhu et al. 2018) and captures the methodological concerns and changes over time. The five research questions listed below guided our inquiry:

  1. 1.

    What are the dissemination outlets of empirical MOOC research published in the last ten years?

  2. 2.

    What are the research methods employed in these empirical MOOC?

  3. 3.

    What are the research topics or foci of MOOC studies published in the last ten years?

  4. 4.

    How are researchers of these empirical MOOC studies geographically distributed?

  5. 5.

    What countries attracted the most MOOC research in the last ten years?

In addition to the above, we will investigate MOOC research from the standpoint of phases of MOOC evolution; Phase 1 (2009–2016): MOOCs primarily were free and open, and Phase 2 (2017–2019): MOOCs increasingly discussed from the standpoint generating revenue and offering credentials (Shah 2018a). The rationale for dividing the data into these two phases hinges on several trends which evolved over time but became more noticeable in 2016. Such trends include: (1) the average MOOC enrollment has shrunk for around 40,000 participants in the initial years (Jordan 2014) to 8000 by 2016 (Chung & Ho 2016) indicating that MOOCs are no longer as massive as they once were, (2) MOOCs are increasingly offered for college credits, credentials, and degrees (Hollands & Kazi 2019; McKenzie 2018; Moody 2018; Pickard 2019; Shah 2018b), (3) MOOCs provide an important service to businesses in reskilling and upskilling employees (Schaffhauser 2018; Shah 2019b), (4) regional MOOC providers have emerged (Shah 2016a), (5) a decreasing number of stand-alone MOOC courses (Shah 2016a), and (6) an increase in paid only courses (Shah 2016a). As Schroeder (2019) documents in his historical look at this phenomenon, MOOCs have significantly matured and evolved during the past decade.

Given such changes in the evolution of MOOCs in just a little more than a decade, it is vital to ask how have trends in dissemination outlets, research methods, research topics, and geographic origins of researchers and location of the research shifted during the period 2017–2019 when MOOCs certificates, micro-credentials, and degrees began to accelerate? Stated another way, have there been noticeable changes in MOOC research brought about by the monetization of MOOCs.

Theoretical perspectives

Massive open online courses (MOOCs) received extensive interest among people in the higher education sector, such as the learners, educational professionals, administrators, and instructors/professors (Gupta & Sambyal 2013; Yuan & Powell 2013). Such wide attention is interesting given the roots of MOOCs and similar forms of open education are relatively recent. In fact, the MOOC phenomenon started in 2007 when David Wiley created the first MOOC or Proto MOOC (Mota & Scott 2014). However, Dave Cormier and George Siemens in 2008 (Creed-Dikeogu & Clark 2013) first used the term MOOC to describe a course by George Siemens and Stephen Downes in Canada which emphasized connectivistic instructional approaches and open learning (Zhang 2013). MOOC began to be popular in 2012 shortly after the appearance of PLENK2010 (Creed-Dikeogu & Clark 2013; Kop 2011) and quickly expanded globally. More MOOC courses around the world are now offered today, including those from well-known MOOC providers such as Udacity, Coursera, and edX (Watson, Watson, Yu, Alamri, & Mueller 2017; Zhang 2013). Despite some skepticism on the future of MOOCs, current data show that MOOC numbers are still growing. In fact, Shah (2019a) reported that more than 11,400 MOOCs were offered by 900 + different universities in 2018 with enrollments topping 100 million learners.

According to Ross, Sinclair, Knox, Bayne, and McLeod (2014), MOOCs can be grouped into xMOOC and cMOOC. Furthermore, they explained that xMOOCs are more teacher-led in terms of content, structure, and assignment, whereas cMOOC are considered more social and non-hierarchical. Regarding MOOC features and characteristics, MOOCs are offered for free or at a minimum cost (Zhang 2013), promote discussions involving a large number of students (Kellog 2013), provide learning flexibility in terms of time and place (Pérez-Sanagustín, Hernández-Correa, Gelmi, Hilliger, & Rodriguez 2016), and allow diverse tasks in one course (Soffer & Cohen 2015).

Having an Internet connection is the main requirement for anyone to enroll in a MOOC and gain access to the associated learning materials (Kop 2011; Koutropoulos et al. 2012). As a result, MOOCs are providing increased access to higher education worldwide (Bali 2014; Bulfin, Pangrazio, & Selwyn 2014; Carver & Harrison 2013; Jacobs 2013; Liyanagunawardena, Parslow, & Williams 2013; Zhang, Bonk, Reeves, & Reynolds 2020). MOOC data from year to year not only represent a huge jump from previous years (Shah 2014, 2016b), but they offer a glimpse into the rapidly expanding resources and research attention being paid to this young and evolving field.

Recently, a new trend is MOOC-based credential and degree programs. Not only the large MOOC providers, but also some of the newer MOOC providers launched their own credentials and degree programs (Shah 2016a). For instance, edX has been running paid courses, “Professional Education” since 2014. In 2016, Kadenze started “Kadenze Programs,” which is its own credential system (Shah 2016a). In June 2016, Coursera tested a pilot program where all the course materials, including the videos, need to be paid for. Coursera has added more paid only courses since then. In addition, Coursera also launched Coursera for Business, in which companies could purchase Coursera content for their employees in 2016 (Shah 2019b). At the end of 2017, more than 500 companies purchased the service (Shah 2019b). Gradually, credentials and degree courses were offered in Coursera. Consequently, it was reported that Coursera has made around $140 million dollars in 2018 (Shah 2019b).

As noted in the introduction, a number of extensive reviews of MOOC research have been conducted by other researchers. For instance, Liyanagunawardena et al. (2013) analyzed 44 papers based on topics, types of inquiry used, and future research directions. The next year, Hew and Cheung (2014) recapitulated 25 articles regarding the utilization of MOOCs by students and instructors. Raffaghelli et al.’s (2015) study focused on reviewing the methodological approaches, while Kennedy’s (2014) published review of MOOC research focused on reviewing the characteristics of MOOCs. The first of two studies by Veletsianos and Sheperdson in 2015 analyzed MOOC articles in terms of interdisciplinarity of MOOC research, while in 2016 they looked at “the geographic distribution, publication outlets, citations, data collection and analysis methods, and research strands of empirical research focusing on MOOCs” between 2013 and 2015.

More recently, in 2018, Paton, Fluck, and Scanlan revised learners’ engagement and retention in MOOCs. In the same year, Sanchez-Gordon and Luján-Mora (2018) concentrated on accessible MOOCs, while van de Oudeweetering and Agirdag (2018) reviewed MOOC studies from digital inequality perspective. The most recent MOOC research review by Lee, Watson, and Watson targeted self-regulated learning in MOOCs. Those extensive reviews revealed that MOOC research topics and trends are continuing to progress and evolve (Zhu et al. 2018). It is also important to add that even though those literature reviews above differ in terms of their focus of interest and pool of articles, their works provide a summary of prevailing MOOC research practices and an identification of experts in the field. Such research can serve as valuable information for educators and policymakers and a base to employ new research (Okoli 2015).

As a relatively new field, a comprehensive systematic review of the empirical research literature on MOOC methods, topics, and dissemination outlets literature to date is needed due to the fact that: (a) there are now thousands of MOOCs enrolling in tens of millions of potential learners each year (Shah 2019a); (b) MOOCs have inspired discussions of many vital topics in education including issues of sustainability, credentialing, completion rates, self-directed learning, learner motivation, accreditation, and quality (Jona & Naidu 2014); (c) better understanding of the research methods and data collection and analysis is needed (Raffaghelli et al. 2015); (d) research findings should inform practice (Jona & Naidu 2014); (e) MOOCs are not an homogeneous practice (Jona & Naidu 2014); and (f) MOOC research is increasingly diverse (Liyanagunawardena et al. 2013; Veletsianos & Shepherdson 2015).

Method

This literature review followed the Cooper’s (1988) procedure for systematic review. It includes: (1) form the research problem, (2) collect data, (3) evaluate the data, (4) analyze the data, and (5) present the results.

Data collection

The data in this study were collected from Scopus and peer-reviewed journals and had to meet the following criteria for the selection (see also Zhu et al. 2018). First, given that MOOCs first emerged in 2007 and 2008 (Downes 2008; Fini 2009; Mota & Scott 2014), the studies of this review were published between 2008 and 2019. Second, the studies had to be empirical studies. Third, the studies examined MOOCs from educational perspectives, and were not just about technical issues or business models. Fourth, we used key words “MOOC” and “Massive Online Open Course(s)” to screen titles, abstracts, and the literature selected. Fifth, the studies were published in academic journals rather than as book chapters, blogs, magazines, etc., and were published in English. We only included peer-reviewed journals because such papers typically represent higher standards of research rigor and credibility (Utah State University Library 2020).

To achieve efficiency and enhance the trustworthiness of this study, the first two authors did the initial search in an equal division of journal sources. One researcher searched articles from five key journals in Scopus which tended to publish articles related to MOOCs (i.e. Computers & Education, British Journal of Educational Technology, The International Review of Research in Open and Distance Learning, Distance Education, and Educational Media International). She also conducted a search in several other journals not indexed by Scopus but have been known to publish MOOC research (e.g., Online Learning, the International Journal on E-Learning, Journal of Interactive Media in Education, Journal of Online Learning Research, and the Journal of Open Flexible and Distance Learning). The second researcher searched the rest of the articles found in the Scopus search.

To increase validity, the two researchers cross-checked the data, discussed any discrepancies, and reached consensus on analyses. The final number of published research articles identified was 541 through 2019. The total number of codes with agreement divided by the total number of codes was used to calculate the inter-rater reliability. The overall inter-rater agreement across all items was 96%. The recorded data of each study included a variety of dimensions such as the year of publication, title, journal name, general research approaches (e.g. qualitative, quantitative, or mixed-methods), data collection methods, data analysis methods, general study focus, specific study focus, and article URL. Other pertinent information we collected included the names of the authors and their affiliations, the location of the authors, the country of origin for the MOOC delivery, the MOOC provider, and the duration of the MOOC for each study.

Data analysis

For Research Question (RQ) #1, the authors calculated the number of publications from each publication outlet. To answer RQ #2, the authors coded research approaches based on three general categories: quantitative research, qualitative research, and mixed-methods (Creswell & Plano-Clark 2017) (see Table 2). To analyze data collection methods, the researchers utilized categories identified by Tashakkori and Teddlie (2003) such as interviews, surveys, focus groups, tests, and observations; in addition, discussion forum, platform database, and learning analytics were added. Platform database in this study refers to data from the MOOC platform such as enrollment information, clickstream logs, percentage of finished assignments, and video/page views.

Table 2 MOOC research methods and foci coding scheme
Full size table

To increase the validity of the analyses related to RQ #3, the researchers used the five coding categories identified by Veletsianos and Shepherdson (2015); namely: (1) student-focused, (2) instructor-focused, (3) design-focused, (4) context- and impact-focused, and (5) other (see Table 2). These five categories effectively encapsulated the highly diverse MOOC research areas. To answer RQ #4, we calculated the locations of all the MOOC first authors’ affiliations in this study. For RQ #5, the researchers calculated the countries of the MOOC being studied. For the studies which did not specify the location of MOOC delivery, the authors coded them as “Global.”

Additionally, this systematic review can be divided into two phases: Phase I: 2009–2016 and Phase II: 2017–2019. Although this review covers papers published since the initial piloting of MOOCs, fewer MOOC research publications appeared between 2008 and 2013; including three years with no data which met our above-mentioned criteria; namely, 2008, 2010, and 2012. In effect, the primary data were from 2014 to 2019. Thus, this research divides a systematic review of research paradigms and focus areas of MOOCs into two phases; namely, Phase I: 2009–2016 and Phase II: 2017–2019. An additional justification for this division into two phases was the recent emphasis on MOOCs as part of a degree program, credential, or for credit that Shah and others (Hollands & Kazi 2019; McKenzie 2018; Moody 2018; Pickard 2019; Shah 2016a; 2018b) highlighted in 2016, which countered the commonly held notion of MOOCs as free and open educational resources. We were interested in the potential shift in MOOC research as a result of such changes.

Results

In this study, the authors collected 541 MOOC empirical studies (see Fig. 1). Among these articles, one article (0.2%) in this review was published in 2009, three articles (0.6%) in 2011, 10 articles (1.8%) in 2013, 52 articles (9.6%) in 2014, 66 articles (12.2%) in 2015, 57 articles (10.5%) in 2016, and 90 articles (16.6%) in 2017, 135 articles (25.0%) in 2018, 127 articles (23.5%) in 2019 (see Fig. 1).

Fig. 1
figure 1

The number of empirical MOOCs studies published each year (2009–2019) (n = 541)

Full size image

Research question #1 (RQ #1): What are the dissemination outlets of empirical MOOC research during 2009 and 2019?

In this study, 541 empirical studies of MOOCs were identified. These studies were published in 172 different journals. The articles were primarily published in the following journals: International Review of Research in Open and Distance Learning (IRRODL) (n = 75), Computers & Education (n = 45), British Journal of Educational Technology (n = 25), Online Learning (n = 20), Distance Education (n = 18), Computers in Human Behavior (n = 15), Educational Media International (n = 14), and Journal of Online Learning and Teaching (n = 12) (see Fig. 2).

Fig. 2
figure 2

The number of empirical MOOCs studies published in each journal (2009–2019) (n = 541)

Full size image

Regarding to the trends of the publication outlets during the two phases, 189 articles (35%) in this review were published in Phase I, and 352 articles (65%) in Phase II. Several journals showed an increase in terms of MOOC articles that being published, such as Computers & Education (phase I, n = 12; phase II, n = 33), Computers in Human Behavior (phase I, n = 4; phase II, n = 11), Educational Media International (Phase I, n = 5; Phase II, n = 9), and The Internet and Higher Education (Phase I, n = 4; Phase II, n = 8) (see Fig. 3). On the contrary, the papers that were published in The International Review of Research in Open and Distributed Learning (IRRODL) had minor decrease (Phase I, n = 39; Phase II, n = 37). It is worth noting that no papers were published in Journal of Online Learning and Teaching in Phase II as it no longer accepted papers after 2016.

Fig. 3
figure 3

The number of empirical MOOCs studies published in each journal in two Phases (2009–2016 and 2017–2019) (n = 541)

Full size image

RQ #2: What are the research methods that researchers employed in empirical MOOC studies during 2009 and 2019?

In terms of research approaches, 269 articles (49.7%) were quantitative, 157 studies (29.0%) were mixed methods, and the remaining 115 articles (21.2%) were qualitative (see Fig. 4).

Fig. 4
figure 4

Research methods used in empirical MOOCs studies (2009–2019) (n = 541)

Full size image

The total number of each approach or technique used in MOOC empirical studies increased from Phase I to Phase II. Figure 5 reveals that the percentage of quantitative studies increased over time (Phase I, 44.4%; Phase II, 52.6%). However, the percentage of mixed methods decreased from 33.3% in Phase I to 26.7% in Phase II. The percentage of qualitative studies remained nearly the same across the two phases.

Fig. 5
figure 5

Research methods used in empirical MOOCs studies (2009–2016 and 2017–2019) (n = 541)

Full size image

Data collection methods

The authors also analyzed the primary data collection methods. More than half of the empirical MOOCs studies (54.0%) used one method of data collection. Fewer used two data collection methods (27.2%) and even fewer relied on three (12.7%) or more than three (6.1%).

Among the diverse data collection methods, survey techniques were used most frequently (n = 302) (see Fig. 6). It is worth noting that 128 out of 541 studies (23.7%) utilized surveys as the sole data collection method, whereas 174 studies (32.2%) combined surveys with other data collection methods. Other frequently used data were platform data (n = 209), followed by interviews (n = 92), assessments (n = 64), discussion forums (n = 49), focus groups (n = 28), and observations (n = 25).

Fig. 6
figure 6

Data collection methods used in empirical MOOCs studies (2009–2019) (n = 541) (Note: Some studies have more than one data collection methods and this figure only include the main data collection methods)

Full size image

Data analysis methods

These articles adopted a variety of data analysis methods. Among the 541 studies, 347 articles used descriptive statistics (64.1%), followed by inferential statistics (43.1%) (see Fig. 7). Besides the quantitative data analysis methods, the results also revealed that content analysis (n = 180) and thematic analysis (n = 45) were used for qualitative studies. In addition, computational data analysis methods such as learning analytics or data mining (n = 25) emerged in MOOC empirical studies. Social network analysis (n = 16) and discourse analysis (n = 8) were also adopted in MOOC empirical studies.

Fig. 7
figure 7

Specific data analysis methods for MOOC research (2009–2019) (n = 541) (Note: Some studies contain more than one data analysis method)

Full size image

RQ #3: What are the research foci in MOOC studies during 2009 and 2019?

As mentioned earlier, we divided the research focus into five different categories. More than half of these articles (n = 300) were primarily related to students, followed by design-focused (n = 156), context and impact-focused (n = 51), and instructor-focused (n = 37) (see Fig. 8).

Fig. 8
figure 8

Primary/general focus of MOOC delivery (2009–2019) (n = 541) (Note: Some studies contain more than one area of focus)

Full size image

The percentage of student-focused MOOC research increased during Phase II (56.4%) compared to Phase I (52.7%) shown in Fig. 9. Similarly, the percentage of instructor-focused studies increased from 3.8% in Phase I to 8.4% in Phase II. However, published papers related to context and impact-focused decreased from 13.4% (Phase I) to 7.3% (Phase II). The percentage of the design-focused MOOC research decreased from 30.1% (Phase I) to 27.9% (Phase II).

Fig. 9
figure 9

Primary/general focus of MOOC studies (2009–2016 and 2017–2019) (n = 541) (Note: As reflected in Fig. 9, some studies contained more than one area of focus.)

Full size image

Among those student-focused studies, they tended to target instructional design, students’ retention and completion, learner experience, social learning, motivation, engagement, performance, and interaction (see Fig. 10). It is worth noting that a few new topics such as communication patterns, motivation, and self-regulated learning have recently emerged.

Fig. 10
figure 10

Specific research topics of MOOC studies (2009–2016 and 2017–2019) (n = 541) (Note: As reflected in Fig. 10, some studies contained more than one area of focus)

Full size image

Each research focus used a variety of research methods in these 541 studies (see Fig. 11). In student-focused studies, 178 out of 300 articles (59.3%) employed quantitative methods followed by mixed-methods (27.3%) and qualitative methods (13.3%). Among the 152 design-focused studies, quantitative methods (44.7%) were used most often followed by mixed-methods (29.6%) and qualitative methods (25.7%). Another research focus relates to context and impact-focused (n = 51). Here, quantitative methods were commonly used (n = 20), but were followed closely by studies employing mixed methods (n = 17) and qualitative methods (n = 14). Instructor-focused studies were the lowest in numbers. However, among the 37 instructor-focused studies, 19 (51.4%) were qualitative studies, followed by mixed-methods (37.8%). Interestingly, only four instructor-focused studies used solely quantitative methods.

Fig. 11
figure 11

Research methods used in each research topic (2009–2019) (n = 541) (Note: As reflected in this figure, some studies had more than one research focus. However, we did not include the “other” category in this table.)

Full size image

RQ #4: How are researchers of empirical MOOC studies geographically distributed?

Based on the location of the affiliations of the first authors of 541 MOOC studies, the five countries with the most empirical MOOC research were the U.S. (n = 162), China (n = 64), the UK (n = 55), Spain (n = 44), and Australia (n = 26) (see Fig. 12).

Fig. 12
figure 12

The location of the first author of MOOCs studies (2009–2019) (n = 541) (Note: This figure only includes the main countries researching MOOCs)

Full size image

The percentage of first authors from China increased the most from Phase I (8.0%) to Phased II (21.3%) as you can see in Fig. 13. In addition, the percentage of the researchers from the Spain also increased over time (Phase I, 5.5%; Phase II, 14.6%). On the contrary, the percentage of the researchers from the U.S. decreased from 48.5% in Phase I to 34.7% in Phase II. Nevertheless, the U.S. still remained the country with the largest number of researchers conducting MOOC empirical studies. The percentage of first authors from the UK also decreased from 18.4% in Phase I to 10.5% in Phase II. This could be partially due to the fact that we selected articles that were published in English.

Fig. 13
figure 13

The location of the first author of MOOCs studies (2009–2016 and 2017–2019) (n = 541) (Note: This figure only includes the main countries researching MOOCs)

Full size image

Our findings also revealed that most researchers collaborated with others (see Fig. 14). Among these 541 studies, only 12.8% (n = 67) were single authored, whereas 24.2% articles had two authors, followed by articles with three authors (21.3%). Interestingly, 42.1% of MOOC research journal articles (n = 228) in this study had more than three authors.

Fig. 14
figure 14

The number of authors in one study (2009–2019) (n = 541)

Full size image

In addition, the percentage of single authors decreased from Phase I (13.8%) to Phase II (11.6%) (see Fig. 15). One explanation for the decrease in single author scholarship is due to the growing number of researchers collaboratively applying for research funds to get access to resources (O’Brien 2012). The percentage of two authors and four authors and more increased in Phase II compared to Phase I.

Fig. 15
figure 15

The number of authors in one MOOC study (2009–2016 and 2017–2019) (n = 541) (Note: This figure only includes the main countries)

Full size image

In terms of collaboration, 40.7% of MOOC studies involved authors collaborating within their own institution and another 25.0% involved cross-institutional collaboration in the same country. In addition, 22.0% of MOOC studies (n = 119) involved international collaboration (see Fig. 16).

Fig. 16
figure 16

Collaboration among the authors of MOOCs studies (2009–2019) (n = 541)

Full size image

The percentage of international collaborations increased markedly from 14.8% in Phase I to 25.9% in Phase II (see Fig. 17). Similarly, the collaborations in one institution showed a minor increase from 39.7% in Phase I to 41.2% in Phase II. In contrast, the percentage of the collaborations in different institutions within one country decreased over time.

Fig. 17
figure 17

Collaboration type among authors of MOOC empirical studies (2009–2016 and 2017–2019) (n = 541) (Note: This figure only includes the main countries)

Full size image

In terms of research methods used by first authors of each country, a majority of studies conducted by a first author from China (71.9%) used quantitative research approaches (see Fig. 18). Researchers from Spain also primarily used quantitative methods. In contrast, researchers in the UK and Australia relied on a more balanced approach of employing quantitative, qualitative, and mixed methods.

Fig. 18
figure 18

The research approaches used by the location of the first author of MOOCs studies (2009–2019) (n = 541) (Note: This figure only includes the main countries)

Full size image

RQ #5: In terms of the delivery of the MOOC, what are the countries which MOOCs are being researched the most?

In these 541 MOOC research studies, most of them described the specific country in which MOOCs were being researched, but some studies that we analyzed did not specify the country. The research articles that fall into the latter category typically were conducted by institutions using Coursera and edX. MOOCs from the U.S. were researched the most (n = 127) followed by MOOCs from China (n = 50), the UK (n = 47), Spain (n = 41), Australia (n = 15), and the Netherlands (n = 14) (see Fig. 19). Interestingly, 108 studies were conducted on MOOCs without a clear description of the country of origin.

Fig. 19
figure 19

Countries of MOOC delivery in which the research was conducted (2009–2019) (n = 541) (Note: This figure only includes the main countries researching MOOCs)

Full size image

Similar to the first authors’ geographic distribution, the percentage of MOOCs that were researched in China increased from 6.6% in Phase I to 14.7% in Phase II, whereas the percentage of MOOCs researched in Spain increased from 6.0% in Phase I to 11.7% in Phase II (see Fig. 20). However, the percentage of MOOCs researched in the U.S. decreased (Phase I, 36.7%; Phase II, 24.8%). Similarly, MOOCs researched in the UK had similar trends with the US (Phase I, 13.9%; Phase II, 9.0%). It is also worth noting that the percentage of MOOCs researched in the Netherlands increased from 0.6% in Phase I to 4.9% in Phase II.

Fig. 20
figure 20

Countries of MOOC delivery in which the research was conducted (2009–2016 and 2017–2019) (n = 541) (Note: This figure only includes the main countries researching MOOCs.)

Full size image

Discussion

The purpose of this systematic review of the research paradigms and topics related to MOOCs as well as MOOC research publication outlets and authors’ geographical distributions was to gain a deeper understanding of the MOOC phenomenon. The 541 studies examined in this systematic review revealed several interesting trends regarding the empirical research on MOOCs published between January 2009 and 2019. Trends were explored by year first and across Phase I and Phase II. The present study analyzed the publication journals for MOOC research as well as research methods conducted, data collection methods, data analysis methods, research foci, author’s geographic information, authors’ collaboration types, geographic information regarding the delivery of MOOCs, and the distribution of the MOOC research by year of publication. The findings of this study provide extensive information that can serve to raise the awareness of researchers, funding agencies, policy makers, and other interested stakeholders of the methodological issues, research topic trends, publication outlets, authors’ geographical distributions, and collaborations in empirical MOOC studies.

The 541 empirical MOOC research was published in 172 different journals. Interestingly, approximately 14% of the studies were published in The International Review of Research in Open and Distributed Learning (IRRODL) which had most empirical MOOC studies during the time period of the present research project. This finding aligns with the result of a previous study by Zhu et al. (2018). During Phase II, the percentage of studies published in Computers & Education and Computers in Human Behavior increased compared to Phase I. The finding is in line with Guo, Zhang, and Guo’s (2016) result that Computers & Education is one of the major publication outlets in educational technology field.

The results indicated that quantitative methods were most frequently employed when it comes to the MOOCs empirical studies. In addition, the percentage of quantitative methods continues to increase in Phase II compared to Phase I. Among the various data collection methods, survey methods were used most frequently. In addition, descriptive statistics were the most used data analysis method, which concur with our previous study findings (Zhu et al. 2018). Interestingly, when examining the research methods employed in MOOC research and the country represented by the first author of the study, a majority of MOOC studies from China (71.9%) and Spain (54.5%) used quantitative research approaches. The fact that MOOC researchers predominantly used quantitative research methods seems likely to be the result of the relatively easy access to MOOC data sets that lend themselves to quantitative forms of analysis (Veletsianos, Collier, & Schneider 2015).

Mixed methods were the second most used methods, while qualitative research methods were least used. Such findings support Veletsianos and Shepherdson’s (2016) advocacy of methodological diversity in MOOCs research. Therefore, MOOC researchers might consider balance the methods used in MOOC research, which might provide comprehensive picture of MOOC phenomenon. For example, researchers might utilize in-depth interviews with learners and instructors, focus groups, and virtual observations, as well as social networking and thematic analyses. Some previous studies have used course data and discussion forums for social network analysis to better understand communication and interaction patterns of MOOC learners (Kellogg, Booth, & Oliver 2014; Park, Jung, & Reeves 2015; Skrypnyk, Joksimović, Kovanović, Gašević, & Dawson 2015).

Regarding research topics, this study found that research topics primarily focus on MOOC learners’ needs and experiences. Such findings are in line with research findings of Coffrin, Corrin, de Barba, and Kennedy (2014) and Zhu et al. (2018). The specific research foci of such MOOC empirical studies often target instructional design, students’ retention and completion, learner experience, social learning, motivation, engagement, performance, and interaction. It is worth noting that new topics related to learners’ behavior, motivation, self-regulated learning (Lee et al. 2019), integrating MOOCs to campus learning, and learner interaction increased in Phase II compared to Phase I.

Instructional design is the second most frequently researched area. In contrast, only few studies focus on MOOC instructors or the context and impact of MOOCs; particularly rare are the MOOC instructor-related studies that utilize quantitative methods. Therefore, we need more MOOC research related to MOOC instructors, such as instructor motivations for offering MOOCs, instructor design and development experiences related to MOOCs, instructor pedagogical practices, and instructors' interaction with TAs, guest experts, and instructional designers. Thus, future studies related to MOOC instructors should be conducted using quantitative or mixed methods.

As mentioned in the results section, the number of empirical MOOC studies increased significantly from 2009 to 2019. Approximately, one-third of the papers were published by the first authors whose affiliations were from the U.S., followed by China, the UK, and Spain. These results concur with a previous study by Veletsianos and Shepherdson (2016) and Zhu et al. (2018). The similar findings indicated that MOOC researchers are geographically concentrated in specific areas. Analyzing research conducted by researchers from different countries or regions of the world to see different perspectives would be valuable, given that scholars from different countries might bring in different perspectives. It is worth noting that, other factors might influence these results given that the papers included in this study were all written in English. To address this issue, future research might explore studies written in other languages and geographic regions of the world.

As mentioned earlier, a notable trend is that the percentage MOOC research studies with first authors from China increased the most from Phase I (8.0%) to Phased II (21.3%). This might be a result of the rapid growth of MOOCs in China, and also due to high concern given in China to rankings and research outputs (Zhang, Perris, Zheng, & Chen 2015). Currently, many prestigious universities in China such as Tsinghua University and Peking University offer MOOCs through collaborating with Coursera or edX. In addition, Chinese educational companies and universities also actively launched localized MOOC such as XuetangX, Coursera Zone, MOOC Academy (mooc.guokr.com), and CNMOOC (Zhang et al. 2015). It is worth noting that XuetangX has 14 million registered users and is now one of the top three MOOC providers worldwide in regard to the registered users (Shah 2019a). Consequently, it is likely that MOOC research in China will increase significantly during the coming decade.

Regarding collaboration among authors, the percentage of the collaboration among authors increased from Phase I to Phase II. In particular, the percentage of international collaborations increased tremendously from 14.8% in Phase I to 25.9% in Phase II. Other researchers also reported similar findings concerning collaboration among authors. For example, Guo et al. (2016) found that international collaboration in educational technology field increased between 2000 and 2012. Guo et al. also found that the U.S. is the most active country regarding collaboration. Given that the number of MOOCs is increasing in developing countries, research collaboration among scholars in different countries will enhance the exchange of ideas among MOOC instructors/providers and better serve learners around the world.

Study limitations

It is important to highlight the limitations of this study. First, we only reviewed peer-reviewed journal articles that were published in English. Thus, we might likely have excluded some important discussions of MOOCs in the form of book chapters, conference proceedings, institutional reports, newspapers, and theses. Consequently, we were unable to obtain a complete picture of the country of origin of MOOC research due to our restriction to publications in English journals. Second, we had to set the limit of our research to the Scopus database and several other journals that focused on online learning in order to be as comprehensive as possible but also to be feasible in our efforts. These limitations could be addressed through future research endeavors.

Conclusion and future research

The analysis results of 541 empirical MOOC studies that were published between 2009 and 2019 indicated that quantitative methods were the dominant research approach, whereas qualitative studies were the least frequently used methods. These results continue to support previous research findings from Veletsianos and Shepherdson (2016) and Zhu et al. (2018). Quantitative methods increased during Phase II and remained the dominant research approach, whereas qualitative studies remained the least employed. Surveys were specified as the most frequently used data collection method, while MOOC research studies using platform database and interviews were the other two data collection methods extensively used in MOOCs. Not surprisingly, descriptive and inferential statistics were widely used in data analysis. At the same time, learning analytics and data mining appear to be fast emerging data analysis methods in MOOC studies.

Regarding the research topics, most MOOC research to date has focused on learner issues such as the learner experience, social learning, engagement, self-regulated learning, motivation, performance, and MOOC completion, whereas research on MOOC instructors remains minimal. To address this gap, MOOC researchers in the future might target instructors or design more comprehensive studies of various MOOC stakeholders such as learners, instructors, instructional designers, or program administrators. More research on MOOC instructors’ design process and perceptions would enrich the understanding of MOOC phenomenon. Such research could promote a deeper understanding about the quality of MOOCs, cultural sensitivity in MOOCs, MOOC pedagogies including course interactivity and engagement, and assessment practices from MOOC instructors’ perspectives.

As for the location of the affiliations of the first authors of the empirical MOOC research, the U.S. was the country that most widely conducted empirical MOOC research. However, the percentage of first authors from the U.S. decreased, while the percentage of first authors from China and Spain increased. The geographic distribution of the delivery of MOOCs that were researched followed similar trends as was found with the geographic distributions of the first authors of these 541 studies. Apparently, most MOOC researchers conducted research on the MOOCs offered in their own countries. Given such findings, it is highly plausible that greater international collaboration among the MOOC researchers would result in a cross-fertilization of ideas related to MOOC research perspectives, methods, topics, and analyses. It is quite plausible that such international collaboration would springboard entirely new forms of research inquiry into MOOCs and other forms of open education. At the same time, it is clear from this comprehensive study that research on MOOCs keeps growing across disciplines and locations worldwide.

The consistent trends revealed by this study can help MOOC researchers build upon and extend the MOOC research through addressing new topic areas, leveraging various research methods, and conducting joint research. The findings of the study provide implications for MOOC design practice. Through understanding the topics that were intensively researched as well as those that were insufficiently researched, MOOC instructors and instructional designers can obtain insights from the research findings and implement them into practice. They can investigate MOOCs in regions that were researched in the reviewed studies and glean insights from the existing MOOCs to enhance their design practice.

By knowing which countries’ MOOCs were more frequently studied, we can identify the potential gaps in the research. Additionally, the findings might inform those investigating the shifting of MOOCs being studied from developed countries to those in the Global South (Zhang et al. 2020). As such, our results revealed here can inform government agencies of what MOOC topic areas they might fund as well as the type of research methods they might endorse. At the same time, countries might strategically plan for MOOCs in different ways. This systematic review of the research can also help those in countries not appearing high on the list of MOOC adoption to make a case for more significant governmental funding and to start piloting targeted educational initiatives.

Regardless of the country of origin, it is clear that extensive collaboration among authors is a predominant phenomenon in empirical MOOC studies. The resulting insights into the process and the evolution of collaborative partnerships in MOOCs research and other forms of open education with the attention to the intended goals and outcomes of such partnerships could be beneficial for the entire field of distance education. Through understanding such trends and gaps, MOOC researchers will be able to build upon and extend the MOOC studies to date by addressing new and innovative research topics, utilizing more diverse and time intensive research methods, and conducting even more globally collaborative and widely impactful research.

The world can no longer wait for the results of a solitary MOOC researcher to find its way into a journal or a book chapter several years later. The findings of this research indicate that teams of cross-institutional and cross-cultural MOOC researchers are emerging which have the potential to redirect educational initiatives of entire country or region of the world and make a marked impact on those in the Global South (Zhang et al., 2020) whose access to Internet-based learning is increasing rapidly. With over one million people each day obtaining access to the Internet for the first time (Kemp 2019), the learning potential from MOOCs and other forms of open education multiplies.

Given the growth of MOOCs in the past few years toward revenue models, it is important to extend the previous line of research that concerned the initial era of free and open MOOCs, namely the ones by Liyanagunawardena et al. (2013), Gašević et al. (2014), Veletsianos and Shepherdson (2015, 2016), and by Deng and Benckendorff (2017). This study provided a more comprehensive systematic review by including MOOC empirical research from the first arrival of MOOCs to present. We suggest future research continue to expand upon methodological approaches and topics that are perceived to be critical to MOOC sustainability, growth, and evolution in the coming decade.