Keywords

1 Introduction

Massive Open Online Courses (MOOCs) attract great numbers of learners due to the wide range of opportunities they offer for online learning. Despite their growing popularity and their large enrollment, a critical issue they face is the high learner dropout rate, which puts the efficacy of MOOCs into question [1]. In their survey, Hone and El Said [2] investigated the main factors that affect learner retention in MOOCs. It was found that effective interaction with the instructional staff may affect learner retention directly, while the quality of course content seems to affect learners through its perceived effectiveness. Several other studies also address the problem of learner retention and reveal that a key factor to this issue is the lack of adequate support and interactivity in the discussion forum [3, 4].

The discussion forum is a crucial part of a MOOC platform. Through asynchronous communication and active participation in the forum [5], learners can receive support through discussions with their peers or with the course instructional staff. It has been suggested that a well-run discussion forum provides a sense of community that promotes engagement across learners and may have a positive impact on their motivation [6]. On the other hand, the main actors that provide support to learners within the discussion forum are the instructors and the teaching assistants [7]. Teaching assistants (TAs) have a crucial role in keeping learners motivated and engaged with the course [8]. Their role is to keep track of the forum discussions and make prompt interventions to help learners with their problems related to the course.

A key requirement of the MOOC discussion forum is to promote the main principles of social constructivism [9], which posits that “each learner constructs means by which new knowledge is both created and integrated with existing knowledge” [10]. According to this theoretical framework, the TAs step aside to a new role as facilitators in the learning process by connecting learners with peers and learning processes, while the students create their own knowledge and open up new learning pathways [11]. Moreover, it is understood that the way TAs handle discussions within the forum and the pedagogical strategies they follow, can play an important role in motivating learners enhancing their learning experience [12].

The pedagogical approaches that are promoted within a MOOC, determine the course’s instructional design [13]. Several studies have been performed to assess the instructional design of MOOCs [14, 15]. In their research, Guàrdia et al. [16] revealed that a deep pedagogical approach is still missing from the instructional design of MOOCs. In another study, Margaryan et al. [17] investigated the quality of the instructional design in 76 MOOCs by using an evaluation framework that they proposed. This framework includes the First Principles of Instruction, known as Merril’s criteria [18], and has its roots on the theory of social constructivism. The results of their high-impact study revealed that the majority of the MOOCs performed poorly judged by most instructional design principles. On the other hand, in terms of quality and presentation of the course material, most MOOCs were described as ‘well-packaged’. The evaluation process focused more on the activities that were designed by the MOOC instructors but did not address the issues that are related to the discussion forum. TAs have an important role in facilitating learners and in promoting learning within the forum, but this aspect was not considered during the evaluation process.

Being motivated by the work of Margaryan and colleagues [17], in this paper we extend their analysis on the activity that takes place within the discussion forum of a MOOC. We present a mixed-methods study, which aims to investigate the main intervention strategies that TAs followed in the discussion forums of two MOOCs and assess their instructional approaches by using the framework proposed by Margaryan and colleagues [17]. These MOOCs were delivered through the OpenEdX platform, one of the major MOOC platforms [19]. This study reveals some important issues related to the TAs’ instructional approaches that may be related to the instructional design of the courses. These issues should be considered by MOOC instructors and designers in order for them to focus, not only on their courses’ material quality and activities, but also on the instructional approaches that the TAs follow within the forum. This way, learners may be motivated and effective learning promoted.

2 Literature Overview

Despite the growing interest in the assessment of MOOCs’ instructional design, little research exists that focuses specifically on the facilitation strategies and pedagogies of the MOOC instructors [20]. In their study, Watson et al. [21] applied the ‘Community of Inquiry’ framework to examine a team of MOOC instructors’ use of social presence and teaching presence by examining course announcements and the team’s participation in the discussion forums. Results of this study highlight the need for further research in the field of MOOC instruction and facilitation and their importance for an effective instructional design. Evans and Myrick [22] performed a mixed-methods survey on 162 professors with the goal to understand how MOOCs are perceived by them, in the role of instructors. It was found that most MOOC professors were experienced faculty members with relatively little prior experience in teaching online. This issue led to insufficient instructional approaches regarding the MOOCs they created. In another research, Haavid and Sistek-Chandler [8] revealed that the main issue that the instructors faced was the massive audience they had to satisfy and the fact that they had to adapt their pedagogies to them. From these studies, it is evident that, even instructors who are experienced teachers, face difficulties in following adequate instructional approaches in the MOOCs that they create.

For the instructors, one of their main challenges is the massiveness of MOOCs. Wiley and Edwards have called this challenge as the teacher ‘bandwidth problem’ [23], which is especially an issue in MOOCs if teaching is understood as more than lecturing. To overcome this problem, instructors hire relatively inexpensive teaching assistants into their courses [24]. TAs have a supportive role in MOOCs, usually with-in the discussion forum, and their goal is to reduce the workload of the instructor during the MOOC’s time schedule and facilitate learners with their problems. The number of TAs required to provide sufficient learning assistance to all students of a MOOC with thousands of registrants is prohibitively high. To resolve this issue, several studies have attempted to build forum posts classification models that will assist TAs in the discussion forum of a MOOC [24, 25]. The results of these studies suggest that post classifiers may contribute in resolving the issue of massiveness in MOOCs, as they support TAs in identifying posts that require their intervention.

Most of the studies, in the field of MOOC instructional design evaluation, focus on instructors’ pedagogical approaches, and on the quality of the course material and the activities that they provide to learners. Limited research has been performed on the pedagogies that TAs follow during their supportive role in the forum. It seems that the instructional approaches followed by the TAs are mostly considered as ‘black-box’ during the design of the courses. This is an important issue that should be considered by MOOC evaluators due to the importance of TAs’ role in promoting social construction of knowledge [9]. The evaluation framework proposed by Margaryan et al. [17] is based on social constructivism, and can be used to effectively assess the quality of support that is provided within the MOOC discussion forum. It is important to include the TA supporting activity during the MOOC evaluation process due to the fact that it reflects an important part of the course’s instructional design.

In the next section, we discuss the method we used in our study, which was in-spired by this background research and was based on this theoretical framework.

3 Methodology

3.1 Research Design

As discussed in the previous section, the main purpose of this study was to investigate the instructional approaches that TAs followed in the discussion forum of two MOOCs and assess them according to the evaluation framework proposed by Margaryan et al. [17]. To achieve this goal, we followed a mixed-methods approach, and more specifically a Convergent Parallel Mixed-Methods Design [26] (Fig. 1). According to this design, we triangulated different qualitative and quantitative data col-lection techniques in order to capture the TAs’ instructional approaches. This method allowed us to increase the quality, reliability, and rigor of our results [27]. Next we performed the evaluation of the TAs’ instructional approaches through the selected framework (Table 1).

Fig. 1.
figure 1

Convergent Parallel Mixed-Methods Design of this research

Full size image
Table 1. Evaluation framework of the TA instructional approaches.
Full size table

3.2 Context of Study

The study was performed on two MOOCs offered in the mathesis.cup.gr, a major Greek MOOC platform based on OpenEdX technology. The first course, ‘Introduction to Python’ (PY course), aimed to introduce learners to computer programming through Python. The second one, ‘Differential Equations 1’ (DE course), aimed to introduce learners to the mathematical theory of differential equations and their practical use. The duration of both courses was 6 weeks, and the enrolled learners were 5569 for PY and 2153 for DE. Within each course discussion forum support was pro-vided by TAs. The TAs were mostly learners that had attended former MOOCs of the same instructor with high engagement and performance. They were subsequently contacted by the instructors, assigned the role of TAs and were asked to contribute to subsequent editions of the courses. For the courses in our study, the active TAs were 5 for the PY course and 2 for the DE one.

Using the data derived from the two courses we focused on gaining insight on the main instructional approaches that the TAs followed during their interventions within each discussion forum. Then, during the evaluation process, we assessed which of the principles listed in Table 1, were promoted and which were violated or neglected judging from the nature of their interventions. Thus, the main issues of the instructional design, which are related to the way the support is provided within the discussion forum, will be revealed.

3.3 Data Collection Sources

To reveal and record the instructional approaches the TAs used for their interventions, we employed different data collection methods, both qualitative and quantitative (Table 2).

Table 2. Data collection methods.
Full size table

Table 3 gives the questionnaire that guided the semi-structured interviews with the TAs of the two courses. In the next section the obtained results are presented.

Table 3. Guide of the semi-structured interviews with the TAs.
Full size table

4 Results

In this section the collected results are presented. Due to the Convergent Parallel Design that was followed, we present the results from the qualitative and quantitative methods separately.

4.1 Discussion Forum Log Analysis

The log files from the two MOOCs in our study provided information on all activities that were taking place in the discussion forums.

Table 4 presents the descriptive statistics for a number of important variables, such as the ‘Total number of discussions in a forum’ and the ‘Number of discussions with or without TA participation’, as measured from the discussion forum of each course. Comparing the two courses, the discussions that took place in the PY forum were almost double the discussions in the DE course. This reflects the fact that PY had more than double the number of enrolled learners, compared to DE.

Table 4. Descriptive statistics of the discussion forum in each course.
Full size table

It is further observed that the PY TAs intervened in 40.54% (493 out of 1216) of all discussions while in the DE TAs in 52.01% (285 out of 548) of all discussions, while 21.79% and 15.87% of the courses’ discussions respectively received zero replies. The fact that the TAs in both courses did not participate in about half of the discussions of the corresponding forum in conjunction with the number of discussions that didn’t receive any replies, could be the effect of teacher’s bandwidth problem [23] discussed in Sect. 2. Another important observation for both courses, is the large percentage of discussions where a TA provided the first reply to a starting post of a discussion. For the PY course this was 73.02% (360 out of 493) and for the DE course 85.61% (244 out of 285). Lastly, the mean length of discussions with TA participation was found significantly higher than those without TA participation (p ≪ 0.01) for both courses. It seems that learners mostly chose to participate in discussions with TAs instead of their peers.

4.2 Participatory Ethnography

During this part of the study, the TA interventions were studied by the researcher who participated in the forum and recorded several observations. The observations referred to three possible characteristics of the interventions, formality, directness and promptness, while at the same time they were judged for posting any problems. These observations are briefly discussed next.

Formality of the Interventions.

All TAs in both courses were very supportive throughout the entire duration of each course. In addition, their behavior was very polite and formal towards all learners. They did not attempt to develop any personal relationship with the learners, by extending discussions onto non content-related topics or by changing their attitude towards a more informal communication. Apparently that was an indication and this may imply that they took their role very seriously (Table 5, Formality-A, B).

Table 5. Selected extracts of evidence from TA interventions within the discussion forum.
Full size table

Directness of the Interventions.

In the PY course, most learner questions were related to the code they had to write, and in many of their interventions, TAs responded by giving the correct answer directly (Table 5, Directness-A). By adopting such an approach, in a way they were putting an end to the discussion and were not promoting any initiatives from the learners’ side. Moreover, in some occasions they even provided alternative solutions and examples related to their problems (Table 5, Direct-ness-B). For the DE course, learner questions were mostly related to mathematical problems and theories. The TAs of this course were also providing very analytical replies with many theoretical explanations (Table 5, Directness-C), even though often the required content of the answers could easily be found in the video lectures of that same week.

Promptness of the Interventions.

The participator in the discussion forums observed that the way TAs were intervening in discussions was quite similar in both courses according to promptness. In many occasions the TAs were the first to reply to a post that was starting a new discussion. This is verified by the results of the discussion forum log analysis presented in Table 4. A possible reason may be that they were keeping track of the forum discussions quite frequently. This can be confirmed by the fact that many interventions were performed only a few minutes after the original learner’s post (Table 5, Promptness-A).

Registered Problems.

A problem that was observed quite often was the repetition of certain questions, posted by learners in different discussions (Table 5, Registered Problems-A, B). This was an issue for both courses and TAs expressed frustration. Another issue that was recorded, mostly at the PY course, refers to learner’s questions that were related to more advanced courses. These questions were still answered by the TAs (Table 5, Registered Problems-C). Learners seemed to take advantage of the willingness TAs exhibited in intervening in the forum and didn’t seem to comply with their prompts. The fact that TAs still provided full-fledged answers probably encouraged learners to keep acting likewise.

4.3 TA Interviews

The main findings of the interviews are provided here per question (Table 3).

Q1 (TA’s Education).

Each one of the TAs had a different educational background. In the DE course, TA1 was a military person (Table 6, [Q1]-A) that had built a mathematical background through participation in related online courses, while TA2 had pre and post graduate degree in physics. In the PY course, both TAs had a degree in computer science. It is evident that all TAs had an adequate educational back-ground in order to provide support to learners within the discussion forum.

Table 6. Selected extracts of evidence from the interviews.
Full size table

Q2 (Instructions to TAs).

All four TAs gave the same answer, that there were no specific instructions related to the way that they should provide support within the discussion forum (Table 6, [Q2]-A, B). They were also not prompted to have a strict timetable in terms of their forum participation. The only instruction they received was to chasten learners that do not behave according to the forum’s policies, thus acting more as forum moderators.

Q3 (Forum Tracking).

The TAs discussed the methods that they used to keep track of forum discussions. PY-TA1 reported that he used to enter into the discussion forum during late hours or morning hours before he went to his work. PY-TA2 was entering in the forum every two hours during the day. He followed this strict schedule so as not to leave lots of unmanaged workload for PY-TA1 (Table 6, [Q3]-A). Apparently they cooperated quite smoothly. For the DE course DE-TA1 stated that the fact that he works in an office allowed him to be in the Internet during the day and keep track of the forum discussions. Lastly, DE-TA2 was spending mostly midnight hours in the forum, and that was the reason that he rarely participated in dialogues with learners.

Q4 (Intervention Criteria).

The criteria that TAs followed in considering which discussions needed their intervention seemed to have been affected by the available time for forum participation. PY-TA1 and DE-TA2 said that they did not have enough time to assess every new discussion (Table 6, [Q4]-A). They just intervened in random unanswered questions they found. On the other hand, PY-TA2 reported that selected questions to answer, according to their nature. Some learners needed support, as they were inexperienced in programming. There were also learners who used the provided support on trivial or more advanced questions (Table 6, [Q3]-B). This led to TA’s frustration and there were times that he refused to answer. Finally, DE-TA1 had also constructed his own intervention criteria. He stated that he put a time threshold of 1 to 2 h in each discussion and if no one responded, he intervened (Table 6, [Q3]-C). This strategy tallies with the available time he had within the day, according to his replies in question Q3.

Q5 (Reply Structure).

In the PY course there was a contrast between the approaches that TAs followed in structuring their replies during their forum interventions. The main goal of PY-TA1 was to help the learners reach the solution by themselves. By providing extra questions, PY-TA1 was prompting learners to make an effort and figure out the solution (Table 6, [Q5]-A). He considered this approach as a more constructive way to learn. On the other hand, PY-TA2 considered that more comprehensive answers followed by examples are more appropriate for learners (Table 6, [Q5]-B). In the DE course, both TAs seem to have almost the same approach on the way they form their forum interventions. They considered important to provide learners with the proper theory related to the problem’s solution.

Q6 (Own Evaluation).

The last interview question was related to their satisfaction according to their effort as TAs. All TAs were pleased with their contribution (Table 6, [Q6]-A, B). This is due to the fact that they are highly motivated, they participate in a voluntary basis and yet they choose to spend a lot of time in the forum.

4.4 Evaluation of TA Instructional Approaches

During the interviews, all TAs stated that no specific instructions were given to them by the course instructor. This was one of the reasons that the TAs followed different instructional approaches. According to their educational background, they were able to provide adequate support to learners. The fact that there were signs of cooperation between the TAs of each course implies that they were well-organized and felt responsible for their role.

The study findings, revealed that the instructional approaches of the TAs were not promoting Collaboration (p7) and Collective Knowledge (p8), see Table 1 for instruction principles. The fact that TAs provided the first reply in many discussions did not promote further discussions between learners. This observation was verified during interviews where most TAs said that there were no criteria in terms of when to intervene. According to social constructivism, participating in group discussions allows learners to generalize and transfer knowledge and thus evolve in their communication skills [9]. In addition, building the sense of a community within the discussion forum is of great importance [6] and TAs should be directed to follow approaches that pro-mote interactions among learners.

A serious problem that TAs faced was the large number of duplicate and advanced questions. Specifically, PY-TA2 reported that there was a specific group of learners that were causing this issue and they were exploiting the TAs’ support. This issue may be related to the instructional approach of the TAs. The fact that TAs were so responsive in the forum may have encouraged some learners to post continually assuming that TAs will promptly reply, thus monopolizing their attention.

Despite these problems, the TAs were flexible enough and promoted differentiation (p8). In MOOCs there are learners from different educational backgrounds, prior experience and motivation, so it is very important to treat them differently according to their needs, hoping that this may prevent dropout due to disappointment [3]. The TAs were aware of this issue and they appeared to have implemented different instructional strategies for specific categories of learners. Specifically, PY-TA2 mentioned that discussions created by inexperienced learners were the first in priority that he responded to. On the other hand, feedback principle (p10) seemed to be absent from the TAs’ strategies. This is reasonable because TAs did not have the time to remember each learner’s progress so as to provide a proper feedback to each one of them. The main reasons were the limited available time of TAs and the large number of active learners in the course.

A major problem of TAs’ instructional approaches was that they were not promoting problem-centered (p1) learning. In both courses TAs were providing the correct solution to learners directly. The only exception was PY-TA1 who stated that he didn’t follow such approach. His approach was to lead learners to the correct solution through intermediate questions so as learners could divide the main problem into sub-problems. The TAs’ goal was to provide full-fledged answers to learners by adding complementary theory (DE course) or Python code (PY course), but this approach affects the activation (p2), application (p4) and integration (p5) principles in a negative way. From one perspective, learners receive high quality support but on the other they do not explore the problem and construct new knowledge. This may be another reason why learners kept exploiting the TAs’ support due to the fact that TAs encouraged them to do so with their willingness to intervene frequently and provide comprehensive replies.

Finally, as discussed, in the PY course, the instructional approaches of TAs promoted demonstration (p3) and authentic resources (p9) principles by providing alter-native solutions and examples in their replies. This way learners were provided with a variety of approaches to tackle their problems. On the other hand, TAs of the DE course did not seem to promote this kind of learning. This may be related to the subject matter of mathematics. Comparing the subject matter of the two courses, in computer programming there is a flexibility of different approaches that learners could follow to solve a problem, while in mathematics alternative solutions are limited in many cases.

5 Discussion and Conclusion

In this study, we attempted a contribution to the study of the instructional approaches of TAs in MOOC forums. By using a mixed-methods approach we investigated the instructional approaches used in the forums of two MOOCs and evaluated them using the framework proposed by Margaryan et al. [17]. The main findings are: The key observation was that TAs acted more as “omniscient interlocutors” rather than as “knowledge facilitators” according to our results from both the participatory ethnography and the TA interviews. The fact that they were so active in the forum in con-junction with the instant and comprehensive answers that they provided resulted in their exploitation by many learners. TAs’ frustration was conspicuous on this issue. The ‘direct reply’ approaches that TAs followed did not seem to promote interactions among learners and moreover this violates a key principle of social constructivism, i.e. collaboration [9]. In the discussion forum learners should be the main actors of communication so as collective knowledge is endorsed. TA should facilitate them [10] in resolving their issues and not provide them with the direct answers. Learners should make an effort to construct their knowledge, and by implementing a problem-centered approach towards learning they can also improve their critical thinking skills [30]. Thus, activation (p2) of their gained knowledge is achieved and can be applied in future problems [18]. On the other hand, TAs were promoting demonstration (p3), which is also an important principle for skill-oriented courses. It is important for learners to observe examples of the knowledge that they will acquire and this principle was the most common characteristic of the TAs’ instructional approaches. Finally, the fact that the feedback (p10) principle was absent, raises the need for the development of new run-time tools that will assist TAs not only to keep track of the forum discussions, but also to track learners’ progress. By using such tools, even if TAs spend limited time in the forum, they will have the chance to provide feedback to learners, according to their progress in their future interventions.

The factors that led to the observed instructional approaches of TAs are multiple and highly inter-related. Firstly, the fact that no instructions were given to them by the course instructor means that each TA had to follow a personal approach according to her intuition. They often have domain knowledge capacity to support learners but they do not necessarily have the instructional skills. As a result they adopted different strategies in the forum. Another factor that seems to have affected their instructional approaches is the available time that they had, as they participated in voluntary basis [24]. From the interviews, it was revealed that they spent limited time in the forum and this may have led to their ‘direct reply’ behavior. By having time restrictions caused them the need to fulfill every learner’s needs, in the fastest way.

The results of this study highlight some important issues related to the instructional approaches that TAs followed and this may be related to the lack of explicit instructional design of the course forum. Course instructors and designers should consider these issues and not limit their instructional design on the quality of the course con-tent, but also focus on the quality of the support that should be provided in the forum, in order to promote effective learning. In future research we will focus on further investigating TAs instructional approaches on courses of different subject matters in order to study the effect of different domains. Previous studies [31] has shown that intervention characteristics of the TAs may depend on the subject matter of the course. The exploration of such issues may lead to the development of guides that can assist course instructors and designers in order to better structure their future instructional design of their courses. We will also perform experimental research on the development of machine learning run-time tools that will provide automatic intelligent support to TAs and assist them to properly design and orchestrate their interventions.