Keywords

1 Introduction

In recent years, the Chinese Ministry of Education has clearly put forward the “Internet plus(+) Teacher Education” Action Plan, pointing out the necessity of making full use of new technology such as MOOCs in teacher education. As a response, a growing number of studies focusing on MOOCs have been carried out. The main topics discussed in the literature include MOOCs’ influence in education (e.g. De Freitas et al. 2015), student behavior and learning outcomes (e.g. Park et al. 2015), and course evaluation (e.g. Margaryan et al. 2015). However, there have been few empirical studies of MOOC syllabus design, which constitutes the basis for MOOC development. In current practice, MOOCs content is generally designed by reference to the syllabuses of relevant offline courses, teachers’ previous experience or through some other unknown ways.

There are some differences in syllabus design between a traditional course and a MOOC. The traditional course is limited in the time and place of teaching, and the number of students. Its syllabus is pre-designed, systemic and more comprehensive. However, a MOOC is open to all the students and teachers as long as they can access internet. Its syllabus needs to be designed to focus on important and key issues in teaching, and to adapt students of various knowledge levels. The syllabus of MOOC is problem-driven, so the teachers’ feedbacks are necessary in the design process. From an empirical perspective the lack of syllabus design might negatively influence the MOOC content regarding its systematic, effective and unique nature, which motivated us to carry out evidence-based research into the syllabus design of our MOOC aimed at developing Chinese English teachers’ assessment literacy. The present study attempts to provide insight into MOOC construction methodology, especially how a course syllabus is designed using both quantitative and qualitative methods.

2 Literature Review

2.1 MOOCs

In 2008, the term MOOC was coined by George Siemens and David Comier to describe Connectivism and Connective Knowledge, a free online course delivered to thousands of students (Ebben and Murphy 2014), which is the mixed product of education and technology. Since then MOOCs have been widely used in higher education around the world. In 2012, some platforms were set up in the USA, including Udacity, Coursera, EdX. In China, universities such as Peking University and Tsinghua University have also begun to seek cooperation with international established MOOC platforms. The Chinese MOOCs platforms were developed jointly by local enterprises and universities, such as XuetangX, NetEase Cloud classroom and Chaoxing MOOCs. For example, “The Chinese University MOOCs”, a network of universities employing MOOCs co-founded by the Chinese Ministry of Education and NetEase in 2014, includes 289 universities and has attracted millions of learners till January 2019. This rapid expansion of MOOCs is attributed to their fundamental characteristics of being open, participatory and distributed (Baturay 2015).

To date, studies of MOOCs fall into three groups. One group examines the advantages and challenges of MOOCs. Charging no registration fees, having an open curriculum, and having flexible outcome requirements (Zhu et al. 2018), MOOCs attract students with high motivation (Hew and Cheung 2014). However, MOOCs also face some challenges, such as difficulty in evaluating students’ work, lack of immediate feedback on students’ work, being burdened by heavy demands of time and money, and lack of student participation in online forums (Hew and Cheung 2014). A second group of studies explores their participants. Researchers discussed learners’ retention, perception, motivation, performance and interaction in MOOCs. For instance, Jordan (2014) investigated 279 completed courses from Coursera, EdX, and Udacity and found high dropout rates of students with only 6.5% retention. To retain the students in MOOCs, Tang et al. (2015) developed a framework to help lecturers decide on necessary actions. A third group of studies focuses on the quality of MOOCs, assessing the content of MOOCs. Margaryan et al. (2015), for example, analysed 76 MOOCs and found that most scored highly on organization and presentation of course materials, but that the quality of their instruction design was not good. Zhang et al. (2017) also reviewed 150 Chinese articles about MOOCs and made a summary of the research foci, including the discussion of MOOC concept, MOOC platform, teaching mode and design, and MOOC assessment.

Current studies offer some references for developing the syllabi of MOOCs. However, there are few empirical studies of the construction of a content selection framework for MOOCs. The following discussion will shift the focus to the theme of one particular type of MOOC, i.e. English teachers’ assessment literacy.

2.2 Teacher Assessment Literacy

There is no universal agreement on the definition of assessment literacy. For example, Stiggins (1991) defined it generally as the ability to know and understand key principles of sound assessment. Derived from the general assessment literacy, language assessment literacy seeks to prescribe the distinctive assessment competencies for language teachers. Davies (2008) identified three components for language assessment literacy: skill, knowledge and principles. Davies highlighted the knowledge of language and language teaching methodologies. Fulcher (2012) pointed out that language teachers should be familiar with the standardized or classroom test (test processes and principles). Besides the knowledge base, teachers’ self-awareness and interpretation in assessment also shape their practical language assessment literacy (Scarino 2013). The assessment literacy in the Chinese context was also discussed by Chinese scholars. Lin and Gao (2011) defined the language assessment literacy with the framework of “What”, “Why” and “How”. Tang (2013) argued that English teachers should possess multilayer knowledge and skills about general assessment and language features. But these research claims were built on western contexts, lacking empirical evidence specifically with reference to the Chinese context.

The literature also discussed teachers’ levels of assessment literacy and their training needs. Mertler (2004) designed a Classroom Assessment Literacy Inventory and measured the literacy level of teachers in the USA, showing that both pre-service and in-service teachers felt inadequate to assess students’ performances. Zheng (2010) found that Chinese teacher assessment literacy is on a lower level by investigating 954 primary and secondary school teachers in China. Jin (2010) investigated language testing and assessment courses at tertiary level in China. She found that educational and psychological measurements and student classroom practice received less attention in training courses. To improve teachers’ assessment literacy, courses that train teachers in assessment are needed as the single “textbook mode” could not satisfy teachers’ needs in practice. Many new training modes have emerged recently, such as workshop, distance learning, blended learning and self-access approaches (Malone 2013). Under the “Internet+Teacher Education” Action Plan, the use of the MOOC platform promises to improve teacher assessment literacy.

3 Methodology

3.1 Research Questions

The main purpose of the study is to select appropriate content for the assessment MOOC. Two research questions are formulated to achieve the objective:

  1. (1)

    What key components constitute the syllabus of the assessment MOOC?

  2. (2)

    To what extent is the proposed syllabus of the assessment MOOC valid?

In the present study, the syllabus refers to the selection and grading of content for the MOOC. As the MOOC aims at improving teachers’ assessment literacy, an ALI, i.e., a list of assessment competencies, will therefore be considered and used to develop the syllabus. The first research question addresses the components of the syllabus. After identifying the key elements of the ALI, the key components of the MOOC syllabus can be decided and a specific content framework can be proposed. The second research question concerns the validity of the proposed syllabus, that is, how valid and reliable is the syllabus suggested.

3.2 Research Design and Methods

3.2.1 General Design and the Participants

English teachers’ feedbacks play an important role in designing the syllabus of MOOC. To fully understand their opinions, this three-phase study adopted a mixed methods approach (Cohen et al. 2011), integrating a quantitative method, i.e. a questionnaire survey, with a qualitative method, i.e. an interview. The first and second phases are related to the first research question and the third phase targets the second research question.

In the first phase, an open-ended questionnaire was used to brainstorm English teachers’ perceptions of ALI components and their expectations of the assessment training course. The results derived from this questionnaire contributed to the formulation of the second questionnaire. The first phase involved 468 English language teachers from Guangzhou, the capital city of Guangdong province, including 147 primary, 168 junior and 153 senior high school English teachers. All the participants were selected randomly. The responses of primary school English teachers were collected immediately after they finished the questionnaire. The other participants answered the questionnaire online, which was administered by the local education department.

In the second phase, an online questionnaire was developed to extract the ALI components through factor analysis and then to propose the syllabus. The survey was delivered via WeChat, a popular social communication platform in China. In total, 318 valid responses were collected from 31 primary English teachers, 182 junior English teachers and 105 senior high school English teachers from four cities of Guangdong.

In the third phase, five experts in English language teaching and information technology were asked to judge the appropriateness of the syllabus. Their responses provide the evidence for validating the main findings of the study.

3.2.2 Instruments

3.2.2.1 3.2.2.1 Questionnaires

A questionnaire was used as the main instrument for two reasons: one is that, it is a relatively cheap and convenient way to capture data from a wider target population, and the other is that it can be used to measure and describe generalized features. In the first phase, the questionnaire was to brainstorm participants’ perceptions of ALI and their expectations of the assessment training programs. Three open-ended questions were included in the first questionnaire: What knowledge and skills do you think English teachers are expected to have in order to assess students’ work? To what degree have you obtained such knowledge and skills? What do you expect to learn in assessment training courses?

In the second phase, a three-part questionnaire was developed in a stepwise process. The item pool was established based on the findings derived from both the first survey and the literature about assessment literacy, with effective items written out and sequenced. After being piloted among five postgraduates who had majored in language assessment and teaching, the item descriptions were refined to be clearer and more readable. Finally, the questionnaire was divided into three parts. The first part comprised five close-ended items to elicit respondents’ demographic information about gender, teaching experience, academic qualification, school level and administrative division. The second part comprised 50 items in the form of statements, investigating the teachers’ perceptions of the importance of different assessment literacy competencies. Specifically, items 1–5 concerned the competence of understanding concepts and theories in assessment; items 6–10, assessment planning; items 11–18, assessment task construction; items 19–27, formative assessment; items 28–30, scoring rubric design; items 31–34, assessment result analysis; items 35–38, teacher reflection & doing assessment research; items 39–42, application of information technology; items 43–45, student & parent caring; items 46–50, assessment administration & teaching adjustment. The options took the form of a five-point Likert scale ranging from 5 (“Strongly Important”) to 1 (“Unimportant”). The last part comprised one open-ended question to encourage respondents to add more information about ALI components.

3.2.2.2 3.2.2.2 Interview

In the third phase, an interview was used to collect qualitative data and five experts were invited to express how they regarded the proposed syllabus from their own point of view. Considering the interviewees were experts with rich teaching experience, an unstructured interview was employed with the aim of collecting richer data. Only one open-ended question was asked at the beginning of the interview: Do you think the proposed syllabus reflects the requirements of teaching practice and teachers’ needs?

4 Results

4.1 Initial Ideas About ALI Components and Training Needs

The results of the first phase indicated the relevant knowledge that teachers of English as a Foreign Language (EFL) should have. They had some initial ideas about ALI components, including assessment functions, such as washback effects, language knowledge and competence, assessment design principles, and the reliability and validity concept related to assessment. EFL teachers’ lack of knowledge is also an integral part of assessment literacy, such as the relation between assessment purpose and types, formative assessment, analysis and application of assessment results, etc. A questionnaire was designed based on the teachers’ ideas and the related literature. As for teachers’ training needs, the predominant need is for the design of various assessment tasks in line with National English Curriculum Standards.

4.2 Assessment Literacy Inventory

To extract the ALI components, all the data obtained in the second phase were analysed via Exploratory Factor Analysis (EFA), using SPSS 22.0. Prior to EFA, each item was tested to have significant discriminability power in the process of item analysis. The expected value of KMO (0.961) and Bartlett’s test of sphericity (χ2 = 10782.228, p < 0.001) proved the suitability of EFA. Then Principal Components Analysis (PCA) and Varimax rotation were used to extract factors. The screen plot and the variance explained indicated six factors could be extracted, which could explain 64.95% of total variance. According to the common construct of the items in each factor, the factor names were as follows: (1) Factor 1 indicated basic knowledge of assessment; (2) Factor 2 recommended formative assessment; (3) Factor 3 indicated administration of assessment task and the use of assessment results; (4) Factor 4 indicated the use of information technology; (5) Factor 5 indicated assessment task construction; (6) Factor 6 indicated the expertise needed in doing assessment research. Table 1 details the EFA results. The six assessment literacy elements together provide the foundations for the construction of the assessment MOOC syllabus.

Table 1. The EFA results of assessment literacy inventory
Full size table

4.3 Assessment MOOC Syllabus

The survey findings indicated that the central concept, knowledge and skill related to assessment were the foci of the MOOC. With reference to the factor analysis results of the ALI, we designed five modules with two layers, main topics and sub-topics, for the MOOC content with the course title “How to Assess English Learning”. The first module comprises the overview of testing and assessment, including the observable variables in factors 1, 3, 4 and 6. It discusses general assessment knowledge, such as basic assessment concept, procedures, rubrics, classification, test specification, information technology and item analysis skills. The other four modules focus on assessment of vocabulary, grammar, listening, speaking, reading and writing. The main contents of the MOOC were drawn from factors 2 and 5. These five modules were in accordance with Chinese National English Curriculum Standards, taking language knowledge and language skills as carrier and simultaneously assessing students’ awareness of culture, quality of thinking and learning ability. Each module had four or five main topics, delivered in the forms of document, video clip, interview, quiz and discussion, covering the essential knowledge and ability related to assessment. As Table 2 shows, topic 1 covers general assessment knowledge, such as assessment objective and assessment methods, while the other topics introduce ways of designing objective and subjective tests and performance assessment tasks. The sub-topics cover the specific assessment tasks frequently used in primary and secondary English teaching practices.

Table 2. Syllabus of assessment MOOC
Full size table

4.4 Validity of the Proposed Syllabus

The proposed syllabus was validated by expert judgments. For example, Expert A, a professor of language testing from a local university made comments on the relationship between the syllabus and the requirements issued by the Ministry of Education:

“The syllabus sheds light on the assessment requirement of the National Curriculum Standards. It’s a good idea to introduce both formative and summative assessment in the MOOC, especially how to design a formative assessment task in classroom teaching.”

Expert D, a very experienced university teacher of Educational Technology, paid special attention to the technological element included in the ALI:

“I think this inventory is very comprehensive and it even includes the technology item. Actually automated scoring, computer-based testing are very popular now. So, it’s important for teachers to understand the situation and try the new techniques.”

Generally, the five experts believed that the selection and grading of the content was appropriate for English teachers in Chinese schools, and that the content design was also appropriate for a training course delivered on line.

5 Discussion and Conclusion

Overall, the results from the study indicate that there are six components of ALI in the Chinese EFL context, i.e. basic knowledge of assessment, formative assessment, administration of assessment task and the use of assessment results, assessment task construction, use of information technology in assessment, and expertise in doing assessment research. The first four components are not surprising and our findings confirm the claims made by some other assessment literacy researchers, e.g. Davies (2008); Fulcher (2012); Lin and Gao (2011); Mertler (2004) and Scarino (2013). As language teachers, they are expected to understand some essential concepts and methodology in assessment such as the functions and methods of assessment, procedures of designing assessment task, how to develop test specifications in order to assess their students’ progress. It is interesting to find that information technology and expertise in doing research are also included in the ALI, which is new to the field of ALI research. With the development of information technology in education, more and more techniques are introduced to the language assessment area, such as automated scoring in writing tests, and natural language processing in speaking tests. Information technology is extremely necessary and important in China with its large population of test takers. English teachers face new challenges in coping with their professional development and it is no wonder that they believe IT knowledge should be an ALI component. There is another important component that is relevant to assessment research. It is commonly believed that doing research has nothing to do with doing assessment practice in teaching; however, the present study indicates that this is not the case. Teachers’ understanding of research methods such as how to analyse test items helps with the construction of the item pool in preparing a test.

The evidence-based five-module two-layer course syllabus is designed systematically and is unique in nature. It follows a top-down approach and includes a series of topics from general to specific in the area of assessment. It also reflects current EFL teaching practice in the Chinese context, covering assessment of language knowledge (i.e. vocabulary and grammar) and language skills. In addition, the syllabus is specially designed for an assessment MOOC and it is unique in three ways, compared to traditional offline assessment training courses. First, informativeness. The syllabus contains a very comprehensive list of assessment topics at schools and the course can offer a huge amount of resources in different modes for learners. Second, flexibility. Our MOOC is offered online and regardless of their educational and teaching background, everyone can be a course participant. For example, one main topic is classified into three sub-topics to cater for differences between teachers from schools at different levels. As for the main topic “formative assessment task design” in the module of Listening and Speaking, there are three sub-topics: “chanting in primary school”, “story-telling in junior high school” and “debate in senior high school”. Last, methodology. Concerning the advantages of MOOC, topics are introduced via different modes, e.g. document, video clip and discussion. The selection of mode is decided by the importance of the topic, the teacher’s familiarity with the topic and the learning styles in English teacher education.

The assessment MOOC based on the syllabus discussed in this paper has been put into use since April 2018 on a national MOOC platform in China. So far comments from participants have generally been positive and the content of the course is regarded as scientific, systematic and operable. On the other hand, it has to be admitted that there are some limitations such as the lack of teachers’ own reflections on assessment practice in the topic list.

To sum up, some implications can be drawn from the present study for MOOC developers and teacher educators. Quantitative methods such as factor analysis can assist in constructing a syllabus. MOOC is a promising way to improve the effectiveness of teacher training. Generally, course content construction can be a dynamic process, allowing for constant modification and improvement with feedback from the learners. Therefore, empirical studies of continuous syllabus refinement with more precise feedback data analysis should be conducted in future.