Keywords

1 Introduction

1.1 Background

To solve common problems with Q&A webpages, including text heaviness and cognitive overload, massive open online course (MOOC) providers are becoming interested in using natural language processing-based chatbots to provide more prompt responses to individual queries. Chatbots are software programs that communicate with users through natural language interaction interfaces [11,12,13]. Although chatbots are limited in their ability to correctly respond to every possible question, especially in the initial adoption phase, they can provide human-like responses through dialogue-based interactions which feel more immediate and customized compared to webpages. However, considering the distinctively broad spectrum in a MOOC student population, providing an inclusive learning environment becomes extremely important. Therefore, MOOC providers should make sure any demographic factors do not create inequitable learning experiences for certain groups of students upon making any technological changes, including the utilization of chatbots. A study with a small sample size examined whether the learning experience from using an FAQ chatbot was disparate from using an FAQ webpage [6]. The results indicated a significant difference between the two interfaces in the level of perceived barriers (i.e., a higher level for the chatbot group) and the level of intention to use the assigned interface (i.e., a lower level for the chatbot group). However, their Q&A quality and enjoyment levels were equivalent. Among the demographic factors investigated, region and native language factors were significantly influential in creating disparate experiences. Most importantly, implications of the necessity of a chatbot interface and how to improve the students’ Q&A experience with an FAQ chatbot were found in the study.

This study will primarily examine students’ learning experiences with an enhanced Q&A chatbot with a larger sample size to build upon the previous study results. In detail, this study compares the levels of students’ behavioral intentions before and after using the chatbot, following the definition of ‘inclusive learning’ in MOOCs [6] from the Universal Design of Learning (UDL) approach. Next, this study investigates students’ levels of several other learning experience domains after using the chatbot—teaching presence, cognitive presence, social presence, enjoyment, perceived use of ease, and so on from the Community of Inquiry framework [4] and Technology Adoption Model [3]. After examining students’ possible disparate learning experiences in these domains, this study investigates how age, gender, region, and native language factors influence students’ learning experiences with the chatbot. Lastly, but most importantly, this study explores how demographic factors influence students’ perception of chatbot interactions. Next, if any are found, this study will focus on possible negative demographic factors that affect only certain groups of students to further examine how to improve a Q&A chatbot for inclusive learning in MOOCs.

Notably, the meaning of inclusiveness will be further investigated from the UDL approach first in this study. The result of this conceptualization will be utilized in multiple aspects: evaluating students’ experiences with the chatbot and examining better chatbot response design strategies to support students’ equitable learning in MOOCs.

1.2 Research Questions

What demographic factors are influential in creating different learning experiences with a Q&A chatbot in massive open online courses, and what measures can be taken to mitigate possible negative learning experiences by changing chatbot response designs to promote an equitable learning environment?

Detailed Research Questions.

There are three detailed research questions as follows:

  • What is the relationship between students’ demographic factors (i.e., age, gender, region, and native language) and their learning experience domains (i.e., teaching presence, cognitive presence, social presence, enjoyment, perceived use of ease, etc.) with a Q&A chatbot in massive open online courses?

  • What is the structural relationship between their learning experience domains? Are there any differences among the demographic groups investigated in the study?

  • From a phenomenological perspective, what are their lived learning experiences like with the chatbot when they possess demographic factor(s) identified as negatively influencing their interaction with the chatbot? How do they want the chatbot to respond to their questions?

2 Methods

This mixed methods explanatory sequential study has two phases: a user testing-oriented survey and interviews. For the user testing-oriented survey phase, I will enhance the FAQ chatbot developed for the previous studies [6, 7] to evolve into a learning assistant Q&A chatbot by taking the following two measures. First, a topic analysis will be conducted utilizing an unsupervised machine learning modeling technique, Latent Dirichlet Allocation, with relevant online forum content of the research site’s courses provided for the recent three years. The extracted topics will be compared with the existing training set on the Dialogflow platform, and new topics and their content will consist of a new training set. Second, training set refinement (for the old training set) and language adjustment (for both old and new training sets) will be conducted based on the findings from the previous studies. In summary, the chatbot’s knowledge base will be enhanced by a new training set from the accumulated forum posting contents of the research site in the site’s learning management system for recent years. The manner of presenting the chatbot’s response will be changed based on the findings from the two previous studies, which suggested some implications of promoting students’ equitable learning experiences.

Once the chatbot on the Dialogflow platform is sufficiently trained based on the new and refined training sets, it will be deployed on a website enabling the study participants to interact by participating in the user testing-oriented survey. The survey questions will consist of five categories in the following order: initial behavioral intentions, user-testing, post-usage learning experience domain levels, expectations/challenges, and demographics. The questions regarding demographics and some regarding post-usage learning experience domains are designed to collect quantitative data, and the others of learning experience domains and expectations/challenges will collect qualitative data. Appendix A shows the tentative students’ learning experience domains with the chatbot in the user-testing-based survey.

In the interview phase, phenomenological interviews will be conducted with a small group of students. Purposeful sampling will be conducted according to the quantitative analysis of the survey data to see the possible inequitable learning experiences influenced by students’ demographic factors while interacting with the chatbot. With interview data, the students’ self-reported elaborations about the challenges they experienced will also be utilized to see if any certain positioning influenced their perceptions in using the chatbot. Appendix B includes some examples of interview questions and prompts.

The recruited research site is located in the Southwestern U.S. and provides MOOCs for journalists’ professional development. This site launches 15 courses on average per year, attracting students from 160 unique countries, and student numbers per course have ranged from 1000 to 5000 in the recent three years. Course(s) in this study will be delivered in English and last for four weeks in 2022 and 2023. The active students who make progress in the courses during the designated time period will be asked to participate in this study after receiving approval from an institutional review board. This study will aim to recruit over 100 students to attain statistical power in the quantitative data analysis for the survey. For interviews, students (n = around 10) will be recruited. The preliminary quantitative data analyses will determine the criteria for recruiting interview participants.

3 Significance of the Study

According to a generally-accepted value proposition of MOOCs, MOOCs should support everyone willing to learn with open access by utilizing the flexible features of affordable courses [2]. Considering MOOC students often report they are not sufficiently guided by instructors, course providers, or peers in how to become successful learners with their course activities [c.f., 1, 8, 10], providing a Q&A chatbot could improve students’ learning experience by adding one more tool for student support. More importantly, investigating students’ learning experiences with new technology in MOOCs, such as chatbots, requires multiple aspects: learning theory, technology-enhanced learning environment, and technology acceptance. Considering a theory-agnostic research approach has been arguably pervasive in the MOOC study community due to its multidisciplinary nature [5, 9], this study will contribute to the field by providing what aspects to consider for inclusive learning when adopting new technologies in the MOOC space from concrete theoretical backgrounds. Moreover, the findings will suggest critical chatbot response design points that better serve diverse student needs in MOOCs.