Abstract
In this paper, we empirically examine and assess the effectiveness of a chatbot workshop as experiential teaching and learning tool to engage undergraduate students enrolled in an elective course “Doing Business with A.I.” in the Lee Kong Chian School of Business (LKCSB) at Singapore Management University. The chatbot workshop provides non-STEM students with an opportunity to acquire basic skills to build a chatbot prototype using the ‘Dialogflow’ program. The workshop and the experiential learning activity are designed to impart conversation and user-centric design know how and know why to students. A key didactical aspect which informs the design and flow of the chatbot workshop is that novice learners with no or very little knowledge about A.I. recognize and create the important linkage between knowledge inputs and outputs of conversational agents powered by natural language processing (NLP) so that user queries can be effectively addressed. According to the study results, 90.7% of all surveyed students (n = 43) were satisfied with the experiential learning chatbot workshop; 81.4% of the respondents felt engaged while 81.3% of the participants reported moderate to high levels of competencies (81.3%) as result of the hands-on workshop. Almost all students surveyed (97.7%) felt that the experiential chatbot workshop had met the expected learning outcomes. Besides presenting empirical data that underscore the pedagogical usefulness of conducting an experiential Chatbot workshop in basic A.I. classes with special reference to NLP, we are trying to corroborate a conceptual model developed from learning theories and technology mediated learning (TML) models aimed at measuring the effects of a chatbot practicum on students’ engagement and motivation as potential drivers of successfully acquiring basic NLP skills and learner satisfaction. The paper provides useful practical information for instructors interested in deploying a practical chatbot workshop as effective TML tool in a tertiary educational context aimed at making learners ‘future-ready’.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
1 Introduction
Business leaders around the world continue to digitally transform their enterprises to optimise processes to meet changing market requirements. Alongside the proliferation of A.I. technologies in the workplace, institutions of higher learning are also experiencing an unprecedented push to integrate digital technologies such as A.I. into the education ecosystem. A.I. in education (AIED) has the propensity to enrich teaching and learning in higher education by personalizing students’ learning courses, automating assessment tasks or providing 24/7 access to learning resources (Karandish, 2021). According to estimates by the A.I. Market in the US Education Report, the AIED market will grow at a CAGR of 47.77% during the period 2018–2022 (Report, 2018).
A popular application of AIED is deploying chatbots as conversational agents to engage students by supporting their learning both within and beyond the classroom. For instance, chatbot prototypes can be integrated with e-learning platforms, using NLP to interact with students by interpreting their queries, and drawing up relevant information from the knowledge base module to assist students in their revision (Clarizia et al., 2018). Especially in the digital age, as educational chatbots can be seamlessly integrated into various social platforms and easily accessed on mobile devices, educators find it to be an effective tool for engagement and sustaining students’ interest (Topal et al., 2021).
Chatbots are also a viable solution to bridge the communication gap between students and instructors for distance learning (Tamayo et al., 2020) and effective when deployed appropriately by educators in areas such as teaching and learning foreign languages (Kim et al., 2019; Nghi et al., 2019). They are also suitable pedagogical tools to equip students with the necessary knowledge and skills to better understand emerging A.I. technologies and how they are applied beyond the classroom in areas such as customer support, customer communication or sales & marketing.
As educators, we are cognisant of the importance of the effects of motivation and student engagement on learning outcomes. Navarro et al. (2020) likens motivation to a resource that can be tapped onto as a ‘natural source of learning’ and to spur the achievement of learning objectives. Similarly, Monteiro et al. (2015) consider motivation to be the central factor in terms of performance and engagement of students.
In this paper, we report experiences made during the implementation of an experiential chatbot workshop integrated into an introductory undergraduate management course on ‘MGMT240—Doing Business with A.I.’ at Singapore Management University (SMU) that provides non-IT students with an opportunity to build a chatbot prototype using the ‘Dialogflow’ program. One basic premise is the assumption that such a novel, learning outcomes-related, hands-on chatbot workshop is motivational and engages learners provided its pedagogical approach is effective.
The research questions for this study are:
-
RQ1 To what extent do students find the experiential chatbot workshop motivational and engaging?
-
RQ2 What are possible relationships between the experiential chatbot workshop, students’ intrinsic motivation and engagement as well as desired (learning) outcomes such as students’ satisfaction and acquisition of chatbot-related competencies.
2 Theoretical framework
2.1 Dimensions of AIED
Experiential learning refers to a process that is designed for learners to acquire knowledge and skills by “doing”. Having critically reviewed the teaching experience from past terms, the instructors of the course ‘Doing Business with A.I.’ concluded that incorporating hands on elements (ie. activities that enable students to apply what’s being taught) into the course design qua a workshop component creates a more valuable learning experience for students.
In view of the nascent space of AIED and technology mediated learning (TML) at course level, the design of the experiential chatbot workshop was influenced by learning theories such as Kolb’s Experiential Learning Model and Gagne’s Nine Events of Learning (Menkhoff & Teo, 2021). One important intent was to create a novel and interesting learning experience for students to enhance student engagement by deploying an attractive, activity-based instructional method.
A key learning objective of MGMT240 is to enable students to understand and explain the workings and applications of A.I. in various business functions:
“Explain how private and public sector organizations engaged in customer service management, finance, marketing, supply chain management and manufacturing use machine learning, deep learning, neural networks, image analysis etc. to potentially become more competitive and ‘effective’ in real- time”.
The intended learning outcomes of the experiential chatbot workshop can be categorized into three main pillars of competencies as illustrated in Fig. 1.
One key educational tool to support the course objectives is the deployment of a 3 h long Chatbot workshop aimed at enabling students to demonstrate a basic level of understanding of Natural Language Processing (NLP), the AI technology that facilitates interaction between natural human language and computers, as exemplified by conversational agents. Aligned with the popularization of chatbot usage beyond the classroom (Deloitte Digital, 2018; Han, 2021), skills related to creating chatbots, or even understanding how chatbots are developed, are deemed as competencies valued in the workplace (Brynjolfsson & McAfee, 2017; Davenport & Ronanki, 2018; Zawacki-Richter et al., 2019). Hence, the experiential workshop provides students with an opportunity to acquire skills to build a chatbot prototype using the ‘Dialogflow’ (a Google-owned developer of human–computer interaction technologies) program during the hands-on activity (see Fig. 2). Experience with using Dialogflow forms the basis of NLP-related knowledge and familiarity, making it easier for students to pick up or utilize other chatbot platforms or programs in the future. The workshop and the experiential learning activity were designed to impart students with other relevant skills such as conversation design and user-centric design, important competencies that are transferrable to other situational contexts beyond the course.
Part I: Introduction (Warm-Up) | Duration: 30–45 min | |
---|---|---|
• Introducing chatbots • Types of chatbots • Introduction to Dialogflow, NLP • Dialogflow demonstration • (using freshman orientation as case study) | Content covered: • Types of chatbots • How to select chatbots Technology involved (NLP) Dialogflow functions • Conversation design • User requirements, user-centric design, needs assessment | Teaching Objectives: • Overview on chatbot options and technology powering its capabilities • Demonstrate real-life application of chatbots using case studies |
Part II: Class Activities | Duration: 45 min | |
Activity 1 (25 min) • Create conversational agent on Dialogflow • Test chatbot prototype • Volunteers to present • In-class discussion Activity 2 (20 min) • Discussion: Implementing chatbot in the workplace | Content covered: • Design of conversation flow on chatbot • Step-by-step process of creating chatbot on Dialogflow (coding not required for workshop) • User-centric design | Teaching Objectives: ••• “Hands on” segment on Dialogflow to practice conversation design • Create a complete set of (‘Intents’, ‘Entities’, ‘Response, ‘Fallback’) • Students to test their chatbot agent • Critical thinking: • application of chatbots in different • contexts (beyond example given in class) |
2.2 Motivation
Motivation has been identified as a crucial factor central to student engagement and performance (Monteiro et al., 2015). Hence it is studied as a key variable in this research study. We consider motivation at the process stage of Winkler & Söllner’s input-process-output model (2018) and argue that one’s motivation influences the learning experience during a practical workshop designed to acquire a particular A.I.-related skill such as NLP.
A well-researched theoretical framework for studying motivation is the Self-Determination Theory (SDT) by Deci and Ryan (2008). Intrinsic motivation refers to doing things “for their own sake” or acting as the task is perceived to be inherently interesting or pleasant to the individual. Intrinsic motivation is associated with positive learning outcomes in formal education such as school performance and achievements (Augustyniak et al., 2016). SDT posits that humans have proactive tendencies that are manifested in “learning, mastery and connection with others” (Ryan & Deci, 2020). In an educational setting, basic psychological needs for autonomy, competence and relatedness need to be met for effective learning to occur.
In our chatbot workshop, we tried to create such as a needs-supportive setting to create and harness this motivational resource, e.g. by enabling students to feel confident that they can achieve the workshop goals and their own goals based on meaningful assignments, step by step explanations, learning experiences, attention check questions and constant (positive) feedback as exemplified in Table 1.
As part of our data collection efforts, we used The Intrinsic Motivation Inventory (IMI) as it is derived from the Self-Determination Theory. As a valid and reliable measurement instrument, IMI has been widely used to measure intrinsic motivation (Augustyniak et al., 2016; Heindl, 2020; Leng et al., 2010; Monteiro et al., 2015; Navarro et al., 2020). The IMI is a suitable scale which we incorporated into our survey instrument as its items consists of simple, short sentences which are easily comprehensible by undergraduate students. Besides a few items which were modified to do justice to the context of the experiential chatbot workshop, we used five of the most relevant sub-scales of intrinsic motivation, namely interest/enjoyment, perceived competence, effort/importance, pressure/tension and value/usefulness. The sub-scales of perceived choice and relatedness were excluded (Ryan & Deci, 2020; Navarro et al., 2020) because the experiential chatbot was a compulsory component of the course syllabus which all students had to attend (i.e. they had no choice). Similarly, relatedness was perceived to have a minimal role as the students mostly interacted with the workshop instructor over a short duration via a Zoom meeting.
2.3 Engagement
Student engagement through worthwhile tasks (and interaction with others) is closely linked to learning outcomes, ie. it is an “integral component” for learning effectiveness according to Mandernach (2015). Higher levels of engagement often translate into better learning outcomes, as engaged students who devote physical and psychological energy to their assignments are considered to be ‘good’ learners (Handelsman et al., 2005). Engagement research across different educational levels has shown that a learner’s level of engagement can vary due to intrinsic factors, or “learner variables” such as intrinsic motivation, and extrinsic factors that influence the learning process such as the involvement and quality of instructors, the use of relevant concepts, challenging and competency-enhancing assignments, support, positive feedback etc. (Handelsman et al., 2005; Jung & Lee, 2018).
During the experiential chatbot workshop, we tried to foster a safe and open learning environment for collaborative learning. Learners could ask questions, interact with the instructor or peers to respond to any topic-related questions or comments at any point during the workshop. Similarly, students were encouraged to volunteer and present their work to the class to share constructive feedback in the spirit of mutual, engaged learning. The workshop also included attention checkers for students to indicate their stage of progress during the guided demonstration, and they could make use of both the audio/video and text (chat box) function to share their input or reach out to the instructor for help.
To effectively examine and measure student engagement (a multidimensional construct), it is important to choose an instrument that is suitable for the respective “unique” (Mandernach, 2015) research context. We used the Student Course Engagement Questionnaire (SCEQ) as the scale of reference because the 23-item measure is robust and easy to administer. The items capture four important dimensions of engagement, namely (i) skills engagement, (ii) participation/interaction, (iii) emotional engagement, and (iv) performance engagement (Handelsman et al., 2005; Nasir et al., 2020). Some items from the SCEQ were modified slightly to fit the context of the Chatbot workshop. The SCEQ provides instructors with deeper insights into student engagement and how it affects learning beyond the information that can be observed from student’s in-class responses during the workshop and making hindsight assessments and inferences based on their grades (Handelsman et al., 2005).
Figure 3 illustrates our conceptual model which we intend to further corroborate during a future quantitative study with a bigger sample size. Its development is based on the learning theories and TML models introduced earlier aimed at examining the possible correlations between the various model variables depicting possible effects of the hands-on chatbot workshop on students’ engagement and motivation as drivers of acquiring A.I.-related competencies such as understanding and applying NLP.
2.4 Hypotheses
The following hypotheses were derived from the conceptual model as illustrated in Fig. 3:
-
H1: Students with a high level of intrinsic motivation will be more engaged than students with a low level of intrinsic motivation.
-
H2: Students with a low level of pre-workshop (functionality-related) proficiency will be more motivated than students with a high level of pre-workshop (functionality-related) proficiency.
-
H3: Students with a high level of engagement as result of the chatbot workshop experience will have a higher level of chatbot-related competencies than students who have a low level of engagement.
-
H4: Students with a high level of engagement will be more satisfied with the workshop than students who have a low level of engagement.
-
H5: Students with a high level of intrinsic motivation will be more satisfied with the Chatbot workshop than students who have a low level of intrinsic motivation.
-
H6: Students with a high level of intrinsic motivation will report a higher level of chatbot-related competencies than students who have a low level of intrinsic motivation.
-
H7: Students with a high level of pre-workshop (functionality-related) proficiency will have a higher level of post-workshop competencies than students with a low level of pre-workshop (functionality-related) proficiency.
-
H8: Students with a low level of pre-workshop (functionality-related) proficiency will be more engaged than students with a high level of pre-workshop (functionality-related) proficiency.
-
H9: Students with a low level of pre-workshop (functionality-related) proficiency will be more satisfied with the chatbot workshop than students with a high level of pre-workshop (functionality-related) proficiency.
-
H10: Students with a high level of alignment between their expectations and the actual workshop experience will be more satisfied with the Chatbot-related learning approach than students with a low level of alignment.
3 Method
3.1 Sample
An online survey was created and released at course level to all participants of the experiential chatbot workshop. In educational research, convenience sampling of an entire class is a common sampling type. Of 44 students enrolled in the business elective ‘MGMT240 Doing Business with A.I.’, a total of 43 students responded to the post-workshop survey with a response rate of 97.7%. The response of one student was disregarded because it was incomplete. All respondents in this study were from the Lee Kong Chian School of Business (LKCSB). 53.5% of the survey participants were female (n = 23), and 46.5% of the respondents were male (n = 20). Third and fourth year university students made up more than 50% (n = 26) of the sample vis-à-vis 17 first and second year students (Table 2).
While the small sample size limits the generalizability of the study (Tipton et al., 2017), obtaining more data from other (future) classes was not feasible due to time constraints as the course is only offered once a year and the unavailability of the chatbot trainer. Looking at the generalizability challenge from a pragmatic point of view, we would argue that our results are useful for other instructors interested in experimenting with NLP-powered bots in class and that our findings are also applicable to different non-ICT disciplines such as marketing. Our survey helps to increase the contextual understanding of engaging undergraduates with ‘smart’ conversational agents. Therefore, one could argue that our “small” sample research study has ‘good’ generalizability as suggested by Etz and Arroyo (2015).
3.2 Measures
Individual learner-related measures include proficiency and expectations.
Proficiency: The item scale was adapted from Dreyfus’ Model of Skill Acquisition (2004) to measure proficiency towards the functionality of chatbots by comparing the change in respondent’s self-assessed level of proficiency before and after attending the experiential chatbot workshop. Response options ranged from (0) “No Knowledge” to (5) “Expert”.
Alignment with learner’s expectation: To assess the alignment of learner’s expectations with the actual workshop-related learning objectives we used a 5-item measure. Sample items include ‘The workshop helped me to recognize that soft skills such as user-centric conversation flow design related to AI-powered chatbots are important competencies for the workplace of the future’, and ‘The chatbot workshop adequately fulfilled my learning expectations regarding the use and adoption of chatbots as smart solutions in business and beyond.’ Response options ranged from (1) “Strongly disagree” to (5) “Strongly Agree” for each of the items. The alpha reliability in this study is 0.844.
Process-related variables include intrinsic motivation and engagement.
Intrinsic Motivation: The 30-item measure was adapted from the Intrinsic Motivation Inventory Questionnaire (Ryan & Deci, 2020; Leng et al., 2010; Monteiro et al., 2015; Augustyniak et al., 2016; Heindl, 2020; Navarro et al., 2020). It features five sub-scales: interest/enjoyment, perceived competence, effort/importance, pressure/tension and value/usefulness. Sample items include ‘I believe I am pretty good at the hands-on exercise of building a basic chatbot on Dialogflow’ (for perceived competence), and ‘I think that participating in this workshop is useful for gaining critical competencies that will be important for future work opportunities’ (for value/usefulness). Response options ranged from (1) “Not true at all” to (5) “Very true” for each of the items. The alpha reliability in this study is 0.861.
Engagement: The 23-item measure was adapted from the Student Course Engagement Questionnaire (Handelsman et al., 2005) to assess behaviors and attitudes indicative of engagement. Sample items include ‘Asking questions when I do not understand the instructor’ and ‘Participating actively in small group setting during activities’. Response options ranged from (1) “Not at all characteristic of me” to (5) “Very characteristic of me” for each of the items. The alpha reliability in this study is 0.952.
Outcome-related measures include learner’s chatbot-related competencies (post-workshop) and workshop satisfaction.
Chatbot-related competencies: To assess to what extent the chatbot workshop helped students to acquire related key competencies (outcome level), we constructed a 20-item measure. Sample items include ‘Explain why an effective chatbot requires NLP and other Deep Learning techniques to understand the underlying intent of human language’ and ‘Draw a simple representation of a conversational chatbot workflow’. Response options ranged from (0) “No Knowledge” to (5) “Expert” for each item. The alpha reliability in this study is 0.959.
Workshop satisfaction: The item measure is included as a general indicator of respondent’s level of satisfaction toward the experiential chatbot workshop in general. Response options ranged from (1) “Strongly disagree” to (5) “Strongly Agree”.
4 Data analysis
Correlation analysis using Pearson correlation coefficient was applied to determine the degree to which the model variables are associated with each other. For each model variable, we calculated the average of all question responses on a 5-point scale. For example, there are 30 survey questions for the "Intrinsic Motivation" model variable where all questions are on a 5-point scale. For each survey participant, we calculated the average of the 30 5-point scale values, and the resulting average is the "Intrinsic Motivation" model variable value used for correlation analysis. Additionally, since the questions were categorized by the different groups of traits, the Cronbach’s alpha provides additional insights into the internal consistency of the results for these specific groups. It would help consider the scale of reliability of the results found over a set of items rather than just being unidimensional. Further explanations are provided in footnotes 1–4.
5 Results
According to the survey results (see Fig. 4), 90.7% (n = 39) of all respondents were pleased with the experiential learning chatbot workshop of which 60.5% (n = 26) were satisfied while 30.2% (n = 13) reported high satisfaction scores. None of the students felt dissatisfied with the experiential learning workshop. As shown in Table 3, the mean workshop satisfaction score is 4.2 (mode and median are both 4.0) suggesting that most of the workshop participants exhibited positive sentiments towards the workshop.
As illustrated in Fig. 4, 81.4Footnote 1% (n = 35) of the chatbot workshop participants surveyed reported to be engaged of which 18.6% (n = 8) were highly engaged. As shown in Table 4, the mean of the average engagement score is 3.7 (mode is 4 and median is 3.8), indicating that the experiential learning approach of the chatbot workshop might be an effective method for engaging students.
Figure 4 shows that altogether 83.8% of the surveyed students (n = 38) who attended the experiential learning process reported to be motivatedFootnote 2 (including 2 highly motivated students). As shown in Table 5, The mean of the average motivation score is 3.5 (mode is 3.7 and median is 3.5) suggesting that students’ average motivation is moderately high. One conclusion which can be drawn from the data is that the chosen chatbot teaching and learning approach is indeed motivational. The core aspect of the experiential learning approach is to “learn by doing”, and it is reasonable to deduce that students feel intrinsically motivated when they are empowered and enabled to put into practice the skills and techniques taught during the activity segment of the workshop.
Most students reported moderate to high levels of competenciesFootnote 3 after the participation in the experiential learning workshop. 48.8% (n = 21) of the participants categorized their newly acquired chatbot-related competencies as ‘professional working proficiency’ while 32.5% (n = 14) rated it as ‘full professional proficiency’. Two (4.7%) students assessed themselves and their chatbot-related competencies (post-workshop) as / at ‘expert’ level (Table 6).
The mean chatbot-related competency score is 2.9 due to the spread in the students’ reported proficiency levels. Due to the bi-modal distribution in our sample (Fig. 5), the mode scores are 2.7 and 3.0 respectively. The median proficiency score is 3.0 (moderate). The spread with regards to the proficiency levels suggests that students attending the same workshop might have different learning needs and preferences. As we observed during the workshop, some students require more time to replicate the steps demonstrated by instructors, while other students are following at the same pace or are ahead of their peers. Another factor that might affect the learning process and outcome (proficiency level) include the student’s individual cognitive load capacity. Novices unfamiliar with the bot subject matter might require more time to understand and keep pace with the workshop flow.
As shown in Fig. 4, almost all students surveyed (97.7%, n = 42) felt that the experiential chatbot workshop had met their learning outcomes.Footnote 4 As shown in Table 7, the mean of the average scores for learning outcome items is 4.2 (both mode and median scores are 4.0). This positive result suggests that most students who participated in the experiential chatbot workshop found the hands-on session useful and effective, ie. it helped them to achieve the intended learning outcomes (Table 8).
6 Hypotheses and interpretations
6.1 Intrinsic motivation and engagement
-
H1: Students with a high level of intrinsic motivation will be more engaged than students who have a low level of intrinsic motivation
The r-score for intrinsic motivation and engagement is 0.471. The findings suggest that at chatbot workshop level, there is moderate support for the hypothesis that students with a higher score for motivation tend to be more engaged during the workshop compared to students with low motivation. In general, H1 is supported by the correlational analysis, ie. students who score lower in motivation are likely to experience less engagement during the experiential learning workshop. This result is aligned with the literature on student motivation and engagement. From an educational perspective, the challenge remains to keep students motivated and engaged throughout the workshop (and the course of learning). This calls for a continuous review of the course curriculum, a flexible workshop design and student-centric teaching method to optimise students’ motivational levels and their actual engagement.
6.2 Pre-Chatbot workshop proficiency and motivation
-
H2: Students with a low level of pre-workshop (functionality-related) proficiency will be more motivated than students with a high level of pre-workshop (functionality-related) proficiency
As the r-score for pre-workshop (functionality-related) proficiency and high motivation is 0.005, H2 is not supported by the findings. In other words, there is seemingly no correlation between the student’s pre-workshop (functionality-related) proficiency and his/her intrinsic motivation related to the workshop. While this finding is puzzling given the hype about the need for ICT-related upskilling, the data suggest that it is important for educators to know more specifics about students’ prior knowledge about the subject matter in order to adjust the teaching and learning approach.
6.3 Student engagement vis-à-vis chatbot-related competencies and satisfaction (learning outcome level)
-
H3: Students with a high level of engagement will have a higher level of chatbot-related competencies than students who have a low level of engagement
The r-score for student engagement and acquisition of chatbot-related competencies is 0.427 which provides moderate support for H3. The strength of the correlation between engagement and chat-related competencies is moderate. This suggests that there is a relationship between engaged students and the level of skills which they had acquired from the workshop. To further interpret this finding, it is important to note that the workshop design exposes participants to the topic of chatbots at an introductory level only. Most participating students are from a non-computing background. Thus, given the scope and limited time frame of the chatbot workshop, it is not possible that students become fully competent on the topic at hand. As such, the above value suggests that students did in fact acquire a certain level of skills but certainly not mastery skills in line with the pedagogical objectives. Whether the workshop ought to be increased in length and depth needs to be discussed.
6.4 Student engagement and workshop satisfaction
-
H4: Students with a high level of engagement will be more satisfied with the workshop than students who have a low level of engagement
The r-score for engagement and workshop satisfaction at the learning outcome level is 0.298, i.e. there is no strong support for H4. One reason could be the fact that the chatbot workshop was conducted fully online due to the Covid 19 situation. Post-workshop student interviews revealed that those students who reported low satisfaction desired a face-to-face or physical mode of learning. Students may have experienced Zoom fatigue due to hours of exposure to visual cues as nearly all lessons took place online due to the pandemic. Furthermore, the “hands-on” nature of the workshop usually requires the teaching staff to offer help with troubleshooting, which could have been done more efficiently and effectively within a face-to-face learning context. Lastly, some students reported that they wished to have a fully working prototype at the end of the session. However, this was not part of the chatbot workshop design. Thus, it is important that instructors and students’ clearly spell out their mutual expectations (eg. in terms of desired learning outcomes) prior to the workshop.
6.5 Intrinsic motivation, Chatbot workshop-related satisfaction and competencies
-
H5: Students with a high level of intrinsic motivation will be more satisfied with the Chatbot workshop than students who have a low level of intrinsic motivation
-
H6: Students with a high level of intrinsic motivation will report a higher level of chatbot-related competencies than students who have a low level of intrinsic motivation
The r-score for high intrinsic motivation and high level of workshop satisfaction is 0.590 which provides moderate support for H5. The r-score for motivation and post-workshop competencies is 0.196. At the current stage of this study, this finding is not conclusive enough to support H6. While the results highlight the importance of intrinsic motivation with regard to being satisfied with the bot-related teaching and learning approach, they also underscore the educational importance of (i) priming learners at the beginning of the course so that they appreciate the power and use cases of smart chatbots not only in education but also in business and society, and (ii) experimenting with different ways to keep students motivated and making them competent throughout the course in order to optimize their capacity to attain the desired learning outcomes.
6.6 Pre-workshop proficiency, chatbot workshop-induced competencies, engagement and satisfaction
-
H7: Students with a high level of pre-workshop (functionality-related) proficiency will have a higher level of post-workshop competencies than students with a low level of pre-workshop (functionality-related) proficiency
The r-score measuring pre-workshop (functionality-related) proficiency and post-workshop competencies is 0.363, which shows weak support for H7. The results suggests that students with prior knowledge and experience in chatbots might not necessarily have a substantially higher advantage compared to their peers who lack the knowledge and experience.
-
H8: Students with a low level of pre-workshop (functionality-related) proficiency will be more engaged as a result of the chatbot workshop than students with a high level of pre-workshop (functionality-related) proficiency.
The r-score for pre-workshop (functionality-related) proficiency and student engagement during the workshop is 0.050. There is no support for H8.
-
H9: Students with a low level of pre-workshop (functionality-related) proficiency will be more satisfied with the chatbot workshop than students with a high level of pre-workshop (functionality-related) proficiency
The r-score for pre-workshop (functionality-related) proficiency and workshop satisfaction is -0.107. This points to a negative correlation between student’s prior knowledge and experience on the one hand and workshop-related satisfaction on the other hand. Within the current data sample, the results do not support H9.
With regard to H7, H8 and H9, it is worthwhile to ask deeper questions about students’ prior knowledge in chatbots. While accurate and appropriate prior knowledge can aid their learning, inaccurate and inappropriate prior knowledge can hinder their learning during the workshop. For those students that indicated “Yes, I have prior knowledge and experience in chatbots”, a pre-workshop assessment (e.g. quiz) can help to evaluate whether their knowledge is accurate and appropriate.
6.7 Alignment of Chatbot-related learning expectations and students’ workshop satisfaction
-
H10: Students with a high level of alignment will be more satisfied with the Chatbot workshop than students who have a low level of alignment.
The r-score for alignment of learning expectation and workshop satisfaction is 0.572. The results show moderately high support for H10. This suggests that managing the student’s learning expectation could be an important aspect for instructors to consider in designing and facilitating the experiential learning workshop (Table 9).
7 Discussion
According to the survey results, 90.7% of all respondents were satisfied with the experiential learning chatbot workshop. Altogether 83.8% of the surveyed students who attended the experiential learning process reported to be motivated while 81.4% of the chatbot workshop participants felt engaged. Most students reported moderate to high levels of competencies (81.3%) after the participation in the experiential learning workshop. Figure 6 shows the distribution of students’ responses to the survey question where the students were asked to rate their chatbot-related proficiency before and after the workshop. A t-statistic was calculated to be -13.3 with 41 degrees of freedom and a significance level of 0.05. The p-value was 8.99E-17, which is less than the significance level, indicating that the difference between the groups is statistically significant at the 0.05 level. This indicates that according to the students’ assessment of their own proficiency, there is an improvement in their chatbot-related proficiency.
Almost all students surveyed (97.7%, n = 42) felt that the experiential chatbot workshop had met the expected learning outcomes. Overall, the empirical study results underscore the pedagogical usefulness of conducting an experiential Chatbot workshop and that it positively influences students’ intrinsic motivation and engagement levels so that they can acquire basic AI competencies with special reference to NLP (Table 10).
The correlation analysis suggests that students with a high level of intrinsic motivation are gaining more from a hands-on Chatbot workshop than those learners who have a low level of motivation (H1). Students with a high level of engagement in relation to the chatbot workshop experience reported a higher level of chatbot-related competencies than learners with a low level of engagement (H3). Students with a high level of intrinsic motivation turned out to be more satisfied with the Chatbot workshop than those with a low level of intrinsic motivation (H5). Students whose expectations match the way the Chatbot workshop is conducted seem to be more satisfied with the Chatbot workshop than students with a low level of alignment (H10).
One caveat for interpreting our study is the small sample size and the fact that 100% of the respondents were enrolled in the same school (LKCSB) which may limit the model generalization. Despite belonging to different course levels, year of study and subject specialization (course major), the sample variation is limited which in turn may negatively affect the transferability of the results. However, as explained earlier, we believe that our results are useful for other instructors interested in experimenting with NLP-powered bots in class and that our findings are also applicable to non-ICT disciplines such as marketing (Etz & Arroyo, 2015).
Key strengths of our study include: (i) the innovative application of a hands-on chatbot workshop in an introductory AI for business course and initial evidence that such a practicum helps learners to acquire basic A.I./NLP skills; (ii) the evaluative survey design of the study and sharing of lessons learned; (iii) the multi-disciplinarity of the research team, and (iv) the data-driven recommendations for other instructors who wish to experiment with bots in teaching and learning.
Our study adds to the discourse about competency development related to A.I. (Benner, 2004; Tamayo et al., 2020) by sharing evaluative lessons learnt during the roll-out of an experiential chatbot workshop as an engagement component in a basic (elective) A.I. course.
While our empirical study results support the findings of previous research studies on student engagement and proficiency development (Augustyniak et al., 2016; Heindl, 2020; Mandernach, 2015; Monteiro et al., 2015; Navarro et al., 2020); the challenge remains to fully understand the drivers of low motivational and engagement levels amongst some of the students surveyed and what can be done to change that.
One possible reason could be the novelty of the experiential learning approach and the fact that many if not most of the students had no prior experience with Google’s Dialogflow platform, incl. the type of hands-on learning activity practiced during the virtual MGMT240 workshop segment which was delivered via Zoom. To what extent the perceived lack of learner autonomy hampers the autonomous motivation (Gagne, 2003) of our students has yet to be examined. Other possible causes are best explored in a qualitative context via a future focus group discussion. Acquiring more knowledge about student’s prior motivation and bot-related knowledge depth is arguably very important to further increase the effectiveness of our chatbot workshop approach.
8 Conclusion, limitations, and future research direction
In this paper we presented results of an ongoing exploratory, empirical study of students enrolled in an elective course “Doing Business with A.I.” at the Lee Kong Chian School of Business (LKCSB), Singapore Management University, to examine the pedagogical usefulness of conducting an experiential Chatbot workshop that intends to get students motivated and engaged with regards to the acquisition of basic AI competencies, namely NLP.
The empirical study results suggest that a hands-on teaching and learning approach in form of an experiential Chatbot workshop is indeed instrumental in motivating and engaging students to acquire basic bot competencies. To summarise some of the key survey results: 90.7% of all respondents were satisfied with the experiential learning chatbot workshop; 83.8% of the surveyed workshop participants reported to be motivated while 81.4% of the students turned out to be engaged. Most students reported moderate to high levels of competencies (81.3%) after the participation in the experiential learning workshop. Almost all learners who took part in the survey (97.7%) felt that the experiential chatbot workshop had met the expected learning outcomes.
Overall, our empirical study results underscore the pedagogical usefulness of conducting an experiential Chatbot workshop and ascertain the argument that it positively influences students’ motivation and engagement levels so that they acquire basic AI competencies such as NLP. Educators who wish to replicate our hands-on chatbot approach are advised to consider the following recommendations:
-
It is important to know whether and how much the participants do know about the subject matter / workshop topic (bots), including prior experiences and actual competencies (NLP) before the start of the course unit (a pre-workshop quiz can help to ascertain that).
-
Students must be made fully aware before or at the beginning of the workshop why chatbots are increasingly used in ‘real’ business contexts and beyond and how they are powered up by AI techniques such as NLP.
-
Case analyses of high-performance organizations who managed to create novel value by deploying chatbots are useful in supporting students’ understanding of the bot subject matter.
-
Ensure high motivation and engagement levels by allowing for mistakes and providing specific, positive feedback on completed assignments.
-
Create challenging and meaningful learning experiences that are both enriching and competency-enhancing.
Limitations include constraints caused by the Covid-19 situation which led to several changes in the way the experiential chatbot workshop was conducted. Instead of physically attending the session in a classroom with instructors, all students had to participate online via home-based learning / Zoom. This implies that some of the proven instructional strategies to impart chatbot-related know how and know why to students such as small group-based collaborative learning projects and interaction over assigned tasks had to be omitted due to the distance learning arrangement. We were unable to identify to what extent these factors may have affected the overall learning effectiveness.
The small sample size and the fact that 100% of the respondents were enrolled in the same school (LKCSB) do limit the model generalization. Despite belonging to different course levels, year of study and subject specialization (course major), the sample variation may be limited and insufficient to fully understand the implications of the results.
In terms of future research, it would be meaningful to expand the scope of this study to compare responses from a larger sample size of students to conduct model-specific regression analyses and solid hypotheses testing.
Data availability
The datasets generated and/or analysed during the current study are not publicly available due to limitations set by the Institutional Review Board (IRB) at Singapore Management University but are available from the corresponding author on reasonable request.
Notes
Taking the average score for items on the engagement scale, students are grouped into categories based on their level of engagement as reported in the questionnaire survey. The engagement scale consists of items rated on a 5-point Likert scale (excluding the 0 option). The average engagement score is calculated by taking the average of each respondent’s score for each item across the 23-item segment. In this study, engagement scores were grouped into 4 large categories, where students who reported an average engagement score which is larger than 1.0 but lesser than 2.0 are considered as “disengaged”. Scores higher than 2.0 but lesser than 3.0 refer to students who are considered “somewhat disengaged”. Scores more than 3.0 but less than 4.0 concern students who are “engaged”. Students who score higher than 4.0 to 5.0 are considered as “highly engaged”.
Taking the average score for items on the motivation scale, students are grouped into categories based on their level of engagement reported in the questionnaire. The engagement scale consists of items rated on a 5-point Likert scale (excluding the 0 option). The average motivation score is calculated by taking the average of each respondent’s score for each item across the 30-item segment. In this study, motivations scores were similarly grouped into 4 categories. Students who reported an average motivation score more than 1.0 but lesser than 2.0 were considered as “not motivated”. Scores more than 2.0 but lesser than 3.0 pertain to students who are “somewhat motivated”. Scores more than 3.0 but less than 4.0 refer to students who are “motivated”, and students who report score of more than 4.0 to 5.0 are considered to be “highly motivated”.
A student’s average score for chatbot-related competencies is calculated by taking the average of the individual’s scores from the 20-item scale. This scale was constructed by the instructors based on a list of learning objectives to measure specific chatbot-related competencies. In this study, students report their level of chatbot-related competencies using a 5-point scale. Competency scores for lesser or equal to 1.0 indicates “elementary proficiency”, scores greater than 1.0 but less than 2.0 refers to “limited proficiency”. Scores higher than 2.0 but lesser than 3.0 stand for “professional working proficiency”. Scores higher than 3.0 but less than 4.0 reflect “full professional proficiency”. Students who scored more than 4.0 to 5.0 are considered as “experts”.
The average score for student’s meeting of learning outcome is calculated by taking the average of 5 items on the questionnaire which reflects the intended learning outcome of the experiential chatbot workshop. Students report the achievement of the intended learning outcomes by scoring on a 5-point scale (excluding the 0 option). Learning outcomes achievement scores of 1.0 but less than 2.0 refer to learning outcomes “not met”. Scores higher than 2.0 but lesser than 3.0 refer to learning outcomes being “somewhat met”. Scores higher than 3.0 but less than 4.0 indicates that learning outcomes are “met”, and scores higher than 4.0 to 5.0 mean that the workshop has “exceeded” in helping to achieve the student’s learning outcomes.
References
Augustyniak, R. A., Ables, A. Z., Guilford, P., Lujan, H. L., Cortright, R. N., & DiCarlo, S. E. (2016). Intrinsic motivation: An overlooked component for student success. Advances in Physiology Education, 40(4), 465–466. https://doi.org/10.1152/advan.00072.2016
Benner, P. (2004). Using the Dreyfus model of skill acquisition to describe and interpret skill acquisition and clinical judgment in nursing practice and education. Bulletin of Science, Technology & Society, 24(3), 188–219.
Brynjolfsson, E. & McAfee, A. (2017). What’s driving the machine learning explosion? Three factors make this AI’s moment. Harvard Business Review (https://hbr.org/2017/07/whats-driving-the-machine-learning-explosion).
Clarizia, F., Colace, F., Lombardi, M., Pascale, F., & Santaniello, D. (2018). Chatbot: An Education Support System for Student. Cyberspace Safety and Security, 11161, 291–302. https://doi.org/10.1007/978-3-030-01689-0_23
Davenport, T.H. & R. Ronanki (2018). Technology - Artificial Intelligence for the Real World. Harvard Business Review, January–February Issue. https://hbr.org/2018/01/artificial-intelligence-for-the-real-world (accessed on 07 November 2021).
Deci, E. L., & Ryan, R. M. (2008). Self-Determination Theory: A Macrotheory of Human Motivation, Development, and Health. Canadian Psychology, 49(3), 182–185.
Deloitte Digital. (2018). Chatbots Point of View. Deloitte Artificial Intelligence. https://www2.deloitte.com/content/dam/Deloitte/nl/Documents/deloitte-analytics/deloitte-nl-chatbots-moving-beyond-the-hype.pdf (accessed on 07 November 2022).
Etz, K. E., & Arroyo, J. A. (2015). Small sample research: Considerations beyond statistical power. Prevention Science, 16, 1033–1036.
Gagne, M. (2003). The role of autonomy support and autonomy orientation in pro-social behaviour engagement. Motivation and Emotion, 27(3), 199–223.
Han, M. C. (2021). The Impact of Anthropomorphism on Consumers’ Purchase Decision in Chatbot Commerce. Journal of Internet Commerce, 20(1), 46–65. https://doi.org/10.1080/15332861.2020.1863022
Handelsman, M. M., Briggs, W. L., Sullivan, N., & Towler, A. (2005). A Measure of College Student Course Engagement. The Journal of Educational Research, 98(3), 184–192. https://doi.org/10.3200/JOER.98.3.184-192
Heindl, M. (2020). An extended short scale for measuring intrinsic motivation when engaged in inquiry-based learning. Journal of Pedagogical Research, 4(1), 22–30. https://doi.org/10.33902/JPR.2020057989
Jung, Y., & Lee, J. (2018). Learning Engagement and Persistence in Massive Open Online Courses (MOOCS). Computers and Education, 122, 9–22. https://doi.org/10.1016/j.compedu.2018.02.013
Karandish, D. (2021): 7 Benefits of AI in Education The Journal, Transforming Education through Technolgy, https://thejournal.com/articles/2021/06/23/7-benefits-of-ai-in-education.aspx (accessed on 7 November 2021).
Kim, N., Cha, Y., & Kim, H. (2019). Future English Learning: Chatbots and Artificial Intelligence. Multimedia-Assisted Language Learning, 22(3), 32–. https://search.library.smu.edu.sg/permalink/65SMU_INST/1ba19kd/cdi_kiss_primary_3703643
Leng, E. Y., Wan Ali, W. Z. B., Baki, R., & Mahmud, R. (2010). Stability of the Intrinsic Motivation Inventory (IMI) For the Use of Malaysian Form One Students in ICT Literacy Class. Eurasia Journal of Mathematics, Science and Technology Education, 6(3), 215–226. https://doi.org/10.12973/ejmste/75241
Mandernach, B.J. (2015). Assessment of Student Engagement in Higher Education: A Synthesis of Literature and Assessment Tools. International Journal of Learning, Teaching and Educational Research, 12.ß
Menkhoff, T., & Teo, L. (2021). Evaluating the Effectiveness of Chatbots as Experiential Learning Tools in an Introductory "Doing Business With A.I." Course. Springer Nature. Research Book Series. Transactions on Computational Science & Computational Intelligence.
Monteiro, & Peixoto, F., M. L. (2015). Intrinsic motivation inventory: Psychometric properties in the context of first language and mathematics learning. Psicologia, Reflexão e Crítica, 28(3), 434–443. https://doi.org/10.1590/1678-7153.201528302
Nasir, M. A. M., Janikowski, T., Guyker, W., & Wang, C. C. (2020). Modifying the student course engagement questionnaire for use with online courses. The Journal of Educators Online, 17(1).
Navarro, Sanchez-Verdejo, F. J., Anguita, J. M., & Gonzalez, A. L. (2020). Motivation of university students towards the use of information and communication technologies and their relation to learning styles. International Journal of Emerging Technologies in Learning, 15(15), 202–218https://doi.org/10.3991/ijet.v15i15.14347.
Nghi, T., Phuc, H., & Tran & Nguyen Tat, Thang. (2019). Applying Ai Chatbot For Teaching A Foreign Language: An Empirical Research. International Journal of Scientific & Technology Research., 8, 897–902.
Northern Illinois University (n.d.). Northern Illinois University (n.d.). Gagné’s Nine Events of Instructions. Centre for Innovative Teaching and Learning. https://www.niu.edu/citl/resources/guides/instructional-guide/gagnes-nine-events-of instruction.shtml
Report (2018). Artificial Intelligence Market in the US Education Sector 2018–2022, https://www.researchandmarkets.com/reports/4613290/artificial-intelligence-market-in-the-us?utm_source=BW&utm_medium=PressRelease&utm_code=5lshzz (accessed 7 November 2021).
Ryan, R., & Deci, E. (2000). Self-Determination Theory and the Facilitation of Intrinsic Motivation, Social Development, and Well-Being. The American Psychologist., 55, 68–78. https://doi.org/10.1037/0003-066X.55.1.68
Ryan, & Deci, E. L. (2020). Intrinsic and extrinsic motivation from a self-determination theory perspective: Definitions, theory, practices, and future directions. Contemporary Educational Psychology, 61, 101860–. https://doi.org/10.1016/j.cedpsych.2020.101860
Sims, R. (1983). Kolb’s Experiential Learning Theory: A Framework for Assessing Person-Job Interaction. Academy of Management Review, 8, 501–508.
Tamayo, P., Herrero, A., Martin, J., Navarro, C., & Tranchez, J. (2020). Design of a chatbot as a distance learning assistant. Open Praxis, 12(1), 145–153. https://doi.org/10.5944/openpraxis.12.1.1063
Tipton, E., Hallberg, K. and Hedges, L.V. (2017). Implications of small samples for generalization: Adjustments and rules of thumb. Evaluation Review 41(5). https://doi.org/10.1177/0193841X16655665
Topal, D., DilekEren, C., & Geçer, A. (2021). Chatbot application in a 5th grade science course. Education and Information Technologies, 26(5), 6241–6265. https://doi.org/10.1007/s10639-021-10627-8
Winkler, R., & Söllner, M. (2018). Unleashing the Potential of Chatbots in Education: A State-Of-The-Art Analysis. Academy of Management Proceedings (1), 15903–. https://doi.org/10.5465/AMBPP.2018.15903abstract
Zawacki-Richter, O., Marín, V., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education – where are the educators? International Journal of Educational Technology in Higher Education, 16(1), 1–27. https://doi.org/10.1186/s41239-019-0171-0
Author information
Authors and Affiliations
Contributions
All the authors were involved in the writing of this paper and approved the final manuscript for submission.
Corresponding author
Ethics declarations
Competing interest
The authors have no competing interests to declare that are relevant to the content of this article.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Shim, K.J., Menkhoff, T., Teo, L.Y.Q. et al. Assessing the effectiveness of a chatbot workshop as experiential teaching and learning tool to engage undergraduate students. Educ Inf Technol 28, 16065–16088 (2023). https://doi.org/10.1007/s10639-023-11795-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10639-023-11795-5