Keywords

13.1 Introduction

Technological advancement has boosted new learning methodologies in physics education (Sarwi et al. 2019). The emerging technologies have been based on theories of student-centered learning. Some courses continue to use traditional methodologies within physics education, where the instructor assumes the active role, causing the benefits of technology not to be fully exploited (Hwang et al. 2015). For this reason, it is crucial to design didactic strategies that integrate emerging technologies with student-centered activities.

13.2 Literature Review

One such recently implemented strategy is flipped learning, a pedagogical approach with two main characteristics (Nganji 2018; Scager et al. 2016). When applying the flipped learning methodology within physics education, it is necessary to consider two key points: (i) what emerging technologies support direct individual physics instruction? (ii) what type of active learning activities will reinforce and enrich student learning? For the first question, technological resources should be considered, letting the student address the issues individually at the time and space required. This exploration opens the possibility of using tools offering ubiquity, such as chatbots. Such tools can provide reliable information while adapting to the learner's need for information.

Active learning strategies have been employed in physics higher education. One acknowledged methodology is Tutorials in Introductory Physics (TIP) (McDermott and Shaffer 2001). The goal of TIP is to construct concepts and develop reasoning skills using various instructional strategies (McDermott 2013). Many tutorials are available for the purposes and topics. Their instructions provide worksheets for activities recommended to be carried out in teams of three or four people.

The literature discusses how learning must be personally relevant to be effective, such as self-regulated learning anywhere (Fitzgerald et al. 2018). Many works conclude with a reflection for the educational community about personalized learning focused on individual choice and control. Their perspectives contrast with Sarsar et al. (2018), who mentioned that digital learners have three types of expectations from mobile technologies in their courses, classified as pedagogical, personal/individual, and technological. All are disruptive educational approaches.

One disrupting approach could be using a chatbot, a technological tool to assist any user search for information. According to Gupta (2020), chatbots are conversational platforms focused on specialized activities. Chatbots can communicate through text or voice and use artificial intelligence and natural language processing to understand the user's message (Khanna et al. 2015). These characteristics allow them to be used as support tools at any time. On the other hand, Winkler and Söllner (2018) specified that chatbot’s advantages are increasing user satisfaction due to their immediacy and their availability at any time, personalized attention to the user, and lower costs in service areas. Another advantage is that the data allows analyzing user needs to improve processes or services. Other uses are personalized interactions (Gonda et al. 2018), resources for tutorials, and cognitive skills development (Pogorskiy et al. 2018). For these reasons and the chatbot’s adaptability in mobile devices, the education sector has begun implementing them.

Chatbots can enable or accelerate student learning (Becker et al. 2017), but it is necessary to know how to implement them to achieve the desired learning outcomes. Winkler and Söllner (2018) claim to research “the integration of chatbots in the different stages of learning processes with the help of learning theories” to resolve the lack of empirical evidence on how they influence learning. One of the learning outcomes necessary for first-year engineering students is understanding Newton’s laws. Some investigations aim to develop technology strategies that help students improve their understanding of concepts in physics class (González et al. 2019; Pohan et al. 2020). These promote students’ understanding of Newton’s laws with the help of virtual laboratories. The emergence of technologies such as chatbots offers new ways to achieve learning outcomes. This research aimed to implement these tools within a physics class for understanding Newton’s laws. We proposed an intervention integrating a chatbot with a tutorial-type activity. Another purpose was to gather empirical evidence of the impact that this type of implementation has on learning. Thus, this research is based on the following questions:

Q1. What is the learning gain of Newton’s laws for first-year engineering students who carry out a tutorial-type activity with the “Professor Atom” chatbot? Q2. How does the chatbot’s use impact groups with different characteristics? Q3. What are first-year engineering students’ opinions about using the chatbot in the tutorial-like physics activity?

13.3 Design of the Chatbot

This project began in July 2018 to be integrated into Physics I through the modality of telepresence with a hologram. Telepresence is an educational innovation project of Tecnologico de Monterrey (Paredes and Vazquez 2019). It was complemented with artificial intelligence innovations designed and implemented for Physics I in 2017. This project evolved into the Professor Atom chatbot that could receive questions by voice and writing based on natural language and communicate with the student through dialogue. The help, explanations, exercises, criticisms, and discussions of a topic or problem were carried out through the system’s dialogue with the student (Medina et al. 2016).

For instance, Professor Atom was created as a chatbot (Pai et al. 2020), developed with artificial intelligence based on natural language to receive questions or general inquiries from the students about basic Physics topics. This chatbot provides an immediate response, thus simulating an academic tutor with 24/7 attention. The students can use their mobile devices when and where they need to learn a concept, speeding up the learning experience, saving time searching for basic concepts, and being accompanied during their learning process.

From a pedagogical perspective, Professor Atom chatbot was designed with active learning strategies to promote students’ learning by developing information search, analysis, and synthesis skills and motivating them to solve the problem examples presented (Friston et al. 2017; Ballen et al. 2017). The objective was authentic learning, in which the chatbot builds the students’ knowledge by relating new information to add to their previous knowledge, readjusting and reconstructing it during their learning process (Riddell 2017).

It was intended that the student using Professor Atom could resolve questions, reinforce knowledge, review or update notes, study for an exam or exposition, carry out individual and team tasks, and delve deeper into a concept than it would be done in the classroom (Docktor et al. 2016; Weliweriya et al. 2019). Considering the opinions of the Science and Engineering department professors and the Educational Innovation team, we decided that Professor Atom would address 129 basic physics concepts in four categories: definitions, formulas, examples, and video explanations. Some examples are shown in Fig. 13.1.

Fig. 13.1
Snapshots of professor atom’s answer to a question on Angular acceleration. Each query to the chatbot is addressed in the four categories: 1. Definition. 2. Formulas. 3. Examples 4. Video explanations.

Examples of professor atom chatbot answers

Full size image

13.4 Method

This research method consisted of seven steps where 145 students participated from different campuses and engineering majors in a private educational institution in Mexico. First-year students were enrolled in five different groups (classes). Each group had characteristics as follows: 37% of students (groups 1 and 2) were within a traditional educational model, of which 48% (group 1) were in a face-to-face mode, and 52% (group 2) in a remote modality. The other 63% of the participants (groups 3–5) were in a face-to-face model to develop competencies through challenge-based learning. Group 2 had students from four different campuses and cities in Mexico. Groups 3, 4, and 5 belonged to the same campus, and group 1 students belonged to a single campus different from the other groups. Groups 1 and 2 had students in different engineering majors, while groups 3–4 studied engineering within the biotechnology area. All students in group 5 were studying in a program related to computer engineering and information technology. Moreover, groups 2–5 had the same instructor.

The first step was introducing the students to interact with the chatbot in the classroom. This stage aimed to characterize the chatbot and avoid false expectations regarding its use. The chatbot has to support student learning, and the experience with this technology affects the learning outcomes (Harrati et al. 2016).

Second, we administered a pre- and post-test using the Half Force Concept Inventory (HFCI) multiple-choice exam (Han et al. 2015). Since only a few aspects of Newton's laws were to be evaluated on the test, we chose the HFCI exam as a tool. The test had 14 questions about seven force concepts, as shown in Table 13.1. The focus of this research was most related to three of them, namely, Newton’s third law, force motion, and force motion cluster, which represented 64% of the items in the test.

Table 13.1 Items by concepts of the HFCI
Full size table

As a third step of the experimental design methodology, an individual tutorial activity based on TIP was implemented outside the classroom. During the exercise, the student was encouraged to interpret concepts about forces and Newton's laws. Then, TIP was implemented inside the classroom. These phases correspond to the implementation of flipped learning.

In the fourth step, a post-test was implemented to explore the change in the students’ conceptual understanding through the normalized gain defined by Hake (1998). This variable measures the normalized learning gain after the students completed the methodology described in previous steps. The Hake’s gain is calculated as shown in Eq. (13.1), where Xpost represents the mean of the results obtained in the post-test and Xpre the mean of the diagnostic evaluation results.

$$g_{{{\text{Hake}}}} = (x_{{{\text{post}}}} - x_{{{\text{pre}}}} )/x_{{{\text{pre}}}}$$
(13.1)

Hake’s gain was analyzed through a descriptive and statistical inference analysis. Moreover, data exploration and arrangement were directly observed in the physics courses in groups 1–5. The final step was to gather students’ voluntary opinions on the chatbot to know their perceptions of Professor Atom’s usability.

13.5 Analysis and Results

First, the quantitative data were collected from 145 students in the five different groups (described in the Method section) in person and via the Learning Management System. The group analysis of the diagnostic test later moved on to the final test. Then it was decided to observe the behavior of the data, differentiating the results by group, to have a more detailed analysis and examine if there was a different behavior in each one.

The mean of correct answers per student, per group, and the general average were calculated to analyze the results for the diagnostic evaluation (pre-test) and final (post-test). Figure 13.2 displays the distribution of the individual averages of the students in both tests on a scale of 0–100.

Fig. 13.2
The box plot depicts the comparison between the pre-test and post-test. 1. Pre-test 34 percent. 2. Post-test 38 percent.

Comparison of the score (0–100) distribution between the pre-test and the post-test

Full size image

When comparing averages’ distributions obtained by the participants in the pre-test and post-test, we observed that the maximum value increased from 79 to 93. The minimum values for both distributions presented a value equal to zero. It was also observed that the mean increased from 34 to 38, although the median remains at a value of 36. Likewise, there is more dispersion of the post-test data than the pre-test. An increase in the interquartile range was observed, going from 22 in the pre-test to 29 in the post-test.

Three different gains were calculated for the entire group of participants. In the first, the average of all the HFCI items was considered. Only the average of the items that evaluated the concepts addressed by Newton’s laws was considered in the second. The third gain considered the averages of the items addressing other topics. The results are seen in Fig. 13.3a.

Fig. 13.3
Two bar graphs of students’ gains. (a) Percentage gain versus the question topics like H F C I, force and Newton’s law and other concepts is shown. (b)Percentage gain versus group number that compares Hake’s g each group considering Newton’s law concepts compared to Hake’s g for all items.

a Hake’s g by question type. b Hake’s g per group considering only Newton’s law concepts compared to Hake’s g for all items

Full size image

The students obtained an overall gain of 6.8%. If only the questions on the conceptual understanding of forces and Newton’s laws were examined, a gain of 8.1% was obtained, and 3.8% in the other topics. Thus, there was a more significant gain in the topics developed in the intervention. According to Hake (1998), these gains are considered low since a value of less than 30% was obtained. For a focused analysis, normalized Hake’s g was calculated for each of the different groups, considering only the items of Newton’s laws. These were compared with Hake’s g calculated, considering all the items for each group. As shown in Fig. 13.3b, there is a greater Hake’s g in three of the five groups versus Hake’s g of all the concepts. Also, it has a bold black line that remembers the gain tanking account of all the items for each group. Groups of both study plans, groups 2 and 3, showed a Hake g in Newton’s laws topics lower than all the items, with Hake g of 2% and 5%, respectively.

Next, the gain per student was calculated to know the results associated with the intervention. It was found that 50% of the students increased their conceptual understanding by showing a Hake’s gain greater than zero, considering that zero represents a standard reference. On the other hand, it was found that 25% of the students obtained a Hake’s gain equal to zero, and the other 25% scored a Hake’s gain less zero.

13.5.1 Quantitative Analysis: Descriptive and Parametric Statistics

A descriptive analysis was carried out for the total data of the students receiving the methodology described in this research work. First, a process to clean the data and eliminate outliers yielded 145 first-year college students in 5 different groups (classes). Table 13.2 summarizes the primary statistical findings among the five groups. Group 3 had the largest variability (standard deviation), followed by group 4. Notably, the mode and median value observed in each group equal 0 due to the variable Y = Hake’s gain. This behavior indicates a sample centered on the zero value, referring to those students who presented neither Hake’s gain nor loss in this experiment in educational innovation.

Table 13.2 Basic statistical analysis in groups
Full size table

A parametric statistical analysis was carried out first, performing the normality test for each group. Subsequently, a variance test was performed using Bartlett’s method and Bonferroni Confidence Intervals for Standard Deviation (see Fig. 13.4). The next step was to perform the Analysis of Variance test, thus statistically proving the Hake’s Gain behavior in the five groups.

Fig. 13.4
Graphs depict the test for equal variances. (a) Group versus Bonferroni confidence intervals for standard deviation, Bartlett’s test p-value equals 0.000. (b) Group versus Bonferroni confidence intervals for standard deviation, Bartlett’s test p-value equals 0.072.

Bonferroni confidence intervals for standard deviation for groups 1–5

Full size image

Afterwards, a normality test was performed on the data for each interest factor named as “group.” Table 13.3 shows a normal behavior of the data for each level of the factor of interest. All the groups present a p-value > 0.05 with a significance level of α = 0.05. The hypothesis is H0: Data follow a normal distribution; H1: Data do not follow a normal distribution. That is, all the data in the factors of interest (groups) followed a normal distribution.

Table 13.3 Normality test for groups of students
Full size table

Figure 13.4a shows the behavior of the variability of the factor of interest through Bartlett's test for variances. The hypothesis result gave a p-value < 0.05 with a significance level of α = 0.05 where H0: All variances are equal, H1: At least one variance is different. The conclusion was that there is not enough statistically significant evidence to accept H0. Therefore, at least one group differed in their standard deviation due to the Y = Hake’s gain.

Figure 13.4a shows that group 3 has the highest variance of all. Subsequently, Fig. 13.4b shows the result obtained by eliminating group 3 from the test. Then, on the right side, p-value > 0.05 with a significance level of α = 0.05 can be observed. With this iteration, the test conclusion could be defined as the variances between groups 1, 2, 4, and 5 are significantly equal per Y = Hake’s gain.

Immediately, an ANOVA test was performed for the groups with equal variances. The hypothesis defined was H0: All means are equal; H1: Not all means are equal with a significance level of α = 0.05, obtaining p > 0.05. Furthermore, Table 13.4 indicates there is no significant evidence to reject H0; all population means of groups 1, 2, 4, and 5 are equal per Hake’s gain. It is worth mentioning that a normal residual analysis was performed, where the assumptions of normality, homoscedasticity, and independence of the data were also validated.

Table 13.4 Analysis of variance
Full size table

Finally, Fig. 13.5 shows the Confidence Intervals for the mean, presenting similar behaviors per the variable Y = Hake’s gain for each one of the groups (including group 3). Also, group 4 has the highest value due to the studied variable Y, referring to the chatbot and methodology proposed in this research. On the other hand, groups 1, 5, and 2 have the lowest values of loss in this variable Y.

Fig. 13.5
Graph depicts the mean interval plot. The vertical axis represents Hake’s gain and ranges from negative 0.05 to 0.25. The horizontal axis represents the group and ranges from 1 to 4.

Means interval plot of Y = Hake’s gain and groups

Full size image

13.5.2 Qualitative Analysis: Usability Expressed by the Students

After the intervention, 90 students voluntarily expressed their opinions on the use of the chatbot in writing. The research group collected and classified these opinions as positive and negative to improve the Professor Atom chatbot. This classification emerged from students’ opinions. Table 13.5 shows that students preferred the technological and pedagogical functionalities of the chatbot, e.g.: (i) quick access to the content, (ii) ease of use, and (iii) the different (pedagogical) categories of each concept giving definitions and explanations. This feedback indicated that students appreciated the chatbot's representation versatility, which aligns with the experimental and practical postulates of Yuliati et al. (2018). Next, these opinions will be complemented with the other side of the coin.

Table 13.5 Feedback from students
Full size table

The students gave negative feedback (see Table 13.5), mainly regarding the content and technological design. First, the participants mentioned that the number of topics and concepts were limited. Second, the information was synthesized, so they asked for more in-depth explanations. Third, the students perceived that interactions could be improved through natural language questions. The students proposed that the chatbot stay connected for longer in idle time to improve the user experience.

13.6 Discussion

Equal gains in the conceptual understanding of Newton’s Law groups were observed among the groups. We conclude that the impact of the intervention was similar in the different groups despite the differences among them, such as the educational model, the teacher, the program curriculum, and the student campus.

To respond to research question Q1. What is the learning gain of Newton's laws for first-year engineering students who carry out a tutorial-type activity with the “Professor Atom” chatbot? We found that the intervention caused an increase in the level of conceptual understanding of Newton's laws because higher increases in Hake's g were obtained in the analysis of this topic. However, this value was low within the range considered by Hake (1998). Nevertheless, the effectiveness of chatbots depends on the students’ perception of this tool (Winkler and Söllner 2018). This intervention realized that previous user experience with the chatbot could affect Hake’s gain. Hence, future research suggests measuring the previous experience to see how it affects Hake’s gain.

Regarding Q2. How does the chatbot's use impact groups with different characteristics? The ANOVA test validated an equal behavior in four of the five groups studied, related to the gain of the conceptual understanding of Newton's laws. Therefore, it can be concluded that a significant difference among groups was not observed.

Last but not least, Q3. What are first-year engineering students’ opinions about using the chatbot in the tutorial-like physics activity? The students voluntarily expressed opinions, both positive and negative, about technological and pedagogical aspects of the user interaction; they requested more topics and concepts and versatility of representations.

For this research, how the students understand the technology and its impact on the learning outcomes was relevant. Students who perceived the merits of a chatbot showed greater interest in the activities carried out (Fryer et al. 2018). The chatbot’s integration with the activity had a different impact on the students, evidenced by more dispersion of the results in the post-test compared to the pre-test. Likewise, the improvements of the chatbot that students suggested would affect their learning experience, according to Liu et al. (2019).

13.7 Conclusion

The literature review showed how integrating emerging technology with tutorials made it possible for some students to understand Newton’s laws conceptually. In contrast, some other references provide findings of low values of conceptual understanding in students, which are also related to technology use. For this reason, this research brings elements to propose a novel method that incorporates chatbot technology and didactic methods to teach basic sciences to first-year students in engineering programs. With this in mind, the research group designed an experiment to collect qualitative and quantitative data to answer the research questions defined in Sect. 13.2.

The results from analyzing quantitative data indicated the equality of the population means of the groups for the Hake's gain, even with the peculiar characteristics of their teaching in the remote learning modality necessitated by COVID-19, the different campus locations of the professors and freshmen, and different teachers for the groups. These results provided statistically conclusive findings that the proposed methodology and the technological tool (Professor Atom chatbot) had the same significant impact on all the groups. Furthermore, the confidence intervals observed in Fig. 13.5 indicated a high probability that most of the students using this methodology and the chatbot obtained a positive Hake’s gain.

The qualitative data came from the opinions of freshmen volunteers to answer the second research question. The information demonstrates the chatbot utility and effectiveness and the requirements to develop more elaborated content. The chatbot's limitations necessitated students’ training to use it, and when this step was carried out, some students still had difficulties handling the chatbot tool.

In short, the low Hake’s g indicated that the research objective was achieved by integrating emerging technology such as a chatbot with tutorials, typical in teaching physics. The findings also allow visualizing areas of opportunity for future work: it is necessary to know the students’ previous experience with the use of chatbots and establish if there is a relationship with the Hake's gain. Likewise, it would be preferable to increase the number of participants and extend the instructional design to public universities and high schools. The research group must generate an updated Professor Atom process analyzing unanswered questions and generating more content from our teachers, which the students value highly. This update will be implemented to achieve a better understanding of students’ needs and uses.