1 Introduction

Chatbots are tools that combine artificial intelligence and natural language processing or different technologies and provide human communication and interaction through text and voice (Pérez et al., 2020). Chatbots are a technology that effectively supports interpersonal communication and learning (Hwang & Chang, 2021). Although chatbots are used in many areas, especially in customer relations, they can provide great opportunities especially in the field of education (Khan et al., 2019; Wang et al., 2021). The use of chatbots in education can be used to employ interactive methods, to provide learners with different types of information, and to guide learners (Hwang & Chang, 2021; Muniasamy & Alasiry, 2020; Poncette et al., 2020). On the other hand, chatbots help to improve interaction, guidance, and feedback (Dekker et al., 2020; Gonda et al., 2019), increase satisfaction in learning experiences (Kim et al., 2019), peer, content, environment communication (Haristiani et al., 2019; Hill et al., 2015), and personalized learning (Pérez et al., 2020), increasing learning performance (Kumar, 2021; Pereira et al., 2019; Wu et al., 2020) and foster collaborative learning and teamwork (Kumar, 2021; Schmulian & Coetzee, 2019).

Recently, the rapid increase in the penetration of mobile technology and the use of social media environments facilitates the adoption and spread of the use of new technologies in education (Yildiz-Durak, 2019; Yildiz Durak & Saritepeci 2019). The easy integration of chatbots with social media environments and instant messaging applications (WhatsApp, Facebook, Telegram, etc.) also expands the usage perspective (Kumar, 2021). On the other hand, the ability of chatbots to integrate into many platforms and the widespread acceptance of mobile-based chatbots (Siri, Amazon Alexa, Google Assistant, etc.) may pave the way for the use of chatbots in education. Different terms can be used for chatbots due to these various technologies, speech artificial intelligence, natural language processing, use of virtual assistants (Rheu et al., 2021). In this study, while the term chatbot was used as a tool, the intervention to guide learners via chatbot was called conversational agent-based guidance.

The purpose of this study was to apply chatbot technology as a guidance tool in educational environments and to model its effects on visual design self-efficacy, engagement, satisfaction, and learner autonomy at the end of the process. We hope that this study will provide information on the relationship of key variables for the design and implementation of a chatbot focused on digital visual design education and the effectiveness of the learning environment.

2 Conceptual Framework

2.1 Visual design self-efficacy

Visual design, which has a wide range of subject and application areas, is about digital competencies, aesthetics, originality, creativity, and conveying the message correctly. Visuals are used in many different fields, and visual design is taught in many different disciplines (Uzunboylu & Oz, 2016). However, it is important that the visual design is appropriately structured according to various conditions and that the teaching is individualized. Since visual design performance is also related to individual competencies, it was thought that Chatbot designs will be effective in the individualization of teaching. This can positively affect visual design self-efficacy. In this context, the following hypothesis was proposed in this study:

H1: Frequency of using chatbot guidance has a positive effect on visual design self-efficacy.

H2: Chatbot usage satisfaction has a positive effect on visual design self-efficacy.

2.2 Satisfaction

Satisfaction in educational environments is an important and important variable that affects academic performance. Martín-Rodríguez et al., (2015) emphasized the relationship between high satisfaction in the learning process and high academic achievement and performance. Satisfaction is important in an effective and efficient online learning environment. The educational process can be organized more effectively and efficiently by examining the factors that affect students’ satisfaction levels (Gülbahar, 2012). In this study, the effect of chatbot usage frequency and satisfaction on learning satisfaction was investigated. In a study by Nguyen et al., (2021), proficiency and satisfaction were found to be associated with chatbot environments. In this context, the following hypothesis was proposed in this study:

H3: Frequency of using chatbot guidance has a positive effect on satisfaction.

H4: Chatbot usage satisfaction has a positive effect on satisfaction.

2.3 Learner autonomy

Learner autonomy is the ability of the learner to take responsibility for learning, to set learning goals, to define content and progress, to be active in all processes, and to take responsibility for decisions (Holec, 1981). Autonomous learners learn more efficiently and effectively because they have control over their learning processes and control their learning (Lan, 2018). In this context, the use of chatbots can be considered as a supporting element for learners’ autonomy. In the study by Haristiani & Rifai (2021), chatbots are suggested as an attempt to provide an alternative autonomous learning environment. Learners can decide more easily how to learn by using chatbots. In the study by Nguyen et al., (2021), it was found that perceived autonomy is related to chatbot system satisfaction and performance. In this context, the following hypothesis was proposed in this study:

H5: Frequency of using chatbot guidance has a positive effect on learner autonomy.

H6: Chatbot usage satisfaction has a positive effect on learner autonomy.

2.4 Engagement

Engagement is a measure of learner effort to develop knowledge, skills, and competence aimed at learning goals (Newman et al., 1992). Engagement is a concept related to the extent to which students are interested in their learning, to what extent they are involved in learning, and how they connect with other students and their classes (Axelson & Flick, 2010). According to the social constructivism context, engagement, one of the most important components of an effective learning process, is that students spend time and effort to learn the course content and skills, have a meaningful interaction with other people, and are emotionally involved in learning processes. In this study, the engagement framework developed by Dixson (2010) and adapted to the cultural context of Turkey by Polat, Hopcan, and Kamalı-Arslantaş (2022) was used. This framework includes students’ active participation in online learning processes, their feelings of learning, their relationships in terms of content, performance, and emotion.

It was thought that with the use of chatbots in education, a system that responds to learners at any time will support engagement and contribute positively to learning performance. Chatbots that respond appropriately to user input can foster emotions and active participation in learning environments. In this context, the following hypothesis was proposed:

H7: The frequency of using chatbot guidance has a positive effect on engagement.

H8: Chatbot usage satisfaction has a positive effect on learner engagement.

2.5 Chatbot design for design education and conversational agent-based guidance

Chatbots provide feedback that replicates natural speaking style to execute instructions based on specific inputs. According to Adamopoulou & Moussiades (2020), chatbots contain elements that will support communication and interaction in terms of information and guidance during the design and development process. Educational chatbots are designed for clear learning goals and can provide opportunities to help achieve those goals. Learning areas and learning objectives are important for the educational design of the chatbot. For example, Huang et al., (2021) emphasized that chatbots can be important in reducing students’ engagement and speaking anxiety in language education, and these factors will change the design. It can guide learners in reviewing content. The chatbot can actively remind users every day to review learning content, including tips, additional information, referrals to lesson videos, and lesson resources to be reminded of what was learned.

On the other hand, scripts can be written in the educational design of chatbots, as well as pre-set flows, commands, and dialog procedures can be used on various platforms (chatter, Flow XO). Through this defined flow, information inquiry and guidance services can be provided. Integration of these streams with communication platforms such as Facebook Messenger, WhatsApp, Telegram, and Slack can enable their widespread and always use. The chatbot built into communication platforms supports their widespread use as educational, as it does not require downloading additional applications. In this study, the flows were made in Flow XO and the integration of the created bot was made on the Slack communication platform. A guidance framework was created using a customized knowledge base for digital visual design. The chatbot provides the student with the most appropriate guidance/answer for the entered word/input at any time.

3 Method

3.1 Research Model

The model put forward in this study was presented in Fig. 1.

Fig. 1
figure 1

The Research model

Full size image

3.2 Participants and Procedure

The participants of the study are 86 university students studying at a state university in Turkey. 61.6% of the participants have no experience in visual design. 16.3% of the participants of the study were female and 83.7% were male.

Within the scope of this study, it was aimed to develop a chatbot to be used in and after the lesson. For the development of the chatbot, first, the topics of information guidance on visual design were determined. These titles are the definition and scope of visual design, visual design principles, explanations of visual design principles, visual communication, aesthetics, and digital visual production techniques. The clues were created in these titles. Then a flow was created for the chatbot design. In this flow chart, planning was made for which clue will be presented in the question containing which keywords. Chatbot design is in the “Keyword Recognition-Based” category, one of the design categories by Smutny & Schreiberova (2020). Therefore, the condition “containing the relevant word” was used in the query that the chatbot will make and the answer it will produce. For student questions that do not contain any of the words in the definition, the chatbot directs the students to the web page where the contents of the course were presented.

After the chatbot content design, these definitions were placed in the chatbot. First of all, a free environment was chosen for this. The Flow XO environment was chosen for its simple interface, easy-to-use possibilities, and integration with many platforms (https://flowxo.com). Flow XO is a tool that allows you to quickly and simply create AI chatbot solutions that help communicate and increase engagement across a wide variety of sites, apps, and social media platforms. For the definitions created for the chatbot, an account was first created in Flow XO (See Fig. 2). Since the account was a free trial version for 15 days, a different account was opened at the end of 15 days and the flow was created again. Slack was chosen as the messaging platform. The created chatbot was integrated into the Slack messaging platform. All students participating in the study have a slack account and actively use this environment. The reason for choosing Slack was that all students use this environment in their course activities. A name was determined for the bot and Slack API and Workspace were created for the application. Flow XO connection was established with Slack and chatbot students were made available in Slack. The implementation process took a total of 4 weeks. First of all, chatbots and their functions were explained to the students to be applied. Each student managed their learning process. Students performed digital visual design tasks during the process.

Fig. 2
figure 2

Some of the Chatbot Design -Screenshots

Full size image

When the chatbot was opened, the “Opening Message” appears. This message contains information about how to use the robot and its content. Visual design guidance was the subject of chatbot training. If the students do not find the information guidance provided with the chatbot sufficient, they are directed to the youtube lesson videos for detailed information. Looking at the chatbot analytics, it was seen that the students mostly asked one- and two-word questions to the chatbot during the implementation process. The chatbot was mostly used to learn visual design principles and hexadecimal color codes. After the learning process, data collection tools were applied to the students.

3.3 Data Collection and Analysis

In the study, data were collected with four scales. The first scale was the satisfaction scale developed by Kolburan-Geçer and Deveci-Topal (2015). This scale was intended to measure satisfaction levels for e-courses. The scale has 34 items and 5 factors: (a) course content and teaching process, (b) used materials and communication tools, (c) attitudes towards e-learning, (d) media design, and (e) teacher-student interaction. The scale is 5-point Likert type. The validity and reliability findings calculated for the factors in the scale were presented in Sect. 4.

The second measurement tool is the Learner Autonomy Scale, which was developed by Macaskill and Taylor (2010) and adapted into Turkish by Alkan & Arslan (2019). The scale has a 5-point Likert rating. It consists of 12 items and 2 sub-dimensions. One dimension of the scale was used in this study. Scale items are in a 5-point Likert structure. The validity and reliability findings calculated for the factors in the scale were presented in Sect. 4.

The third measurement tool was developed by Dixson (2010) and Polat et al. (2021) is an online engagement scale adapted into Turkish. There are four factors in the scale: (a) skills, (b) emotional, (c) participation, (d) performance. The scale is in 5-point Likert type. The validity and reliability findings calculated for the factors in the scale were presented in Sect. 4.

The fourth measurement tool is the self-efficacy scale for visual design applications. This tool was created by the researcher using the visual designs rating rubrics developed by Kılıç (2020). The created form was submitted to the opinion of 4 experts. The validity and reliability findings calculated for the factors in the scale were presented in Sect. 4. In the analysis of the data, the variance-based structural equation model was carried out with the partial least square method. The data were analyzed in the Smart PLS 3.0 program.

4 Findings

4.1 Measurement model

The measurement model of the structural model established in the research was evaluated with convergent and discriminant validity. Factor loadings, average variance values, and reliability values were evaluated for convergent validity. Items with factor loadings below 0.50 were excluded from the model. The factor loadings of the indicators of the structures vary between 0.50 and 1.00, and all are above 0.50 (see Appendix Table A). Heterotrait-monotrait ratio of correlations (HTMT) values were used for discriminant validity and it was recommended that these values be below 0.90 (Hair et al., 2017). Variance inflation factor (VIF) values of the items were also examined, and all values were below the recommended threshold value of 10 (Hair et al., 1995). According to Hair et al. (1995), VIF values above 10 indicate a multicollinearity problem. As a result, Cronbach’s alpha, composite reliability, and Average variance extracted (AVE) values in the measurement model were found to be within the recommended ranges in the literature (see Appendix Table B). Sufficient evidence was obtained regarding the validity and reliability of the measurement model. HTMT values for discriminant validity were examined and the findings are presented in Appendix Table C. It can be said that all HTMT values are below 0.90 and provide sufficient evidence for discriminant validity.

4.2 Structural model

In this section, findings related to the structural model were given. When the structural model proposed in this study was tested, it was determined that it explained 58.6% of the variance in the visual design self-efficacy dimension. The coefficients between the structures were presented in Table 1; Fig. 3.

Fig. 3
figure 3

SEM

Full size image

Figure 3 shows the results of hypothesis testing. Details on hypothesis testing were presented in Table 1. Hypothesis 1 argues that there will be a relationship between the frequency of using chatbot guidance and visual design self-efficacy. H1 hypothesis was not supported (p > 0.05). Hypothesis 2 argues that a relationship will be observed between chatbot usage satisfaction and visual design self-efficacy. The H2 hypothesis was supported (p < 0.05). This indicates that students with higher chatbot usage satisfaction have higher visual design self-efficacy. Hypothesis 3 argues that a relationship will be observed between the frequency of using chatbot guidance and course satisfaction. There are 5 sub-hypotheses under this hypothesis. H3a, H3b, H3c, H3d, and H3e hypotheses were not supported (p > 0.05). Hypothesis 4 argues that a relationship will be observed between chatbot usage satisfaction and course satisfaction. There are 5 sub-hypotheses under this hypothesis. While hypotheses H4a, H4c, H4e were supported (p < 0.05), H4b, H4d, and hypotheses were not supported (p > 0.05). Hypothesis 5 argues that there will be a relationship between the frequency of using chatbot guidance and learner autonomy. H5 hypothesis was not supported (p > 0.05). Hypothesis 6 suggested that a significant relationship will be observed between chatbot usage satisfaction and learner autonomy. H6 hypothesis was not supported (p > 0.05). Hypothesis 7 argues that a relationship will be observed between the frequency of using chatbot guidance and engagement. There are 4 sub-hypotheses under this hypothesis. Hypotheses H7a, H7b, H7c, and H7d were not supported (p > 0.05). Hypothesis 8 argues that a relationship will be observed between chatbot usage satisfaction and engagement. There are 4 sub-hypotheses under this hypothesis. H8a, H8b, H8c, and H8d hypotheses were supported (p < 0.05). This indicates that students with higher chatbot usage satisfaction have more course engagement.

Table 1 Hypothesis test results
Full size table

5 Discussion

The purpose of this study was to examine the effect of chatbots on visual design self-efficacy, engagement, satisfaction, and learner autonomy in the context of digital visual design education. Research findings and discussions were presented below.

The H1 hypothesis, which argues that a relationship will be observed between the frequency of using chatbot guidance and visual design self-efficacy, was not supported. On the other hand, H2 argues that a relationship will be observed between chatbot usage satisfaction and visual design self-efficacy, and the H2 hypothesis was supported. This result indicates that students with high chatbot usage satisfaction have higher visual design self-efficacy. Visual design is used in many different fields, related to many factors based on individual backgrounds, such as digital competencies, aesthetics, originality, creativity as a subject, and application area (Uzunboylu & Oz, 2016). In this context, a direct study that overlaps or contradicts this finding has not been found in the literature. However, in the study conducted by Yuan and Peng (2021), which is similar in terms of procedural information presentation, it is recommended to place an educational or practical video on the chatbot to improve learning performance and increase students’ interest in learning. In this context, it can be said that the frequency of use of only chatbot-based cognitive guidance will not be meaningful on its own, ensuring the satisfaction of use and the content presentation formats accessed are also important.

Hypothesis 3

argues that a relationship will be observed between the frequency of using chatbot guidance and course satisfaction, and hypothesis H3 was not supported. On the other hand, H4 argues that a relationship will be observed between chatbot usage satisfaction and course satisfaction. Under this hypothesis, H4a, H4c, H4e hypotheses were supported, while H4b, H4d and were not. Nguyen et al., (2021) emphasized the relationship between perceived competence and satisfaction in the use of chatbots. In this context and within the framework of research findings, it will not be sufficient and meaningful to follow the frequency of use in chatbot use to increase satisfaction with learning environments. However, ensuring user satisfaction with the system will play an important role in increasing the overall satisfaction of the course. Indeed, chatbot usage satisfaction was positively related to course content and learning process, course design, course materials, and communication satisfaction.

H5 and H6 hypotheses, which argue that a relationship will be observed between frequency of using chatbot guidance and chatbot usage satisfaction and learner autonomy, were not supported. However, Chatbots are suggested as an attempt to provide an alternative autonomous learning environment in the literature, and it is stated that they can provide a flexible decision-making environment for learner autonomy (e.g. Haristiani & Rifai 2021). In this context, the results of the research do not coincide with some results in the literature. The reason for this may be user factors such as privacy concerns about chatbots, unfamiliarity with chatbots mentioned by Han and Lee (2022). These situations can negatively affect learners in autonomous decision-making.

H7 argues that a relationship will be observed between the frequency of using chatbot guidance and engagement, while H8 argues that a relationship will be observed between chatbot usage satisfaction and engagement. The H7 hypothesis was not supported, while the H8 hypothesis was supported. This finding shows that chatbot usage satisfaction has a positive effect on students’ course engagement. In their educational chatbots review study, Smutny & Schreiberova (2020) stated that the main features of chatbots are that they are a tool that can be used to support interaction and determine the goals, strategies, and results of learning and education. In this context, it is not surprising that chatbot usage satisfaction positively affects students’ course engagement in terms of emotion, performance, participation, and abilities.

6 Limitations and future research directions

One of the main limitations of this study has to do with the number of participants. Visual design training was given and data were collected from 86 students at the end of this training. Validation of the proposed research model with a wider audience can be done in future studies. On the other hand, it was seen that the use of chatbots is frequently applied especially in language education. This study constitutes an example of its use in visual design education. In future studies, chatbot technology can be used in different learning areas by supporting it with learning pedagogy and andragogy. In this way, the relevant literature can be enriched.

Another limitation of the study was related to the demographic characteristics of the participants. The majority of the participants are male. This may have affected the research results due to gender-related different technology usage intentions and habits. The Chatbot was designed only for five topics of visual design in this study. Broader subject frameworks such as visual design programs applications, print and publishing processes, advanced digital design, and web programming procedures can be realized with chatbot-based teaching.

Another limitation of the study was the self-report data collection, not based on real chatbot usage analytics. On the other hand, it was stated in the literature that participants’ attitudes towards chatbots changed in terms of perceiving chatbots as human, non-human, or something in between (e.g. Han & Lee, 2022). In this context, based on real user data and learning analytics, perceptions of chatbots as Conversational agents can be clustered and the differentiation of the results according to the determined profiles can be examined. On the other hand, no study was done on the usability of the chatbot designed in this study. Before the use of chatbots in education, usability tests can be developed, and a design framework can be developed according to the intended use of the chatbot.

7 Conclusions

Conversational agent-based guidance was given to students through chatbot technology. Thus, with cognitive guidance, the time for students to receive feedback and access information was shortened, allowing them to devote more time to learning practices. In this context, the results of the study showed that the frequency of use of the cognitive guidance presented with the chatbot did not positively affect any of the visual design self-efficacy, engagement, satisfaction, and learner autonomy. Interestingly, the frequency of interaction with the chatbot did not positively affect learning engagement. Similarly, if we assume that the frequency of interaction with the chatbot is the frequency of receiving cognitive guidance from this environment, the learning environment satisfaction, visual design self-efficacy, and learning autonomy of the students who received more cognitive guidance were not positively affected. These findings paint a complex picture of how the effects of the use of conversational agent-based systems were reflected in learning processes. On the other hand, satisfaction with this technology positively affects visual design self-efficacy, course content and learning process, course design, course materials, communication satisfaction, and engagement. From this point of view, it can improve students’ learning experiences by providing chatbot usage satisfaction.