Keywords

1 Introduction

A new trend concerning natural language-based human-computer interfaces has emerged in current research: the use of chatbots in university settings [1] or intelligent learning systems to provide individualized and personalized learning support [2, 3], which was also shown in [4]. Driven by the digitization of society in general and of work in particular, chatbots have previously often been introduced in business contexts like customer support or to assist employees in their daily work [5, 6]. In these cases, chatbots should reduce service costs and handle multiple user inquiries at the same time, 24 h a day and independently of the availability of human resources [7]. Due to positive experiences in the business context, chatbots have been transferred to the university setting. Exemplary scenarios are individual learning support or assisting students in their personal study organization. Like in the business context, chatbots in university settings should support learners during the transition process and provide help 24/7 regardless of the device or the interface used. Additionally, they answer individual questions regardless of whether particular university terms are used or concrete university-specific questions are raised [1].

Even though some research on chatbots exists in educational settings, there is, to the best of our knowledge, currently no consideration of actual student requirements for a university chatbot for FAQ-like questions [4]. Prior research studies often only focus on particular use cases and designing corresponding chatbots. However, the results of these first studies promise positive outcomes for a university application. Therefore, as a starting point, first instantiations of university chatbots should address the provision of organizational information based on FAQs to evaluate the acceptance and general requirements at first. In prior research, first studies already investigated this by developing different chatbots for university settings [1, 8]. Hereto, we aim at surveying the actual student’s demands to provide a meaningful chatbot. Thus, the aim of our study is (1) to identify technical requirements for chatbots, and (2) to explore topics and related exemplary questions that should be answered by chatbots in a university setting. Based on an empirical questionnaire study among students at a German university, we address the following research questions:

RQ1::

Which technical requirements do students anticipate for chatbots in university settings?

RQ2::

Which content-related requirements have to be addressed by chatbots in university settings?

To answer these questions, the remainder of this article is structured as follows. Next, we briefly point out related research in Sect. 2. Afterward, we describe the research design in Sect. 3 and present our findings in Sect. 4. We complete our article with a discussion of the results in Sect. 5 and a brief conclusion in Sect. 6.

2 Background

2.1 Chatbot Basics

In general, a chatbot is an application system that provides a natural language user interface for the human-computer-integration. It usually uses artificial intelligence and integrates multiple (enterprise) data sources (like databases or applications) to automate tasks or assist users in their (work) activities [9].

Usually, the chatbot’s architecture is composed of three components that are used via the human-computer interface (see Fig. 1): (1) The natural language processing, which is responsible for (a) processing the user input – audio or text – into a machine-readable form by analyzing, dismantling and pattern extracting, as well as (b) generating a natural language output corresponding to the results of the dialog manager. (2) The dialog manager, which matches the user input against integrated backend systems and extracts content or executes functions. (3) The backend, which contains all relevant application systems or databases that are required for the desired application area in order to be able to process the user request [4].

Fig. 1.
figure 1

The architecture of a chatbot

Full size image

2.2 Chatbots in University Settings

Currently, chatbot research receives a lot of interest, and many researchers focus on this research topic from different perspectives. As shown in Meyer von Wolff et al. [4] and Maedche et al. [10], chatbot research mainly focuses on the application areas of customer support [11], information acquisition [12] as well as on business processes [13]. For university settings or rather educational scenarios, chatbot research is just beginning. Here, different studies and research streams are pursued:

In a recent literature review, Hobert and Meyer von Wolff [2] surveyed the current state of the art for pedagogical conversational agents. As shown in the publication, a trend for designing messenger-like chatbots has been identified. Further results of the analysis are that the current literature lacks on generalizable results. In a similar study, Winkler and Söllner [3] also conducted a literature review. The authors show that educational chatbot research is just in its beginnings, with a suggested potential for this application area. However, they note that the efficiency strongly depends on the individual student requirements, the way the chatbot is built, and the process quality. Those results confirm the need for surveying requirements for chatbots in universities.

Extending this, some studies have already presented first concepts and prototypes in this field of research. For instance, Fonte et al. [14] developed an intelligent tutoring system capable of providing learning content and a possible assessment of the student through the dialog. Mikic et al. [15] conducted a similar study in order to provide course content and a question-based assessment using a chatbot. In Carayannopoulos [1], a chatbot for information acquisition in universities was presented. The chatbot can respond to students’ inquiries about upcoming events or courses, leisure activities, or pending tasks. Additionally, Shawar et al. [8] and Shawar [16] describe an FAQ chatbot in a university setting. In Shawar [16], an extension with preprocessed and stored online available FAQs is shown. Both chatbots generate the answers either on a complete match or on a match based on the first or second most significant word. Additionally, Ranoliya et al. [7] examine university FAQs by developing a concept for a corresponding chatbot. Furthermore, Feng et al. [17] provide a concept for a Q&A chatbot that is capable of answering student questions in a natural way and of creating an efficient learning environment. Hien et al. [18] conducted an empirical study to examine the requirements of a university chatbot for answering students’ questions. The derived requirements are also conceptualized. Finally, Allison [19] surveyed the application of chatbots in libraries. With the presented chatbot, students can get answers on services or available resources of a library.

To sum up, and as shown in Meyer von Wolff et al. [4], one critical aspect of the current state of scientific knowledge is the lack of coverage of the design science process in general. In many cases, only particular phases are addressed. The investigation of specific requirements for selected use cases is missing. Only Hien et al. [18] followed a similar approach to survey the actual students’ requirements for providing a meaningful chatbot. Therefore, as stated earlier, it would be best if, as a starting point, real-case requirements are collected from future users in order to provide a meaningful chatbot in a university setting.

3 Research Design

To identify students’ technical requirements in university settings (RQ1) as well as content-related requirements (RQ2), i.e., topics and questions to be addressed, we conducted a questionnaire survey among students at a German university. Hereto, our study followed a three-step process:

First, we created a questionnaire based on previous findings [4, 9] comprising qualitative and quantitative questions. After a short introduction of the research project, which included a definition of chatbots to ensure clear understanding (see Sect. 2), questions – categorized in three sections – were interrogated: (1) general questions about the participant, (2) questions about the current or previous procedure of the students to acquire information and their satisfaction with it; and (3) questions about their experience and valuation of chatbots as well as topics to support and issues to answer. Before the data collection, we did a pilot test with multiple research associates who already had experience in questionnaire studies. Following, we rephrased some questions and added further questions for assessing a university chatbot and the target platforms. An overview of the final questionnaire is depicted in Table 1.

Table 1. Questionnaire structure
Full size table

Second, we conducted the survey within a two week timeframe in June 2019. Therefore, we announced the survey in different lectures, among student assistants as well as through social media postings, e.g., on Facebook, which was shared in several university groups as well. Overall, 530 students accessed the questionnaire, of which 214 students participated (40%). After cleaning the dataset of invalid data entries, we used 166 data sets (31%) for further analysis. Overall, the processing time for each student took 2 to 13 min (mean: 6:30 min).

Third, we analyzed the datasets in two ways. Hereto, we evaluated the quantitative data with spreadsheet programs. The qualitative data on topics and questions were categorized independently by two researchers on the subject and finally merged during a subsequent joint verification.

4 Survey Results

In the following, we present the results of our study. Therefore, we first show the sample description (Sect. 4.1). Afterward, we highlight the technical (RQ1) and content-related requirements (RQ2) in Sects. 4.2 and 4.3. Lastly, a short usefulness assessment is presented in Sect. 4.4.

4.1 Sample Description

Our study sample (n = 166) consists of mostly male students (58%), followed by 36% of female students. Nine participants have not answered the question.

We mainly acquired bachelor students (n = 87; 52%) followed by master students (n = 58; 35%). Additionally, some participants target a doctoral (n = 4), a state examination (n = 6), or other (n = 6) degrees. Five participants have not answered the question.

Most participants are in their first four semesters: 38% in the first two (n = 63) and 37% in the following two semesters (n = 61). Also, 25% of the participants (n = 15) are in a higher semester (7th semester or greater). Thus, students from all graduation levels and all semesters participated.

For the distribution of the subject area, we aimed at a cross-section among all students from our university. Therefore, we tried to acquire students from all available fields of study. Our participant group consists mostly of economic science students (n = 102; 61%). The following fields of study have a much lower proportion: 16 from mathematics and computer science, 13 from agricultural and forestry science as well as humanities and cultural science, 11 from natural science as well as social science, 9 from teaching professions, 7 from law, as well as 3 from theology and 2 from medical science. Additionally, the students were able to make multiple entries for their field of study. Therefore, economics science is overrepresented (n = 102). This might be explained since we teach in this area and mainly approach students via our lectures. Nevertheless, economic science consists of subgroups that are, in addition, different from each other. Nonetheless, we were still able to acquire participants from all disciplines, at least.

We also measured the actual experience of the participants with chatbots in general (see Fig. 2). Most of our participants (41%) already use chatbots at least on an occasional basis. On the downside, 34% of the students have not used a chatbot at all. Among these, ten participants stated that they have privacy concerns, e.g., “Where they are used, I have concerns about privacy” or “permanent possibility of interception”. In addition, nine participants rated the use as too cumbersome or had problems with the chatbot functions, e.g., “Slow, a lot of unnecessary communication, no good answers, answers too inaccurate, writing often more complex than clicking, etc.”, “Chatbots are good for basic information that you can usually find on the website anyway” or “I find information as a list better”. In contrast, 17 participants stated that there are no reasons against using chatbots. Up to now, no situation has emerged, e.g., “It has not yet happened, there is nothing against it” or “Nothing, rather this has advantages, like a permanent availability”. Based on this, we conclude that many students already use chatbots or are willing to use them. Nevertheless, more than half of the students (n = 107; 65%) have already made first or more extensive experiences with chatbots. Thus, frequent use of the technology, also outside the university context, has already been identified.

Fig. 2.
figure 2

Frequency of previous usage (n = 165)

Full size image

4.2 Technical Requirements

Based on the questionnaire, we first analyzed basic technical requirements for a chatbot application in a university context (RQ1).

Therefore in the first question, we asked the students about the characteristics of chatbots [4] by means of a 5-step Likert scale (1: unimportant; 5: very crucial) (see Fig. 3). Based on the results, it is clearly shown that most students prefer the 24-hours-a-day availability. Therefore, they do not have to wait until human contact persons are available. In addition, the participants appreciate the fast response time combined with the direct assistance for the question that has arisen. Also, we have identified that the chatbot’s ability to respond individually to the user is not considered very important by users. Nonetheless, our participants rated all the characteristics as above average. Therefore, these should be addressed in potential university chatbots.

Fig. 3.
figure 3

Means of chatbot characteristics (n = 165)

Full size image

In a second question, we asked the participants for the chatbot operation platform (see Fig. 4). According to the students, the most relevant platforms for university chatbots are mostly WhatsApp or desktop and web interfaces. Whereas the former is difficult to implement due to the infrastructure and the specifications, the latter two are easier to realize. Among the other-category, we identified mostly Telegram (n = 9) but also XMPP or own apps as well as chatbots integrated into the university portals. Nonetheless, our selection options are not entirely free of overlaps; we could highlight the relevant platforms. Above all, chatbots should be integrated into the interfaces used by students on a daily basis. Due to the many selected platforms, it would be best if a chatbot were not limited to a specific platform. Instead, it should be possible to make a request from all platforms.

Fig. 4.
figure 4

Target platforms for university chatbots (n = 165)

Full size image

4.3 Content-Related Requirements

Furthermore, we identified content-related requirements in the sense of topics to be addressed or questions to be answered by a chatbot in a university setting (RQ2). Based on two open questions in the questionnaire, the participants were asked about short topic mentions and exemplary questions that we categorized afterward. In total, we acquired 503 statements concerning topics and 495 exemplary questions as a starting point. Following the categorization process, we jointly merged them into 36 question sections, partial with sub-sections, in six core topics for a university chatbot for students. A complete overview of the categorization is displayed in Fig. 5.

Fig. 5.
figure 5

Categorization of topics (n = 156) and question areas (n = 154) for university chatbots

Full size image

As shown, chatbots in university settings should address the topic of information around studying in general. At first, students would use a chatbot when looking for study programs or gathering information about the university in general. Also, some organizational issues should be answered like semester dues or times, as well as those regarding studying abroad. Second, the application area of chatbots for (upcoming) events and lectures seems interesting. A chatbot provides content of the offered courses and their dates and times or locations, as well as the responsible persons. In addition to events and lectures, a chatbot should provide support for examination-related questions. Similar to the previous category, information on the examination in general, as well as the room and date, are highly relevant. Moreover, organizational issues like regulations, contact persons, as well as information on prerequisites and how to register should be covered. Furthermore, the participants would inquire (personal) statistics or retrieve/request their certificates. Another application area, which should be taken into account in university settings, are the closely related institutions or departments. In our study, the participants noted the library, canteens and cafes, or the sport offers. These institutions and departments should be extended or adapted to the respective university so that students can obtain information on opening hours; food offers in the canteen, and so on. Furthermore, university chatbots should provide basic (IT-)support. As our participants specified, they want help with the WLAN or printer setup, when password matters occur, as well as with the provision of software provided by the university. Lastly, we identified some different general concerns relevant to chatbots in a university setting. This includes, for example, small talk and university news. Also, general room plans or people’s search should be provided in the form of an information desk. Additionally, the participants would like to have a job board to inquire about open vacancies or possible internships, and so on.

Overall, as the most-mentioned topic, the students voted for a chatbot that can answer questions regarding events or lectures (n = 135) or for examination-related information (n = 122). Even if only indicated by fewer participants, information around the study program (n = 78), the university institutions (n = 73), or the (IT-)support (n = 59) are potential topic areas for a university chatbot. Therefore, first instances, or, rather extensions to existing implementations should definitely address the two most mentioned topic areas if they have not yet been considered. Furthermore, in terms of questions, those two topic areas have most of the questions given by the participants. Out of this, we infer that students have had the most questions regarding these areas so far, as they have cited many concrete example questions.

In the case of questions, we gathered mostly questions regarding the overview and information for events and lectures (65 questions), e.g., “Which modules are offered for the subject this semester?”, “Which contents should be taught during the lecture/seminar?”. In addition, questions about times or deadlines for examination (60 questions), e.g., “When does the exam take place?”, “Until when can I unsubscribe for the exam?” or for events and lectures (40 questions), e.g., “Does the lecture take place on Wednesday?”, “When in the week does the module take place?” were given. Furthermore, we collected some sub-topics with no corresponding questions, e.g., current news and notifications, scholarships, general opening hours, or study guidance. As these sub-topics were stated by the students as potential topics, questions should be developed in order to be able to address these issues in the future.

It should be mentioned that many of the question areas show reciprocal dependency, e.g., questions for contact persons in general and examinations, or times and deadlines in nearly all topics. These highlight relationships to be mapped in implementations or, rather, in the knowledge base.

4.4 Usefulness Assessment

To underline the usefulness of chatbots, we also asked the participants about their assessment of the application of the technology in university contexts (see Fig. 6).

Fig. 6.
figure 6

Evaluation of usefulness by frequency of use (n = 164)

Full size image

Based on a 5-step Likert scale (1: unnecessary; 5: helpful), we wanted to know how the students would rate it if a chatbot were available at our university. Overall, the participants rated this with an average of 3,62, which means a tendency to be helpful could be derived. In a more detailed analysis based on the frequency of usage (see Fig. 2), an interesting trend could be identified. Our results show that the more often a student used chatbots before, the higher the average rating of usefulness is. Even if only a few students regularly use chatbots, they have the highest average rating for usefulness. In addition, in the group of students who have not used chatbots until now, the highest count for helpfulness could be measured. However, this could also point out an exaggerated expectation for chatbot technology. Furthermore, this group also has a nearly balanced distribution of the usefulness. In addition, the more often chatbots are used, the more specific the distribution is in terms of helpfulness.

5 Discussion

Based on our students’ questionnaire survey, we identified which technical requirements in the form of characteristics and target platform (RQ1), as well as content-related requirements in the form of topics and questions (RQ2), are most important.

Our findings show that a chatbot is highly relevant for the application in a university context, as many topics and questions arose, for which a chatbot is usable. Hereby, 65% of the students participating in our study have already had some initial experiences with the technology. However, many of the participants (35%) have not used a chatbot so far, of whom 30% of the participants, in principle, have nothing against usage. This is also shown in Sect. 4.4 as the participants who have not yet used chatbots rate the helpfulness highest. Overall, the students rated chatbots as helpful. Independently of prior experience, the average rating of all groups described in Fig. 6 is above the mean. Thus, we identified a positive attitude of the participants towards chatbots in university settings, which is also shown in Hien et al. [18].

Additionally, we asked the participants to rate the essential chatbot characteristics. As a result, the following characteristics were rated as most important: 24/7 availability, fast and direct response as well as acting as a central platform for information acquisition. Surprisingly, our participants rated the ability to respond personally to the user or previous conversations as lowest. This is in contrast to current purposes of the scientific community, e.g. [5, 18]. Even though we do not have further information on this topic, a reason could be that the participants regard a university chatbot mostly as a tool to provide simple and general organizational information around lectures or events, which is shown in Sect. 4.3. Hereby, non-personalized information or content is delivered. However, the second most named category is examination that requires personalization in order to provide a reasonable answer, e.g., to provide certificates or to respond with personal exam dates. Thus, we identified an inconsistency among our results.

Regarding the target platforms, even if all options were selected, a focus on WhatsApp, Siri, and desktop or web interfaces could be determined. Thus, for the design of a university chatbot, they should be supported. However, this indicates a much more important requirement: the use of a chatbot from different channels, depending on the available device at the time of need. This can also be derived from the characteristics, as a chatbot mostly provides an appropriate answer 24/7 and in a timely manner. For the design, this means that a university chatbot should be programmed openly or should have corresponding interfaces, e.g., as a web application. Additionally, the high mentions for WhatsApp or Siri could hint at a further design requirement: audio or spoken inputs, as they are being used commonly nowadays on these platforms.

As a further result of the analysis, we identified university events and lectures as well as examinations as the most relevant topics to be addressed by a university chatbot. Furthermore, most of the collected questions aim at locations, definitions of content, or dates and can be answered with short sentences. Mostly, these questions are rather task-oriented or pertain to organizational issues for educational concerns but do not focus on education via a chatbot. This can be a hint regarding the expectations of chatbot users and may underline the basic abilities that the technology must fulfill: providing short answers or, rather, solutions for organizational issues in the sense of FAQs whenever needed. Surprisingly, individual learning support or providing lecture content were not mentioned by the participants at all. Maybe our participants only think about their previous experiences with different chatbots and try to transfer this knowledge to the university setting. However, this contradicts the current research approach, which focuses mostly on chatbot-mediated education. Nonetheless, for universities, as surveyed in this study, a chatbot should primarily provide organizational issues around lectures or examinations. This is also reflected partially in the survey conducted by Hien et al.  [18].

As with every empirical study, there exist some limitations that need to be discussed. Firstly, the findings of our study are mainly dependent on the students’ responses and their willingness to participate. Therefore, we have tried to maximize the reach in order to acquire as many participants as possible. We have not limited the disciplines or other aspects to survey a cross-section in the research area. However, the sector of economics science is overrepresented, but we were still able to acquire at least a few students from all disciplines. Despite this, our sample is still suited to indicate the technical and content-related requirements. Secondly, based on the chosen research design in the questionnaire form, maybe some questions were misunderstood by some participants. We tried to mitigate this by conducting a pretest before the actual survey. Thirdly, our derived design requirements are only based on the findings as well as on argumentative deductive conclusions. Therefore, these should be implemented in a prototypical chatbot so that they can be evaluated in real case scenarios.

Even though our study focused on the student’s perspective on the application of chatbots in university settings and may have some limitations, our results seem to be valuable and useful for future applications of chatbots at universities. Based on our findings, we could highlight necessary platforms and characteristics as well as topics and areas of questions, which have to be addressed in the first instances. Nonetheless, our findings have to be verified in real case scenarios. For this purpose, chatbots should be set up with the help of our results. Afterward, the usefulness of chatbots should be evaluated in order to identify gaps in the knowledge base and to be able to assess the use of the technology in university settings or in other educational contexts.

6 Conclusion

In this research paper, we aimed at surveying the application of chatbots in university settings. We questioned our students concerning their technical requirements (RQ1) as well as topics and areas of questions (RQ2) that a chatbot should address. As a result of our 166 participants, we could derive that the characteristics 24/7 h availability and fast solutions, as well as Whatsapp or desktop user interfaces as target platforms, are most important. In addition, we identified six core topics along with 36 question areas, of which events and lectures, as well as examinations, are especially important.

These can be used as a starting base for future implementations. Therefore, our study can contribute to the knowledge base and the understanding of chatbots used in university settings in two ways: (1) as a starting point for implementations or prototypes for the specific area of universities or rather education, as well as (2) for further investigations in this research area in general, e.g., requirement analysis or acceptance studies among future users.