1 Introduction

Requirements Engineering (RE) is a process within Software Engineering that involves activities that are required to gather, create, and maintain a software product’s requirements specification. There has been an increasing interest in RE which has challenged academics to create and deliver high-quality RE courses at universities to provide students with a solid foundation of the subject matter. Requirements engineering courses have been widely introduced in many software engineering curricula as compulsory courses where the fundamental knowledge about activities, concepts, methods, and techniques are taught [32].

Although the importance of RE in software development is well recognized [3], students are not active in learning RE courses and find the courses to be boring without the opportunity to practice the RE methods and techniques that are taught in the classroom [32]. Moreover, several studies report on problems that students face in Requirements Engineering Education (REE), e.g., a lack of requirements elicitation skills in REE [5], lack of practice on RE activities [17, 52], being able to select and combine RE techniques that are suitable for a particular project [27], lack of interest in RE course [33], and a too strong emphasis on theory rather than on practice [33].

One of the main RE-related problems faced by teams is communication between developers and customers [57]. Requirements engineers must be able to communicate with the customers about the problem being solved in a language they understand [57]. Another issue faced in practice is to choose the most effective analysis and modeling techniques that are suited for a particular project [27, 57]. Therefore, to be able to make such a decision, students need to be familiar with a wide range of RE techniques [57].

To successfully teach students these necessary skills, it is not enough to teach RE courses in a traditional lecture/laboratory setting, it would be unrealistic to expect students to use RE techniques in an effective manner in the industry without any practical experience. In addition, according to [6], most students only learn about 50 % of what they see and hear, while they learn about 80 % of what they use and practice in a more realistic setting. According to Polajnar et al. [39], a more realistic Software Engineering (SE) experience is necessary for properly understanding why the discipline exists. However, to create such a realistic setting is difficult as it needs stakeholders who are either real or behave in a realistic way [22]. In REE literature, two possibilities of creating a realistic setting are presented: (1) have access real stakeholders with real needs, or (2) have stakeholders that are simulated by other persons [39].

In this paper we describe the RE course we taught with a focus on one specific technique, namely the group project using a role playing method [57]. The group project addresses the needs for practicing elicitation skills and the ability to choose the right RE techniques for a particular project. We report our experiences and the results from the course evaluations from Lund University, Sweden and Chalmers | University of Gothenburg, Sweden, between the years of 2007 and 2014 where in total 412 students participated in the RE course. We look into if a higher grade in the role playing project has an effect on the students’ exam scores.

The remainder of this paper is organized as follows. Section 2 offers an overview of RE education and the use, the benefits, and the design of role playing in education. Section 3 presents the research methodology and discuss the validity threats of the design of the study. Section 4 presents the RE course, while Sect. 5 describes the project. The results are presented and discussed in Sects. 6 and 7 discuss lessons learned. Section 8 gives a summary of the main conclusions.

2 Literature review

This section begins with an overview of RE education followed by a brief coverage of role playing in education, its benefits, and how to design and use role playing.

2.1 RE education

In the past few years, there has been an increasing emphasis on incorporating RE into university curricula for undergraduate as well as postgraduate students [56]. Hence, several papers about RE education have been published in the Software Engineering (SE) and RE literature. Macaulay and Mylopoulos [34] looked into the issues in RE education by two investigations. Macaulay and Mylopoulos [34] attempted to formulate some general requirements for RE education in three dimensions. First, what the software industry feels that the universities should teach, second, needed skills to develop effective RE, and third, concepts and techniques of RE that student need to know. Naz Memon et al. [33] reports on an initial and exploratory study on finding out problems in RE education. The reported problems in RE education include: lack of interest in RE course, lack of practice on RE activities, insufficient knowledge and practice even after completing the course, and difficulties in knowing how to apply the knowledge of RE in the real world [33]. Several papers in the literature, e.g., [12, 34, 46, 52, 54], describe RE courses in education. All in all, these papers report their experience in teaching different setups of RE courses. The RE course described in [52], is to a large extent based on discussions between students, teachers, and students and teachers.

Berenbach [5] reports about the deficiencies in RE curriculum. Berenbach [5] emphasizes the importance of RE education in SE and Computer Science. Davis et al. [17] proposed a framework for integrating RE research in RE education to improve the educational performance. An experimental approach for teaching RE using a business game board for demonstrating social problems, complexity, and richness has been proposed in [44, 45], while Smith and Gotel [51] designed a board game to introduce good RE practices.

Huijs et al. [26] reported on their experience of integrating RE in several courses where the students were challenged by including real cases from practice in which they had to apply RE theories. Damian et al. [16] reported on their experiences of teaching RE in a global customer–developer relationship situation. The course was taught in collaboration with three universities in different locations, time zones, and culture. The students in the course played the roles of client and developer and experienced the development of a requirements specification in global projects. Auriol et al. [2] presented their experience from two lectures. In the first lecture, the students managed requirements and their effect on product building, while in the second lecture, the students played the role of software architect to design the product.

There are a few reported experiences in using role playing in RE education. Al-Ani and Yusop [1] used role playing within a group environment with the aim of creating an environment in which the students played a more active role in the learning process. Zowghi [58] shared the experiences of teaching an online postgraduate RE course. Role playing was used as a pedagogical tool to give students a greater appreciation of issues and problems associated with RE. In addition, Zowghi [58] reports some of the challenges encountered and suggestions of how to address the problems of distance between students and the instructor.

2.2 Role playing in education

When looking into the education literature on role playing, the term has not been used in the same way in the past. In the earlier literature, terms like ‘game’ and ‘gaming’ are used when referring to what is role playing. Krain and Shadle [28] specifically distinguish between simulations, games, and role playing. In role playing, the students are giving fewer roles in terms of preferences and objectives. Students must work to develop their character and think about how he/she would respond to a set of circumstances. Interactions within role playing are more interpersonal than they are goal-oriented. Role play allows students to “inhabit the issue and think beyond their own perspectives” [47], and typically involve less complex interactions [55].

Role playing as a pedagogical tool has been used in many disciplines for a number of years. Examples include nursing [14], medicine [4], clinical psychology [40], engineering [13], management [25], journalism [11], commercial services [21], and consultancy settings [7]. Moreover, role playing is frequently used for training [49]. Doorn and Kroesen [19] divide the purpose of using role playing into three classes of objectives. First, teaching communication/social skills where students need skills to discuss issues or explore people’s hidden motives. Second, teaching ethics, when the primary focus shifts from process to content. Third, broadening students’ perspectives, i.e., the role play is aimed at making students aware of the different perspectives to a particular problem.

Role playing as a pedagogical tool offers several advantages for both teachers and students. First, student interest in the topic increases. Research has shown that “integrating experiential learning activities in the classroom increases interest in the subject mater and understanding of course content” [41]. Second, there is an increased involvement on the part of the students in a role playing. That is, students are not passive recipients of the teacher’s knowledge. Rather, students take an active part. Poorman observed that “true learning cannot take place when students are passive observers of the teaching process” [41]. A third advantage of using role playing is that it teaches empathy and understanding of different perspectives [41].

2.3 Benefits of role playing

From the very beginning of the use of role playing in education, researchers have tried to identify and characterize the benefits of role playing for student learning. Researchers have noted the benefits of role playing in achieving specific learning objectives that may not have been met through a more traditional lecture format, “simulations have the power to recreate complex, dynamic political processes in the classroom, allowing students to examine the motivations, behavioral constraints, resources and interactions among institutional actors” [50]. In addition, researchers have noted several of other benefits of using role playing; however, many of these benefits are true for active learning in general and not specifically for role playing.

Already in 1959, Bloomfield and Padelford [8] stated that role playing may “produce tangible results over and above what [could] be taught and learned about politics by more usual methods of instruction”. Since then, researchers have generated an extensive list of benefits of role playing in education. In 1973, Greenblat [23] divided all of these claimed benefits in six categories, cognitive learning, affective learning, enhance student motivation and interest, longer-term learning benefits, gain increased self-awareness, and promote better student–teacher relations.

Cognitive learning includes claims that students may gain factual information, concrete examples of abstract concepts, analytical skills, procedural experience, and decision making skills. Furthermore, role playing enables students to apply the content of the course in real-life situations. A study by Boocock and Coleman [9] in 1966 reported that the students did gain an increased awareness.

Affective learning includes claims that students participating in role playing may lead to affective learning, including changed perspectives (e.g., attitudes toward certain issues), and greater insights into challenges faced by others (e.g., by acting as a customer, the students may understand the challenges faced by requirements engineers). Morgan [37] stated that “the most successful active learning may inspire students to take proactive measures in the real world to help bring into being the world as they would like to see it”.

Enhance student motivation and interest includes claims about how role playing may enhance student motivation and interest in the area of problem, the course, and learning in general. However, most of the claimed benefits in this category are anecdotal, e.g., that students do not want to take breaks in order to have more time for negotiation [10]. Several studies, e.g., [36, 48], report that students enjoy role playing and that teachers should use them in their education.

Longer-term learning benefits includes claims that role playing may have longer-term meaning benefits and that it stimulates more relevant inquiries about the real world problem area. In the literature, few studies have looked into this claim using empirical data. However, from learning theories in the fields of education and psychology, role playing can create “memorable events” and thereby creating more enduring and recalled memories [15, 35].

Gain increased self-awareness includes claims that the students in role playing may gain an increased self-awareness. When creating this six categories, Greenblat [23] found that there was little empirical evidence to support this claim.

Promote better student–teacher relations includes claims that role playing may help in promoting a better student–teacher relation. The reason for improving a better relationship lies in a more relaxed study environment for the students and that exchanging ideas comes more naturally. According to [23], this claim is not frequently repeated in the literature. However, Newmann and Twigg [38] reported that their use of role playing increased the student–teacher interactions.

Besides the claimed benefits in Greenblat’s six categories [23], there are many assumptions that role playing improves the students understanding of abstract concepts [50], and that role playing increases student motivation and effort despite the heavy workload that is involved [18]. However, there are few papers that have empirically evaluated the effectiveness of role playing. While many papers argues about the effectiveness of role playing in teaching critical thinking and problem-solving [20], Krain and Lantis [29] found no statistical significant difference in quiz scores by students who participated in role playing compare to the ones who were thought in more traditional classroom techniques. Powner and Allendoerfer [42] found that students who participated in role playing scored slightly better compared to the students who participated in classroom discussion; however, there was no statistical significant difference in the overall performance. Moreover, Raymond [43] investigated whether role playing had statistically significant effect on students’ exam scores. The results show that there were no statistically significant improvements. Thus, it appears that role playing did not help students to meet the learning objectives of the course.

2.4 Designing and using role playing

In the literature, besides papers about benefits of role playing, there are several papers (e.g., [24, 50]) providing instructions of how to design role playing exercises. These papers discuss design specifications, format, and implementation of role playing in education. In general, there are five steps to be taken when designing a role playing exercise.

First, it is important to select a topic and identify clear learning objectives. Second, determine for how long the role playing exercise should last, ranging from 50-min to the entire course, depending on the design of the exercise and the identified learning objectives. The third step is to design the exercises to include both an intragroup discussion as well as an intergroup discussion. The idea is that this part of the exercise teaches the students about cooperation and collaboration. Fourth, prepare the background information about the exercise. For students to fully understand the context of the role playing exercise, it is important that the background information is available to the students before the role playing exercise starts. Finally, the fifth step is to establish a timeline and/or specific phases for the role playing exercise.

In addition to the above five general steps of designing a role playing exercise, the literature include other design considerations. For example, using a fictional or real-life scenario, how much students need to prepare, and how the assessment of the role playing exercise should be conducted. Moreover, some papers describe the teachers role in a role playing exercise. According to Lederman [31], the teacher should guide and encourage discovery. The teachers are not responsible for providing learning; instead, they should provoke it [31]. Lederman [31] suggests that teachers should help the students rather than learning them. Thatcher [53] agrees with Lederman [31], stating that the teacher should be a facilitator and organizer.

3 Research methodology

This study investigates, using a case study research mode and a specific RE course design, if students with a higher grade in a role playing project perform better on an individual written exam compared to students with a lower role playing project grade. Based on potential performance differences, we seek to draw conclusion on the potential impact of role playing projects on student learning.

The case study comprises nine course instances at two universities. In each semester, the role playing occurred between the first and last week of the course running over 8 weeks (for details, see RE Course Description in Sect. 4 and Role Playing Project Setting in Sect. 5).

In this study we use statistical analysis to test if there is a correlation between the grade in the project and the score on the written exam. For all statistical tests, regression analysis at the 0.05 level was used.

3.1 Preliminary assumptions and main hypothesis

We had three preliminary assumptions about the role playing project’s impact on the learning outcomes:

  1. 1.

    If students get a higher grade in the role playing project, then they have a deeper knowledge about requirements engineering.

  2. 2.

    If a student scored higher in the essay part of the written exam, then the student has a deeper knowledge about requirements engineering.

  3. 3.

    High performing students that did not performed well in the role playing project (i.e., got a lower grade in the role playing project), perform as well as students with a higher grade in the role playing project on the theory part in the written exam, but not as well on the essay part in the written exam.

Based on the above assumptions and that role playing may give the benefit of longer-term learning [23], our main hypothesis is that: role playing projects promotes students’ deep learning about requirements engineering.

3.2 Data collection

Data were collected from a total of 412 students that participated in the Requirements Engineering course between 2007 and 2014 at Lund University and Chalmers | University of Gothenburg. Table 1 shows the distribution of students by year and university.

All 412 students between 2007 and 2014 received the same setup of the course. That is, the lectures were built up in the same way, the project setup has been the same, and the written exam has had the same format. At Lund University, the course responsible (the second author) has been the same for all years, while at Chalmers | University of Gothenburg (2013 and 2014) it was a different course responsible (the first author). However, the course responsible at Chalmers | University of Gothenburg had been involved in the RE course at Lund University between 2006 and 2012. Besides the difference of course responsible between the two universities, the only difference between the students has been their involvement, engagement, and their grade in the role playing project.

Table 1 Distribution of students
Full size table

The individual written exam consists of two parts, one theory part and one essay part. Each part is worth 50 % of the total points in the written exam. The theory part consists of multiple-choice questions where the majority of the questions are based on a proposition and a reason, which is illustrated in Fig. 1. The essay part consists of several short essay where a topic and keywords are given to the students. The topics are related to tasks that have been practiced in the role playing project, e.g., prioritisation, elicitation, release planning, or quality requirements.

Fig. 1
figure 1

Example of multiple-choice questions

Full size image

3.3 Validity threats

One threat of this study is that we have only compared project groups using role playing and their score in the written exam. That is, we did not have a control group of students not being exposed to role playing to fully understand the impact that role playing has on students learning. The reason for this is that when we designed the RE course with project groups using role playing, the objective was to setup a good course where the students could learn RE.

The external validity is concerned with the ability to generalize the results, i.e., in this case the applicability of the findings beyond the RE course. One threat to external validity in this study is that only one course has been studied. Thus, the context, the RE course and the role playing project setup have been described in detail, which can help in understanding if this situation is similar to other cases and situations. Hence, it may help to judge the transferability of the setup and the results to other situations. Although the course was designed and executed in the same way at both Lund University and Chalmers | University of Gothenburg, the results from the two universities are similar, i.e., higher project grades lead to higher score in the written exam, which supports the generalization beyond the course at Lund University.

4 RE course description

At Lund University, Department of Computer Science, the Requirements Engineering courseFootnote 1 is an elective course for fourth-year students (i.e., master-level) majoring in Computer Science, Electrical Engineering, Engineering Physics, Industrial Engineering and Management, and Information and Communication Engineering Technologies. At Chalmers | University of Gothenburg, Department of Computer Science and Engineering, the Requirements Engineering courseFootnote 2 is a mandatory course for fourth-year students (i.e., master-level) majoring in Software Engineering and Interaction Design and Technologies. Moreover, it is an elective course for fourth-year students at any other program.

The teaching uses both a traditional approach of lectures and exercises as well as a group project using a role playing method [57], which runs over 8 weeks (the project setting is described in detail in Sect. 5). An overview of the content of the RE course is illustrated in Table 2. Between the years of 2007 and 2014, 289 students from Lund University and 123 students from Chalmers | University of Gothenburg have participated in this course.

Table 2 RE course content
Full size table

The lectures provide an overview of the literature, i.e., a high-level structure of RE theory and thereby aid self-studies of the literature. The main objective of the exercises is to support the project by connecting theory to practice and to provide an opportunity to discuss RE techniques in detail. The project involves a number of deliverables and a final project conference where learning outcome of each project is presented. In addition to lectures, exercises, and the project, the RE course includes two laboratory sessions. The first laboratory session is about requirements prioritization where the students get acquainted with a prioritization tool, and takes place in the third week of the course. The second laboratory session is about Release Planning (RP) and takes place in the fifth week of the course. In the RP laboratory session, the students should get familiarized with the task of deciding which subsets of the requirements accumulated in the requirements backlog shall be assigned to upcoming releases using a release planning tool.

The final grade for the RE course is based on both the written exam (60 %) and the group project (40 %). To include the group project as an assessment criteria was based on feedback from previous students and is supported in the literature, e.g., Naz Memon et al. [33] discovered that as many as 72 % of the students suggested that their group project should be the assessment criteria for their RE course. The role playing project is described in more detail in the following section.

5 Role playing project setting

The main goals of the project work are to connect theory to practice, to give the students a concrete experience of practical requirements engineering techniques (related to cognitive learning in [23]), to promote student motivation (related to enhance student motivation and interest in [23]) through real stakeholders, and to provide the students group-learning with a focus on realistic problems (related to cognitive learning in [23]).

Each project were instructed to act two roles simultaneously (related to affective learning in [23]). The first role was that of acting as requirements engineers. This role’s main task was to do requirements engineering, including elicit requirements, specify and document requirements, validate requirements, prioritise requirements, and manage the requirements, according to a Project Mission (PM) and deliver a Requirements Document (RD). The second role was to act as customer, which includes provide knowledge about the project and information about the domain, for another requirements engineering project by delivering a PM. The two different roles and the setup of the project are illustrated in Fig. 2. Each student is expected to spend 80 h for the entire project where 80 % of his or her effort should be devoted to the first task.

Fig. 2
figure 2

Project overview

Full size image

In Fig. 2, ‘Your Project’ is a project group that consists of 4–5 students. In each project, three different roles should be appointed. All students are project members, but in addition to project members, two roles should be appointed by the project to two different project members. One student should be the project manager with the responsibility for distributing the RE work. Another student should be appointed the customer coordinator with the responsibility for distributing the work of acting as customer for another project.

In the second, fourth, and sixth week of the course, meetings between the project supervisor (a teacher in the course) and the student group are scheduled (to promote intragroup discussions). The meetings include, but are not limited to, checking the progress and status according to plan, plans for coming weeks, and discussions about challenges, difficulties, and open issues across projects.

5.1 Project process overview

The project has five phases (see Fig. 3). Phase one, Definition, includes writing, establish, and document the PM. The second phase, Planning, is about developing a Project Plan (PP) for the group work. In the First Iteration (third phase) is requirements engineering work conducted with focus on elicitation, prioritization, and specification, while in the Second Iteration (phase four) is the requirements engineering work focused on elicitation, specification, and validation. Finally, in the last phase, Finalization, the requirements engineering work is focused on specification and validation. In addition, the results and lessons learnt are packaged for presentation and course assessment. In the flowing subsections, each phase is described in more details.

Fig. 3
figure 3

Project process overview

Full size image

5.1.1 Definition phase

In the definition phase, each project group shall prepare a Project Mission (PM in Fig. 2) for another project where they act as customers. The Project Mission (part of a PM is shown in Fig. 4) defines what type of system for which the other group (that act as requirements engineers) shall elicit, prioritize, specify and validate requirements. The project group creates a project of their choice (it is important that the group have a deep understanding of the application domain, a genuine interest in the system, and that they are able to assess the value of the detailed requirements) where they take one of two roles as customers. The first role is as a potential customer, i.e., the students act as a part of the market and may be a potential buyer of the product. In this scenario, the other project owns the product and decides the products’ contents. The second role is as product owner, i.e., the students’ act to develop and sell product to an open market. In the second alternative, the other project is subcontracted to only do requirements engineering. In addition to the two roles, each project group’s PM should define what type of system for which the other project group should elicit, prioritize, specify, and validate requirements for.

Fig. 4
figure 4

An example of (part of) a PM

Full size image

At the end of the first week, each project group submits their PM to the teachers who check the feasibility of the PM. Once the teachers have approved all PMs, they are published on the course homepage for the students to read and prioritize which project they would like to do RE for. The actual decision of which project group gets which PM is decided by a lottery on the following lecture. Figure 5 shows an example of the project setup in the course, i.e., the outcome of the lottery.

Fig. 5
figure 5

An example of project setup

Full size image

5.1.2 Planning phase

In this phase, each project group should plan their work throughout the project by creating a Project Plan (PP in Fig. 3). The PP should contain information such as description of the project, planned activities (with start and stop dates), description of deliverables, cost estimations (in terms of hours) for each activity, and how the results should be packed. Although the PP is only delivered once (at the end of the planning phase) to the teachers, the PP should be a “living” document throughout the entire project and should be updated as the students gain more knowledge. The estimated time for acting as customers should also be included in the PP.

5.1.3 First iteration phase

The requirements engineering work starts in the first iteration. In this phase, the RE work is focused on elicitation, specification, and prioritization of requirements. In the first iteration, the students are encouraged to try out as many elicitation and specification techniques as possible. At the end of the first iteration, each project group should deliver a first version of their Requirements Document (RDv1 in Fig. 3) to the project supervisor. We do not give the students any specific template for the RD. Instead, it is up to the students to decide whether they would like to use any available template from the literature, any industry specific templates, or to create their own template for their RD. The RDv1 should contain, besides business goals and project type, results from elicitation where at least three elicitation techniques [(e.g., interviewing the customer group, some kind of stakeholder analysis (an example from one project group is shown in Fig. 6), brainstorming] have been used, results from requirements specification (i.e., the written requirements) where at least four different specification techniques (e.g., context diagrams, features, virtual windows, task descriptions) have been used, and include requirements and supporting information on goal level, domain level, and product level. In addition, RDv1 must contain at least 20 specified requirements, which will be used in the RP laboratory session. Finally, each requirement must have a unique identity (name or number).

Fig. 6
figure 6

An example of a stakeholder analysis

Full size image

In general, the RDv1 contains 20–40 requirements at all different levels, ranging from very high-level abstract requirements to vey detailed low-level requirements. Since we encourage the students to try several different specification techniques (if they use them incorrectly it will not have any impact on their grades at this stage, it is more focus on try and learn by getting feedback from the project supervisors), most of the delivered RDv1 contains requirements using more than five different specification techniques where all of them includes some kind of context diagram (an example is shown in Fig. 7) and requirements written as features.

Fig. 7
figure 7

An example of a context diagram

Full size image

5.1.4 Second iteration phase

In the second iteration phase, the RE work continues with focus on elicitation, specification, and validation. The project groups should continue to elicit and specify more requirements (and improve the requirements in RDv1 based on feedback from the project supervisor) by using different RE elicitation and specification techniques. In addition to elicitation and specification, each project group should validate their requirements using a checklist-based technique and develop their own checklist for their customers. At the end of the second iteration, each group project delivers a second version of their RD (RDv2 in Fig. 3).

The RDv2 should contain, in addition to the criteria for RDv1, improved and updated requirements from RDv1, and results from the requirements validation. In addition to the Requirements Document (RD), each project group should deliver a Review Report (RR in Fig. 2). Each project group acts as a customer and therefore should review the RDv2 that the other project group has delivered using the provided checklist in RDv2. The goal of the RR is for the acting customers to validate the RDv2 for the PM that they created in the definition phase to suggest improvements to the RD. An example of a RR from one project group is shown in Fig. 8. The only instructions that are given, and are mandatory, for all groups are: (1) the project group should produce relevant and useful changes for improvement of RD2, and (2) each change must be ranked for criticality.

Fig. 8
figure 8

An example of a review report

Full size image

In general, the RD improves greatly from version 1 to version 2, in terms of number of specified requirements (ranging from 30 to 70), but also the quality of the specified requirements and the use of techniques for specifying the requirements. An example of a requirement from the second iteration is shown in Fig. 9. In this phase, the students should validate the RDv2 for their project, i.e., the RDv2 that the requirements engineer group have created. The created checklists for requirements validation varies greatly. The reason is that, similar to the template for the RD, we do not state which checklist they should use. Instead, we show a few different checklists (from the course literature [30]) and then instruct the students that they can use any of them, a combination of the different checklists, or create their own based on the checklists and knowledge they have gained so far in the course.

Fig. 9
figure 9

An example of a requirement in the second iteration

Full size image

5.1.5 Finalization phase

The last phase, finalization, starts with a Conference Presentation (CP in Fig. 3). Each project group should prepare a short (approximately 10 min) presentation about the group project. The presentation should include: a description of the project (based on the PM), an overview of project results including techniques used, and important experiences and lessons learned. The remaining time in the finalization phase is devoted to RE work with focus on specification and validation based on feedback from both the project supervisor and the acting customers, and to package their results and lessons learnt into an Experience Report (ER in Fig. 3). The experience report should include: a description of the requirements engineering work, including experiences and reflections in relation to learning objectives; a description of the chosen methods/techniques for elicitation, specification, validation, and prioritization, and a motivation for why they chose the used methods/techniques; and a reflection of the usage of these methods/techniques in terms of what was successful, what was challenging, and why it was successful and/or challenging. At the end of the last phase, the project groups should submit the ER and their final version of the RD (RDv3 in Fig. 2) to their project supervisors.

The RDv3 should contain the same information as the previous two versions of their RD, but it should be improved and more complete than the previous versions. The ER should contain lessons learnt, what went according to plan and not, which elicitation, specification, prioritization, and validation techniques were used and why these techniques were chosen.

5.2 Assessment

As mentioned in Sect. 5, the project in the RE course consists of 40 % of the finale grade for the course. The general prerequisites are that all deliverables and versions must be in time, no free-riding on the rest of the group and that RDv1 must contain at least 20 requirements. The project deliverables that are part of the project grade is shown in Table 3.

Table 3 Project grading criteria
Full size table

Note that the PM is not part of the grade, nor is RDv1. The reason for not including RDv1 as part of the project grade is because we want the students to feel free to try out as many elicitation and specification techniques as possible without feeling the pressure that everything must be “perfect” already in the first version.

6 Results on the impact of role playing

The results show that students with a higher grade in the role playing project scored, in total, higher in the written exam compared to students with a lower grade in the project, a difference that is statistically significant using regression analysis (\(p<0.05\)). Since all students received the same treatment in terms of lectures, exercises, projects, and had the same course responsible, the role playing may be responsible for the higher exam scores earned by the students with a higher project grade (see further discussions in Sect. 7.3).

Looking into the different parts of the exam, i.e., the theory and essay parts, there was no statistically significant difference in the score of the theory part between students with a higher and lower project grade. However, students with higher project grades scored higher in the essay part compared to the students with a lower project grade, a difference which is statistically significant using the same regression analysis (\(p<0.05\)). This result is not surprising since the topics in the essay part are heavily correlated to tasks that have been performed during the project.

The results presented in this study is, to the best of the author’s knowledge, one of few papers that statistically assess the impact that role playing has on students performance in written exams. According to, e.g., Krain and Lantis [29] and Powner and Allendoerfer [42], the impact on students learning using role playing has been mostly anecdotal where research has found little or no statistically significant improvements in quantitative measures of students performance. In REE literature, many of the published papers (e.g., [16, 26, 52]) have reported on experiences of teaching Requirements Engineering courses in general, and a few papers (e.g., [1, 58]) have reported on experiences using role playing. However, none of these papers have looked into if role playing has any effect on students’ academic performance.

In the general Education literature, Powner and Allendoerfer [42] found that students who participated in role playing scored slightly better compared to the students who participated in classroom discussion; however, there was no statistical significant difference in the overall performance. Moreover, Raymond [43] investigated the impact of role playing simulation’s effect on exam scores. Raymond [43] compared two groups of students, one group had ‘traditional’ lectures, while the second group used simulations. The results show that the simulation was not associated with statistically significant improvements in exams scores. The difference between our result and the results in [43] is that we compared students that performed better (i.e., scored a higher grade) in the projects with students that scored a lower grade. That is, all students in our study used role playing as part of the project; however, the ones with a higher project garde scored better in the exam in total and in the essay part. Unlike Raymond [43], the course evaluations in our Requirements Engineering course have consistently have high scores over the years of teaching the course. Hence, we do not see any relation between lower teaching evaluations and exam scores.

7 Discussion

In this section, we discuss lessons learned during the years of teaching Requirements Engineering with a role playing group project. We also course evaluations and some important limitations of the results.

7.1 Lessons learned

After several of years of applying a role playing method for our group project in the RE course, it has become clear to us that this project setup can be effective in helping students to gain a richer and more realistic understanding of RE. Also, by using one project where the project group consists of the same students throughout the entire eight weeks minimizes the need to warm up students to the process before they can play the roles effectively [28].

For the requirements elicitation parts of the project work, a requirement to pass the project is to use at least three different elicitation techniques including stakeholder analysis and some type of interviewing technique. To motivate the students to try out more elicitation techniques, several more relevant techniques than minimally specified for passing the project is a requirement for the grade high distinction. This has motivated the students to use, in most of the projects throughout the years, more than three relevant elicitation techniques. Furthermore, although it is not a requirement to pass (or to receive the grade high distinction) the project, several projects have elicited requirements for the project using real stakeholders, customers, and user groups. That is, stakeholders that are not teachers or students in the course.

For example, one project conduct RE activities for an intelligent vending machine for soft drinks to be used at a school. The students in this project decided to interview four students (other than the students acting as customers in the RE course) to elicit requirements from “real” users, they interviewed two employees (teachers) at the school, and one employee from the cafe that was intended to maintain the vending machine.

Another example is a project for a digital wallet, not only did the students interview employees from the local public transportation sector to elicit requirements for using the digital wallet to pay for bus and train tickets, they interviewed personnel from two different banks to discover requirements about technical issues between the digital wallet and the bank systems and to find out if the banks would have been interested in a product like the digital wallet.

These two examples indicate that the students are motivated and interested in RE and that it is not just a good grade that motivates them. In addition, the projects that takes these extra steps in their project work do practice RE elicitation skills, which according to Berenbach [5] is an important skill to teach students in a RE course. Moreover, since we require the students to chose relevant RE techniques and motivate their choice in the ER, we believe, based on our experience from the course, that the students learn (to a certain degree) the ability to select and combine RE techniques that are suitable for a particular project based on the characteristics of the project, which is important to learn in REE [27]. A third lesson from these examples is that the students have the opportunity to communicate with real customers and users, which is an important part of RE [57].

Two important activities in RE are negotiation and prioritization of requirements. As reported in [57], these two activities are difficult to handle during exercises. Besides our first laboratory session where students are exposed to prioritizing requirements using a prioritization tool (described in Sect. 3), prioritization and negotiation are included in the project work. Each project should negotiate with their customers (the other project) and come up with a prioritized list of the most important requirements. Our experiences show that this part works very well and that most of the project groups use the prioritization tool from the first laboratory session in the project work. Moreover, over the years we have had a handful of project groups that asked real users/customers (i.e., not the student group acting as customer) to prioritize which requirements (based on a subset of all their requirements) they would like to have in a product similar to the one that the students are performing RE activities for. We believe this would enhance, to some degree, students’ ability to negotiate and prioritize requirements.

7.2 Course evaluations

This section is based on data collected from the official course evaluations (with a total response rate of 45 %) between the years of 2007 and 2014. Based on the Course Experience Questionnaires (CEQ) of student satisfaction with the RE course, our results do not indicate that the students find the RE course boring, which is reported in [32]. Looking into the open-ended questions in CEQ, the results indicate that our students enjoy the course and think it is an important subject to learn for their future career in industry. One student explained, “the group project was very educational where we had the opportunity to practice the theoretical information from the lecture”. Other students pointed out the importance of learning RE, which they believe is an important subject, but is lacking in other SE courses. Another important aspect of why the students were satisfied with the course was the mix of theory and practice. The lectures provide the students with the needed theory, while the exercise and the project that runs throughout the course focus on the practical side. For example, one student stated that the course has a “great mix of practice and theory”, while another student said that “the course is very good due to great teachers and the focus on the project for practicing the techniques”. This is not online with reported REE problems in the literature, e.g., Naz Memon et al. [33] reported that REE has a too strong emphasis on theory rather than the practical side.

In the REE literature, several papers, e.g., [17, 52], have reported that there is a lack of practice in RE activities in the courses. This is not in line with the results from our course experience questionnaire evaluations. Most of the students feel that there is enough time to practice RE activities in the project work, one student explained, we hade the opportunity to “directly transfer theory to practice in the projects”. Although practicing in the project work in the RE course is not the same as a real-life project in industry, we believe it provides the students with a close enough realistic project experience. Several students support this view, where one explained the realisms of the project as “the project in this course is the closest to a real-life project I have come across at the university”. Although most of the students see the projects as realistic ones, a few students would like to see more realistic projects. This was further explained by one student, would like to see “more realistic projects, i.e., the PM created by the other students should be more realistic. Hence, the teachers should check the feasibility and realisms of the PMs”. Although most of the students relate the project work to the importance, one student gave another explanation of why the course is important, it “feels like we master most of the areas in RE after taking this course”.

7.3 Limitations

The difference in project grades and scores in the written exam may be explained by that some students were mainly interested in passing the course, and not aiming for a higher grade. This may particular be a threat to the students from Chalmers | University of Gothenburg since the RE course was mandatory; hence, the students were “forced” to study the course. However, for the majority (over 70 %) of the students in this study, the RE course was optional, i.e., the students actively decided to study RE. Hence, this threat has a limited effect on the results. The difference in scores in the written exam may also be explained by that some students, regardless of project grades, are better in studying and would score higher in the written exam even without the role playing project. This may be a particular threat to the students from Lund University because the students are allowed to choose their own project members; hence, good students may decide to work together and thereby it may be the students ability rather than the role playing that influences the project grades. However, despite that the students from Chalmers | University of Gothenburg are randomly assigned to project groups, i.e., the students have no choice in deciding their project members, the result, when analyzing the project grades and exam scores separately between Lund University and Chalmers | University of Gothenburg students, follows the same pattern. That is, a higher project grade has a statistically significant effect on the students’ exam scores. Hence, this threat has a limited effect on the results.

8 Conclusion

In this paper, we discussed the use of role playing in requirements engineering courses and its impact on students academic performance in terms of scores on the written exam. We illustrated how role playing is used in the requirements engineering course to broaden students’ perspectives and learning. Data were collected (project grades and scores from the written exam) from 412 students from two universities between 2007 and 2014.

8.1 Summary of results and limitations

This case study investigates if there is a correlation between a role playing project grade and the grade of a written exam. The results show that students who performed better (i.e., scored a higher grade) in the role playing project scored higher (in total) in the written exam compared to students who had lower project grades, a difference that was statistically significant. Moreover, the results showed that there was no statistically significant difference in the scores in the theory part (multiple-choice questions) in the exam, but there was a statistically significant difference when it comes to the scores in the essay part of the exam. Although there was a statistically significant correlation between a higher project grade and higher score on the essay part of the written exam, the causal relationship is not clear. Ideally, a control group should have been used to be able to clarify the causal relationship; however, we decided not to use a control group due to ethical reasons. That is, we did not want to withhold the students with what we believe is a good teaching approach with the role playing project. However, the course evaluations, where the students have expressed the importance of the project, support our assumptions of the importance the role playing project has in increasing learning outcome.

8.2 Outline of further work

The following areas are interesting for further work:

  • The use of a control group to clarify the causal relationship between the role playing project and the score on the written exam. This could be done when introducing the role playing project to new courses. Instead of having a big bang introduction of role playing, the role playing project could be gradually introduced in the course. Thus, a control group could be used to evaluate its impact on students learning.

  • Additional data, in particular from Chalmers | University of Gothenburg, but also from other universities, is required to achieve more generalizable results.

  • In this study we have not evaluated whether role playing produce more substantial benefit, in terms of learning requirements engineering skills, in the long-term than in the short-term, which provide an interesting direction for future work.

In addition, it would be interesting if further work includes studies of the impact of role playing techniques in software engineering courses in general not only in courses restricted to the requirements engineering part of software engineering.