Abstract
Requirements Engineering has attracted a great deal of attention from researchers and practitioners in recent years. This increasing interest requires academia to provide students with a solid foundation in the subject matter. In Requirements Engineering Education (REE), it is important to cover three fundamental topics: traditional analysis and modeling skills, interviewing skills for requirements elicitation, and writing skills for specifying requirements. REE papers report about using role playing as a pedagogical tool; however, there is a surprising lack of empirical evidence on its utility. In this paper we investigate whether a higher grade in a role playing project have an effect on students’ score in an individual written exam in a Requirements Engineering course. Data are collected from 412 students between the years of 2007 and 2014 at Lund University and Chalmers | University of Gothenburg. The results show that students who received a higher grade in the role playing project scored statistically significant higher in the written exam compared to the students with a lower role playing project grade.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
1 Introduction
Requirements Engineering (RE) is a process within Software Engineering that involves activities that are required to gather, create, and maintain a software product’s requirements specification. There has been an increasing interest in RE which has challenged academics to create and deliver high-quality RE courses at universities to provide students with a solid foundation of the subject matter. Requirements engineering courses have been widely introduced in many software engineering curricula as compulsory courses where the fundamental knowledge about activities, concepts, methods, and techniques are taught [32].
Although the importance of RE in software development is well recognized [3], students are not active in learning RE courses and find the courses to be boring without the opportunity to practice the RE methods and techniques that are taught in the classroom [32]. Moreover, several studies report on problems that students face in Requirements Engineering Education (REE), e.g., a lack of requirements elicitation skills in REE [5], lack of practice on RE activities [17, 52], being able to select and combine RE techniques that are suitable for a particular project [27], lack of interest in RE course [33], and a too strong emphasis on theory rather than on practice [33].
One of the main RE-related problems faced by teams is communication between developers and customers [57]. Requirements engineers must be able to communicate with the customers about the problem being solved in a language they understand [57]. Another issue faced in practice is to choose the most effective analysis and modeling techniques that are suited for a particular project [27, 57]. Therefore, to be able to make such a decision, students need to be familiar with a wide range of RE techniques [57].
To successfully teach students these necessary skills, it is not enough to teach RE courses in a traditional lecture/laboratory setting, it would be unrealistic to expect students to use RE techniques in an effective manner in the industry without any practical experience. In addition, according to [6], most students only learn about 50 % of what they see and hear, while they learn about 80 % of what they use and practice in a more realistic setting. According to Polajnar et al. [39], a more realistic Software Engineering (SE) experience is necessary for properly understanding why the discipline exists. However, to create such a realistic setting is difficult as it needs stakeholders who are either real or behave in a realistic way [22]. In REE literature, two possibilities of creating a realistic setting are presented: (1) have access real stakeholders with real needs, or (2) have stakeholders that are simulated by other persons [39].
In this paper we describe the RE course we taught with a focus on one specific technique, namely the group project using a role playing method [57]. The group project addresses the needs for practicing elicitation skills and the ability to choose the right RE techniques for a particular project. We report our experiences and the results from the course evaluations from Lund University, Sweden and Chalmers | University of Gothenburg, Sweden, between the years of 2007 and 2014 where in total 412 students participated in the RE course. We look into if a higher grade in the role playing project has an effect on the students’ exam scores.
The remainder of this paper is organized as follows. Section 2 offers an overview of RE education and the use, the benefits, and the design of role playing in education. Section 3 presents the research methodology and discuss the validity threats of the design of the study. Section 4 presents the RE course, while Sect. 5 describes the project. The results are presented and discussed in Sects. 6 and 7 discuss lessons learned. Section 8 gives a summary of the main conclusions.
2 Literature review
This section begins with an overview of RE education followed by a brief coverage of role playing in education, its benefits, and how to design and use role playing.
2.1 RE education
In the past few years, there has been an increasing emphasis on incorporating RE into university curricula for undergraduate as well as postgraduate students [56]. Hence, several papers about RE education have been published in the Software Engineering (SE) and RE literature. Macaulay and Mylopoulos [34] looked into the issues in RE education by two investigations. Macaulay and Mylopoulos [34] attempted to formulate some general requirements for RE education in three dimensions. First, what the software industry feels that the universities should teach, second, needed skills to develop effective RE, and third, concepts and techniques of RE that student need to know. Naz Memon et al. [33] reports on an initial and exploratory study on finding out problems in RE education. The reported problems in RE education include: lack of interest in RE course, lack of practice on RE activities, insufficient knowledge and practice even after completing the course, and difficulties in knowing how to apply the knowledge of RE in the real world [33]. Several papers in the literature, e.g., [12, 34, 46, 52, 54], describe RE courses in education. All in all, these papers report their experience in teaching different setups of RE courses. The RE course described in [52], is to a large extent based on discussions between students, teachers, and students and teachers.
Berenbach [5] reports about the deficiencies in RE curriculum. Berenbach [5] emphasizes the importance of RE education in SE and Computer Science. Davis et al. [17] proposed a framework for integrating RE research in RE education to improve the educational performance. An experimental approach for teaching RE using a business game board for demonstrating social problems, complexity, and richness has been proposed in [44, 45], while Smith and Gotel [51] designed a board game to introduce good RE practices.
Huijs et al. [26] reported on their experience of integrating RE in several courses where the students were challenged by including real cases from practice in which they had to apply RE theories. Damian et al. [16] reported on their experiences of teaching RE in a global customer–developer relationship situation. The course was taught in collaboration with three universities in different locations, time zones, and culture. The students in the course played the roles of client and developer and experienced the development of a requirements specification in global projects. Auriol et al. [2] presented their experience from two lectures. In the first lecture, the students managed requirements and their effect on product building, while in the second lecture, the students played the role of software architect to design the product.
There are a few reported experiences in using role playing in RE education. Al-Ani and Yusop [1] used role playing within a group environment with the aim of creating an environment in which the students played a more active role in the learning process. Zowghi [58] shared the experiences of teaching an online postgraduate RE course. Role playing was used as a pedagogical tool to give students a greater appreciation of issues and problems associated with RE. In addition, Zowghi [58] reports some of the challenges encountered and suggestions of how to address the problems of distance between students and the instructor.
2.2 Role playing in education
When looking into the education literature on role playing, the term has not been used in the same way in the past. In the earlier literature, terms like ‘game’ and ‘gaming’ are used when referring to what is role playing. Krain and Shadle [28] specifically distinguish between simulations, games, and role playing. In role playing, the students are giving fewer roles in terms of preferences and objectives. Students must work to develop their character and think about how he/she would respond to a set of circumstances. Interactions within role playing are more interpersonal than they are goal-oriented. Role play allows students to “inhabit the issue and think beyond their own perspectives” [47], and typically involve less complex interactions [55].
Role playing as a pedagogical tool has been used in many disciplines for a number of years. Examples include nursing [14], medicine [4], clinical psychology [40], engineering [13], management [25], journalism [11], commercial services [21], and consultancy settings [7]. Moreover, role playing is frequently used for training [49]. Doorn and Kroesen [19] divide the purpose of using role playing into three classes of objectives. First, teaching communication/social skills where students need skills to discuss issues or explore people’s hidden motives. Second, teaching ethics, when the primary focus shifts from process to content. Third, broadening students’ perspectives, i.e., the role play is aimed at making students aware of the different perspectives to a particular problem.
Role playing as a pedagogical tool offers several advantages for both teachers and students. First, student interest in the topic increases. Research has shown that “integrating experiential learning activities in the classroom increases interest in the subject mater and understanding of course content” [41]. Second, there is an increased involvement on the part of the students in a role playing. That is, students are not passive recipients of the teacher’s knowledge. Rather, students take an active part. Poorman observed that “true learning cannot take place when students are passive observers of the teaching process” [41]. A third advantage of using role playing is that it teaches empathy and understanding of different perspectives [41].
2.3 Benefits of role playing
From the very beginning of the use of role playing in education, researchers have tried to identify and characterize the benefits of role playing for student learning. Researchers have noted the benefits of role playing in achieving specific learning objectives that may not have been met through a more traditional lecture format, “simulations have the power to recreate complex, dynamic political processes in the classroom, allowing students to examine the motivations, behavioral constraints, resources and interactions among institutional actors” [50]. In addition, researchers have noted several of other benefits of using role playing; however, many of these benefits are true for active learning in general and not specifically for role playing.
Already in 1959, Bloomfield and Padelford [8] stated that role playing may “produce tangible results over and above what [could] be taught and learned about politics by more usual methods of instruction”. Since then, researchers have generated an extensive list of benefits of role playing in education. In 1973, Greenblat [23] divided all of these claimed benefits in six categories, cognitive learning, affective learning, enhance student motivation and interest, longer-term learning benefits, gain increased self-awareness, and promote better student–teacher relations.
Cognitive learning includes claims that students may gain factual information, concrete examples of abstract concepts, analytical skills, procedural experience, and decision making skills. Furthermore, role playing enables students to apply the content of the course in real-life situations. A study by Boocock and Coleman [9] in 1966 reported that the students did gain an increased awareness.
Affective learning includes claims that students participating in role playing may lead to affective learning, including changed perspectives (e.g., attitudes toward certain issues), and greater insights into challenges faced by others (e.g., by acting as a customer, the students may understand the challenges faced by requirements engineers). Morgan [37] stated that “the most successful active learning may inspire students to take proactive measures in the real world to help bring into being the world as they would like to see it”.
Enhance student motivation and interest includes claims about how role playing may enhance student motivation and interest in the area of problem, the course, and learning in general. However, most of the claimed benefits in this category are anecdotal, e.g., that students do not want to take breaks in order to have more time for negotiation [10]. Several studies, e.g., [36, 48], report that students enjoy role playing and that teachers should use them in their education.
Longer-term learning benefits includes claims that role playing may have longer-term meaning benefits and that it stimulates more relevant inquiries about the real world problem area. In the literature, few studies have looked into this claim using empirical data. However, from learning theories in the fields of education and psychology, role playing can create “memorable events” and thereby creating more enduring and recalled memories [15, 35].
Gain increased self-awareness includes claims that the students in role playing may gain an increased self-awareness. When creating this six categories, Greenblat [23] found that there was little empirical evidence to support this claim.
Promote better student–teacher relations includes claims that role playing may help in promoting a better student–teacher relation. The reason for improving a better relationship lies in a more relaxed study environment for the students and that exchanging ideas comes more naturally. According to [23], this claim is not frequently repeated in the literature. However, Newmann and Twigg [38] reported that their use of role playing increased the student–teacher interactions.
Besides the claimed benefits in Greenblat’s six categories [23], there are many assumptions that role playing improves the students understanding of abstract concepts [50], and that role playing increases student motivation and effort despite the heavy workload that is involved [18]. However, there are few papers that have empirically evaluated the effectiveness of role playing. While many papers argues about the effectiveness of role playing in teaching critical thinking and problem-solving [20], Krain and Lantis [29] found no statistical significant difference in quiz scores by students who participated in role playing compare to the ones who were thought in more traditional classroom techniques. Powner and Allendoerfer [42] found that students who participated in role playing scored slightly better compared to the students who participated in classroom discussion; however, there was no statistical significant difference in the overall performance. Moreover, Raymond [43] investigated whether role playing had statistically significant effect on students’ exam scores. The results show that there were no statistically significant improvements. Thus, it appears that role playing did not help students to meet the learning objectives of the course.
2.4 Designing and using role playing
In the literature, besides papers about benefits of role playing, there are several papers (e.g., [24, 50]) providing instructions of how to design role playing exercises. These papers discuss design specifications, format, and implementation of role playing in education. In general, there are five steps to be taken when designing a role playing exercise.
First, it is important to select a topic and identify clear learning objectives. Second, determine for how long the role playing exercise should last, ranging from 50-min to the entire course, depending on the design of the exercise and the identified learning objectives. The third step is to design the exercises to include both an intragroup discussion as well as an intergroup discussion. The idea is that this part of the exercise teaches the students about cooperation and collaboration. Fourth, prepare the background information about the exercise. For students to fully understand the context of the role playing exercise, it is important that the background information is available to the students before the role playing exercise starts. Finally, the fifth step is to establish a timeline and/or specific phases for the role playing exercise.
In addition to the above five general steps of designing a role playing exercise, the literature include other design considerations. For example, using a fictional or real-life scenario, how much students need to prepare, and how the assessment of the role playing exercise should be conducted. Moreover, some papers describe the teachers role in a role playing exercise. According to Lederman [31], the teacher should guide and encourage discovery. The teachers are not responsible for providing learning; instead, they should provoke it [31]. Lederman [31] suggests that teachers should help the students rather than learning them. Thatcher [53] agrees with Lederman [31], stating that the teacher should be a facilitator and organizer.
3 Research methodology
This study investigates, using a case study research mode and a specific RE course design, if students with a higher grade in a role playing project perform better on an individual written exam compared to students with a lower role playing project grade. Based on potential performance differences, we seek to draw conclusion on the potential impact of role playing projects on student learning.
The case study comprises nine course instances at two universities. In each semester, the role playing occurred between the first and last week of the course running over 8 weeks (for details, see RE Course Description in Sect. 4 and Role Playing Project Setting in Sect. 5).
In this study we use statistical analysis to test if there is a correlation between the grade in the project and the score on the written exam. For all statistical tests, regression analysis at the 0.05 level was used.
3.1 Preliminary assumptions and main hypothesis
We had three preliminary assumptions about the role playing project’s impact on the learning outcomes:
-
1.
If students get a higher grade in the role playing project, then they have a deeper knowledge about requirements engineering.
-
2.
If a student scored higher in the essay part of the written exam, then the student has a deeper knowledge about requirements engineering.
-
3.
High performing students that did not performed well in the role playing project (i.e., got a lower grade in the role playing project), perform as well as students with a higher grade in the role playing project on the theory part in the written exam, but not as well on the essay part in the written exam.
Based on the above assumptions and that role playing may give the benefit of longer-term learning [23], our main hypothesis is that: role playing projects promotes students’ deep learning about requirements engineering.
3.2 Data collection
Data were collected from a total of 412 students that participated in the Requirements Engineering course between 2007 and 2014 at Lund University and Chalmers | University of Gothenburg. Table 1 shows the distribution of students by year and university.
All 412 students between 2007 and 2014 received the same setup of the course. That is, the lectures were built up in the same way, the project setup has been the same, and the written exam has had the same format. At Lund University, the course responsible (the second author) has been the same for all years, while at Chalmers | University of Gothenburg (2013 and 2014) it was a different course responsible (the first author). However, the course responsible at Chalmers | University of Gothenburg had been involved in the RE course at Lund University between 2006 and 2012. Besides the difference of course responsible between the two universities, the only difference between the students has been their involvement, engagement, and their grade in the role playing project.
The individual written exam consists of two parts, one theory part and one essay part. Each part is worth 50 % of the total points in the written exam. The theory part consists of multiple-choice questions where the majority of the questions are based on a proposition and a reason, which is illustrated in Fig. 1. The essay part consists of several short essay where a topic and keywords are given to the students. The topics are related to tasks that have been practiced in the role playing project, e.g., prioritisation, elicitation, release planning, or quality requirements.
3.3 Validity threats
One threat of this study is that we have only compared project groups using role playing and their score in the written exam. That is, we did not have a control group of students not being exposed to role playing to fully understand the impact that role playing has on students learning. The reason for this is that when we designed the RE course with project groups using role playing, the objective was to setup a good course where the students could learn RE.
The external validity is concerned with the ability to generalize the results, i.e., in this case the applicability of the findings beyond the RE course. One threat to external validity in this study is that only one course has been studied. Thus, the context, the RE course and the role playing project setup have been described in detail, which can help in understanding if this situation is similar to other cases and situations. Hence, it may help to judge the transferability of the setup and the results to other situations. Although the course was designed and executed in the same way at both Lund University and Chalmers | University of Gothenburg, the results from the two universities are similar, i.e., higher project grades lead to higher score in the written exam, which supports the generalization beyond the course at Lund University.
4 RE course description
At Lund University, Department of Computer Science, the Requirements Engineering courseFootnote 1 is an elective course for fourth-year students (i.e., master-level) majoring in Computer Science, Electrical Engineering, Engineering Physics, Industrial Engineering and Management, and Information and Communication Engineering Technologies. At Chalmers | University of Gothenburg, Department of Computer Science and Engineering, the Requirements Engineering courseFootnote 2 is a mandatory course for fourth-year students (i.e., master-level) majoring in Software Engineering and Interaction Design and Technologies. Moreover, it is an elective course for fourth-year students at any other program.
The teaching uses both a traditional approach of lectures and exercises as well as a group project using a role playing method [57], which runs over 8 weeks (the project setting is described in detail in Sect. 5). An overview of the content of the RE course is illustrated in Table 2. Between the years of 2007 and 2014, 289 students from Lund University and 123 students from Chalmers | University of Gothenburg have participated in this course.
The lectures provide an overview of the literature, i.e., a high-level structure of RE theory and thereby aid self-studies of the literature. The main objective of the exercises is to support the project by connecting theory to practice and to provide an opportunity to discuss RE techniques in detail. The project involves a number of deliverables and a final project conference where learning outcome of each project is presented. In addition to lectures, exercises, and the project, the RE course includes two laboratory sessions. The first laboratory session is about requirements prioritization where the students get acquainted with a prioritization tool, and takes place in the third week of the course. The second laboratory session is about Release Planning (RP) and takes place in the fifth week of the course. In the RP laboratory session, the students should get familiarized with the task of deciding which subsets of the requirements accumulated in the requirements backlog shall be assigned to upcoming releases using a release planning tool.
The final grade for the RE course is based on both the written exam (60 %) and the group project (40 %). To include the group project as an assessment criteria was based on feedback from previous students and is supported in the literature, e.g., Naz Memon et al. [33] discovered that as many as 72 % of the students suggested that their group project should be the assessment criteria for their RE course. The role playing project is described in more detail in the following section.
5 Role playing project setting
The main goals of the project work are to connect theory to practice, to give the students a concrete experience of practical requirements engineering techniques (related to cognitive learning in [23]), to promote student motivation (related to enhance student motivation and interest in [23]) through real stakeholders, and to provide the students group-learning with a focus on realistic problems (related to cognitive learning in [23]).
Each project were instructed to act two roles simultaneously (related to affective learning in [23]). The first role was that of acting as requirements engineers. This role’s main task was to do requirements engineering, including elicit requirements, specify and document requirements, validate requirements, prioritise requirements, and manage the requirements, according to a Project Mission (PM) and deliver a Requirements Document (RD). The second role was to act as customer, which includes provide knowledge about the project and information about the domain, for another requirements engineering project by delivering a PM. The two different roles and the setup of the project are illustrated in Fig. 2. Each student is expected to spend 80 h for the entire project where 80 % of his or her effort should be devoted to the first task.
In Fig. 2, ‘Your Project’ is a project group that consists of 4–5 students. In each project, three different roles should be appointed. All students are project members, but in addition to project members, two roles should be appointed by the project to two different project members. One student should be the project manager with the responsibility for distributing the RE work. Another student should be appointed the customer coordinator with the responsibility for distributing the work of acting as customer for another project.
In the second, fourth, and sixth week of the course, meetings between the project supervisor (a teacher in the course) and the student group are scheduled (to promote intragroup discussions). The meetings include, but are not limited to, checking the progress and status according to plan, plans for coming weeks, and discussions about challenges, difficulties, and open issues across projects.
5.1 Project process overview
The project has five phases (see Fig. 3). Phase one, Definition, includes writing, establish, and document the PM. The second phase, Planning, is about developing a Project Plan (PP) for the group work. In the First Iteration (third phase) is requirements engineering work conducted with focus on elicitation, prioritization, and specification, while in the Second Iteration (phase four) is the requirements engineering work focused on elicitation, specification, and validation. Finally, in the last phase, Finalization, the requirements engineering work is focused on specification and validation. In addition, the results and lessons learnt are packaged for presentation and course assessment. In the flowing subsections, each phase is described in more details.
5.1.1 Definition phase
In the definition phase, each project group shall prepare a Project Mission (PM in Fig. 2) for another project where they act as customers. The Project Mission (part of a PM is shown in Fig. 4) defines what type of system for which the other group (that act as requirements engineers) shall elicit, prioritize, specify and validate requirements. The project group creates a project of their choice (it is important that the group have a deep understanding of the application domain, a genuine interest in the system, and that they are able to assess the value of the detailed requirements) where they take one of two roles as customers. The first role is as a potential customer, i.e., the students act as a part of the market and may be a potential buyer of the product. In this scenario, the other project owns the product and decides the products’ contents. The second role is as product owner, i.e., the students’ act to develop and sell product to an open market. In the second alternative, the other project is subcontracted to only do requirements engineering. In addition to the two roles, each project group’s PM should define what type of system for which the other project group should elicit, prioritize, specify, and validate requirements for.
At the end of the first week, each project group submits their PM to the teachers who check the feasibility of the PM. Once the teachers have approved all PMs, they are published on the course homepage for the students to read and prioritize which project they would like to do RE for. The actual decision of which project group gets which PM is decided by a lottery on the following lecture. Figure 5 shows an example of the project setup in the course, i.e., the outcome of the lottery.
5.1.2 Planning phase
In this phase, each project group should plan their work throughout the project by creating a Project Plan (PP in Fig. 3). The PP should contain information such as description of the project, planned activities (with start and stop dates), description of deliverables, cost estimations (in terms of hours) for each activity, and how the results should be packed. Although the PP is only delivered once (at the end of the planning phase) to the teachers, the PP should be a “living” document throughout the entire project and should be updated as the students gain more knowledge. The estimated time for acting as customers should also be included in the PP.
5.1.3 First iteration phase
The requirements engineering work starts in the first iteration. In this phase, the RE work is focused on elicitation, specification, and prioritization of requirements. In the first iteration, the students are encouraged to try out as many elicitation and specification techniques as possible. At the end of the first iteration, each project group should deliver a first version of their Requirements Document (RDv1 in Fig. 3) to the project supervisor. We do not give the students any specific template for the RD. Instead, it is up to the students to decide whether they would like to use any available template from the literature, any industry specific templates, or to create their own template for their RD. The RDv1 should contain, besides business goals and project type, results from elicitation where at least three elicitation techniques [(e.g., interviewing the customer group, some kind of stakeholder analysis (an example from one project group is shown in Fig. 6), brainstorming] have been used, results from requirements specification (i.e., the written requirements) where at least four different specification techniques (e.g., context diagrams, features, virtual windows, task descriptions) have been used, and include requirements and supporting information on goal level, domain level, and product level. In addition, RDv1 must contain at least 20 specified requirements, which will be used in the RP laboratory session. Finally, each requirement must have a unique identity (name or number).
In general, the RDv1 contains 20–40 requirements at all different levels, ranging from very high-level abstract requirements to vey detailed low-level requirements. Since we encourage the students to try several different specification techniques (if they use them incorrectly it will not have any impact on their grades at this stage, it is more focus on try and learn by getting feedback from the project supervisors), most of the delivered RDv1 contains requirements using more than five different specification techniques where all of them includes some kind of context diagram (an example is shown in Fig. 7) and requirements written as features.
5.1.4 Second iteration phase
In the second iteration phase, the RE work continues with focus on elicitation, specification, and validation. The project groups should continue to elicit and specify more requirements (and improve the requirements in RDv1 based on feedback from the project supervisor) by using different RE elicitation and specification techniques. In addition to elicitation and specification, each project group should validate their requirements using a checklist-based technique and develop their own checklist for their customers. At the end of the second iteration, each group project delivers a second version of their RD (RDv2 in Fig. 3).
The RDv2 should contain, in addition to the criteria for RDv1, improved and updated requirements from RDv1, and results from the requirements validation. In addition to the Requirements Document (RD), each project group should deliver a Review Report (RR in Fig. 2). Each project group acts as a customer and therefore should review the RDv2 that the other project group has delivered using the provided checklist in RDv2. The goal of the RR is for the acting customers to validate the RDv2 for the PM that they created in the definition phase to suggest improvements to the RD. An example of a RR from one project group is shown in Fig. 8. The only instructions that are given, and are mandatory, for all groups are: (1) the project group should produce relevant and useful changes for improvement of RD2, and (2) each change must be ranked for criticality.
In general, the RD improves greatly from version 1 to version 2, in terms of number of specified requirements (ranging from 30 to 70), but also the quality of the specified requirements and the use of techniques for specifying the requirements. An example of a requirement from the second iteration is shown in Fig. 9. In this phase, the students should validate the RDv2 for their project, i.e., the RDv2 that the requirements engineer group have created. The created checklists for requirements validation varies greatly. The reason is that, similar to the template for the RD, we do not state which checklist they should use. Instead, we show a few different checklists (from the course literature [30]) and then instruct the students that they can use any of them, a combination of the different checklists, or create their own based on the checklists and knowledge they have gained so far in the course.
5.1.5 Finalization phase
The last phase, finalization, starts with a Conference Presentation (CP in Fig. 3). Each project group should prepare a short (approximately 10 min) presentation about the group project. The presentation should include: a description of the project (based on the PM), an overview of project results including techniques used, and important experiences and lessons learned. The remaining time in the finalization phase is devoted to RE work with focus on specification and validation based on feedback from both the project supervisor and the acting customers, and to package their results and lessons learnt into an Experience Report (ER in Fig. 3). The experience report should include: a description of the requirements engineering work, including experiences and reflections in relation to learning objectives; a description of the chosen methods/techniques for elicitation, specification, validation, and prioritization, and a motivation for why they chose the used methods/techniques; and a reflection of the usage of these methods/techniques in terms of what was successful, what was challenging, and why it was successful and/or challenging. At the end of the last phase, the project groups should submit the ER and their final version of the RD (RDv3 in Fig. 2) to their project supervisors.
The RDv3 should contain the same information as the previous two versions of their RD, but it should be improved and more complete than the previous versions. The ER should contain lessons learnt, what went according to plan and not, which elicitation, specification, prioritization, and validation techniques were used and why these techniques were chosen.
5.2 Assessment
As mentioned in Sect. 5, the project in the RE course consists of 40 % of the finale grade for the course. The general prerequisites are that all deliverables and versions must be in time, no free-riding on the rest of the group and that RDv1 must contain at least 20 requirements. The project deliverables that are part of the project grade is shown in Table 3.
Note that the PM is not part of the grade, nor is RDv1. The reason for not including RDv1 as part of the project grade is because we want the students to feel free to try out as many elicitation and specification techniques as possible without feeling the pressure that everything must be “perfect” already in the first version.
6 Results on the impact of role playing
The results show that students with a higher grade in the role playing project scored, in total, higher in the written exam compared to students with a lower grade in the project, a difference that is statistically significant using regression analysis (\(p<0.05\)). Since all students received the same treatment in terms of lectures, exercises, projects, and had the same course responsible, the role playing may be responsible for the higher exam scores earned by the students with a higher project grade (see further discussions in Sect. 7.3).
Looking into the different parts of the exam, i.e., the theory and essay parts, there was no statistically significant difference in the score of the theory part between students with a higher and lower project grade. However, students with higher project grades scored higher in the essay part compared to the students with a lower project grade, a difference which is statistically significant using the same regression analysis (\(p<0.05\)). This result is not surprising since the topics in the essay part are heavily correlated to tasks that have been performed during the project.
The results presented in this study is, to the best of the author’s knowledge, one of few papers that statistically assess the impact that role playing has on students performance in written exams. According to, e.g., Krain and Lantis [29] and Powner and Allendoerfer [42], the impact on students learning using role playing has been mostly anecdotal where research has found little or no statistically significant improvements in quantitative measures of students performance. In REE literature, many of the published papers (e.g., [16, 26, 52]) have reported on experiences of teaching Requirements Engineering courses in general, and a few papers (e.g., [1, 58]) have reported on experiences using role playing. However, none of these papers have looked into if role playing has any effect on students’ academic performance.
In the general Education literature, Powner and Allendoerfer [42] found that students who participated in role playing scored slightly better compared to the students who participated in classroom discussion; however, there was no statistical significant difference in the overall performance. Moreover, Raymond [43] investigated the impact of role playing simulation’s effect on exam scores. Raymond [43] compared two groups of students, one group had ‘traditional’ lectures, while the second group used simulations. The results show that the simulation was not associated with statistically significant improvements in exams scores. The difference between our result and the results in [43] is that we compared students that performed better (i.e., scored a higher grade) in the projects with students that scored a lower grade. That is, all students in our study used role playing as part of the project; however, the ones with a higher project garde scored better in the exam in total and in the essay part. Unlike Raymond [43], the course evaluations in our Requirements Engineering course have consistently have high scores over the years of teaching the course. Hence, we do not see any relation between lower teaching evaluations and exam scores.
7 Discussion
In this section, we discuss lessons learned during the years of teaching Requirements Engineering with a role playing group project. We also course evaluations and some important limitations of the results.
7.1 Lessons learned
After several of years of applying a role playing method for our group project in the RE course, it has become clear to us that this project setup can be effective in helping students to gain a richer and more realistic understanding of RE. Also, by using one project where the project group consists of the same students throughout the entire eight weeks minimizes the need to warm up students to the process before they can play the roles effectively [28].
For the requirements elicitation parts of the project work, a requirement to pass the project is to use at least three different elicitation techniques including stakeholder analysis and some type of interviewing technique. To motivate the students to try out more elicitation techniques, several more relevant techniques than minimally specified for passing the project is a requirement for the grade high distinction. This has motivated the students to use, in most of the projects throughout the years, more than three relevant elicitation techniques. Furthermore, although it is not a requirement to pass (or to receive the grade high distinction) the project, several projects have elicited requirements for the project using real stakeholders, customers, and user groups. That is, stakeholders that are not teachers or students in the course.
For example, one project conduct RE activities for an intelligent vending machine for soft drinks to be used at a school. The students in this project decided to interview four students (other than the students acting as customers in the RE course) to elicit requirements from “real” users, they interviewed two employees (teachers) at the school, and one employee from the cafe that was intended to maintain the vending machine.
Another example is a project for a digital wallet, not only did the students interview employees from the local public transportation sector to elicit requirements for using the digital wallet to pay for bus and train tickets, they interviewed personnel from two different banks to discover requirements about technical issues between the digital wallet and the bank systems and to find out if the banks would have been interested in a product like the digital wallet.
These two examples indicate that the students are motivated and interested in RE and that it is not just a good grade that motivates them. In addition, the projects that takes these extra steps in their project work do practice RE elicitation skills, which according to Berenbach [5] is an important skill to teach students in a RE course. Moreover, since we require the students to chose relevant RE techniques and motivate their choice in the ER, we believe, based on our experience from the course, that the students learn (to a certain degree) the ability to select and combine RE techniques that are suitable for a particular project based on the characteristics of the project, which is important to learn in REE [27]. A third lesson from these examples is that the students have the opportunity to communicate with real customers and users, which is an important part of RE [57].
Two important activities in RE are negotiation and prioritization of requirements. As reported in [57], these two activities are difficult to handle during exercises. Besides our first laboratory session where students are exposed to prioritizing requirements using a prioritization tool (described in Sect. 3), prioritization and negotiation are included in the project work. Each project should negotiate with their customers (the other project) and come up with a prioritized list of the most important requirements. Our experiences show that this part works very well and that most of the project groups use the prioritization tool from the first laboratory session in the project work. Moreover, over the years we have had a handful of project groups that asked real users/customers (i.e., not the student group acting as customer) to prioritize which requirements (based on a subset of all their requirements) they would like to have in a product similar to the one that the students are performing RE activities for. We believe this would enhance, to some degree, students’ ability to negotiate and prioritize requirements.
7.2 Course evaluations
This section is based on data collected from the official course evaluations (with a total response rate of 45 %) between the years of 2007 and 2014. Based on the Course Experience Questionnaires (CEQ) of student satisfaction with the RE course, our results do not indicate that the students find the RE course boring, which is reported in [32]. Looking into the open-ended questions in CEQ, the results indicate that our students enjoy the course and think it is an important subject to learn for their future career in industry. One student explained, “the group project was very educational where we had the opportunity to practice the theoretical information from the lecture”. Other students pointed out the importance of learning RE, which they believe is an important subject, but is lacking in other SE courses. Another important aspect of why the students were satisfied with the course was the mix of theory and practice. The lectures provide the students with the needed theory, while the exercise and the project that runs throughout the course focus on the practical side. For example, one student stated that the course has a “great mix of practice and theory”, while another student said that “the course is very good due to great teachers and the focus on the project for practicing the techniques”. This is not online with reported REE problems in the literature, e.g., Naz Memon et al. [33] reported that REE has a too strong emphasis on theory rather than the practical side.
In the REE literature, several papers, e.g., [17, 52], have reported that there is a lack of practice in RE activities in the courses. This is not in line with the results from our course experience questionnaire evaluations. Most of the students feel that there is enough time to practice RE activities in the project work, one student explained, we hade the opportunity to “directly transfer theory to practice in the projects”. Although practicing in the project work in the RE course is not the same as a real-life project in industry, we believe it provides the students with a close enough realistic project experience. Several students support this view, where one explained the realisms of the project as “the project in this course is the closest to a real-life project I have come across at the university”. Although most of the students see the projects as realistic ones, a few students would like to see more realistic projects. This was further explained by one student, would like to see “more realistic projects, i.e., the PM created by the other students should be more realistic. Hence, the teachers should check the feasibility and realisms of the PMs”. Although most of the students relate the project work to the importance, one student gave another explanation of why the course is important, it “feels like we master most of the areas in RE after taking this course”.
7.3 Limitations
The difference in project grades and scores in the written exam may be explained by that some students were mainly interested in passing the course, and not aiming for a higher grade. This may particular be a threat to the students from Chalmers | University of Gothenburg since the RE course was mandatory; hence, the students were “forced” to study the course. However, for the majority (over 70 %) of the students in this study, the RE course was optional, i.e., the students actively decided to study RE. Hence, this threat has a limited effect on the results. The difference in scores in the written exam may also be explained by that some students, regardless of project grades, are better in studying and would score higher in the written exam even without the role playing project. This may be a particular threat to the students from Lund University because the students are allowed to choose their own project members; hence, good students may decide to work together and thereby it may be the students ability rather than the role playing that influences the project grades. However, despite that the students from Chalmers | University of Gothenburg are randomly assigned to project groups, i.e., the students have no choice in deciding their project members, the result, when analyzing the project grades and exam scores separately between Lund University and Chalmers | University of Gothenburg students, follows the same pattern. That is, a higher project grade has a statistically significant effect on the students’ exam scores. Hence, this threat has a limited effect on the results.
8 Conclusion
In this paper, we discussed the use of role playing in requirements engineering courses and its impact on students academic performance in terms of scores on the written exam. We illustrated how role playing is used in the requirements engineering course to broaden students’ perspectives and learning. Data were collected (project grades and scores from the written exam) from 412 students from two universities between 2007 and 2014.
8.1 Summary of results and limitations
This case study investigates if there is a correlation between a role playing project grade and the grade of a written exam. The results show that students who performed better (i.e., scored a higher grade) in the role playing project scored higher (in total) in the written exam compared to students who had lower project grades, a difference that was statistically significant. Moreover, the results showed that there was no statistically significant difference in the scores in the theory part (multiple-choice questions) in the exam, but there was a statistically significant difference when it comes to the scores in the essay part of the exam. Although there was a statistically significant correlation between a higher project grade and higher score on the essay part of the written exam, the causal relationship is not clear. Ideally, a control group should have been used to be able to clarify the causal relationship; however, we decided not to use a control group due to ethical reasons. That is, we did not want to withhold the students with what we believe is a good teaching approach with the role playing project. However, the course evaluations, where the students have expressed the importance of the project, support our assumptions of the importance the role playing project has in increasing learning outcome.
8.2 Outline of further work
The following areas are interesting for further work:
-
The use of a control group to clarify the causal relationship between the role playing project and the score on the written exam. This could be done when introducing the role playing project to new courses. Instead of having a big bang introduction of role playing, the role playing project could be gradually introduced in the course. Thus, a control group could be used to evaluate its impact on students learning.
-
Additional data, in particular from Chalmers | University of Gothenburg, but also from other universities, is required to achieve more generalizable results.
-
In this study we have not evaluated whether role playing produce more substantial benefit, in terms of learning requirements engineering skills, in the long-term than in the short-term, which provide an interesting direction for future work.
In addition, it would be interesting if further work includes studies of the impact of role playing techniques in software engineering courses in general not only in courses restricted to the requirements engineering part of software engineering.
References
Al-Ani B, Yusop N (2004) Role-playing, group work and other ambitious teaching methods in a large requirements engineering course. In: Proceedings of the 11th IEEE international conference and workshop on the engineering of computer-based systems. pp 299–306
Auriol G, Baron C, Fourniols JY (2008) Teaching requirements skills within the context of a physical engineering project. In: Proceedings of the 4th requirements engineering education and training, (REET’08). pp 6–11
Aurum A, Wohlin C (2005) Engineering and managing software requirements. Springer, Berlin
Benbassat J, Baumal R (2002) A step-wise role playing approach for teaching patient counselling skills to medical students. Patient Educ Couns 46:147–152
Berenbach B (2005) A hole in the curriculum. In: Proceedings of the 1st international workshop on requirements engineering education and training (REET 2005), August 2005
Biggs J (2003) Teaching for quality learning at university. Open University Press, Milton Keynes
Baruch Y (2006) Role-play teaching acting in the classroom. Manag Learn 37:43–61
Bloomfield LP, Padelford NJ (1959) Teaching note: three experiments in political gaming. Am Polit Sci Rev 53:1105–1115
Boocock SS, Coleman JS (1966) Games with simulated environments in learning. Sociol Educ 39:215–236
Boocock SS (1967) Life career game. Pers Guid J 46:328–334
Brislin T (1995) Active learning in applied ethics instruction. J Excell College 6:161–167
Callele D, Makaroff D (2006) Teaching requirements engineering to an unsuspecting audience. In: Proceedings of the 37th SIGCSE technical symposium on computer science education (SIGCSE’06). pp 433–437
Chapman GM, Martin JF (1995) Computerized business games in engineering education. Comput Educ 25:67–73
Christiaens G, Baldwin JH (2002) Use of dyadic role-playing to increase students participation. Nurse Educ 27:251–254
Clark JM, Paivio A (1991) Dual coding theory and education. Educ Psychol Rev 3:149–210
Damian D, Al-ani B, Cubranic D, Robles L (2005) Teaching requirements engineering in global software development: a report on a three-university collaboration. In: Proceedings of the 1st international workshop on requirements engineering education and training (REET 2005). pp 685–690
Davis AM, Hickey AM, Chamillard AT (2005) Moving beyond the classroom: integrating requirements engineering research and education to improve practice. In: Proceedings of the 1st international workshop on requirements engineering education and training (REET 2005)
de Freitas SI (2006) Using games and simulations for supporting learning. Learn Media Technol 31:343–358
Doorn N, Kroesen JO (2013) Using and developing role plays in teaching aimed at preparing for social responsibility. Sci Eng Ethics 19:1513–1527
Dorn DS (1989) Simulation games: one more tool on the pedagogical shelf. Teach Sociol 17:1–18
Frash R, Antun J, Kline S, Almanza B (2010) Like it! Learn it! Use it? A field study of hotel training. Cornell Hosp Q 51:398–414
Gabrysiak G, Giese H, Seibel A (2011) Why should i help you to teach requirements engineering?In: Proceedings of the 6th international workshop on requirements engineering education and training (REET 2011). pp 9–13
Greenblat CS (1973) Teaching with simulation games: a review of claims and evidence. Teach Sociol 1:62–83
Greenblat CS, Duke RL (1981) Principles and practices of gaming-simulation. Sage, Beverly Hills
Greenberg J (1993) The role of role playing in organisational research. J Manag 19:221–241
Huijs C, Sikkel K, Wieringa R (2005) Mission 2 solution: requirements engineering education as central theme in the BIT programme. In: Proceedings of the 1st international workshop on requirements engineering education and training (REET 2005)
Jing L, Eberlein A, Far BH (2005) Combining requirements engineering techniques—theory and case study. In: Proceedings of the 12th IEEE international conference and workshops on the engineering of computer-based systems (ECS 2005). pp 105–112
Krain M, Shadle CJ (2006) Starving for knowledge: an active learning approach to teaching about world hunger. Int Stud Perspect 7:51–66
Krain M, Lantis JS (2006) Building knowledge? Evaluating the effectiveness of the global problems summit simulation. Int Stud Perspect 7:395–407
Lauesen S (2002) Software requirements—styles and techniques. Addison-Wesley, Boston
Lederman LC (1984) Debriefing: a critical reexamination of the postexperience analytic process with implications for its effective use. Simul Games 15:415–431
Liang P, de Graaf O (2010) Experience of using role playing and Wiki in requirements engineering course projects. In: Proceedings of the 5th international workshop on requirements engineering education and training (REET 2010). pp 1–6
Naz Memon R, Ahmad R, Salwah Salim S (2010) Problems in requirements engineering education: a survey. In: Proceedings of the 8th international conference on frontiers of information technology (FIT 2010), Article No. 5
Macaulay L, Mylopoulos J (1995) Requirements engineering: an educational dilemma. Autom Softw Eng 2:343–351
Martin J (1993) Episodic memory: a neglected phenomenon in the psychology of education. Educ Psychol 28:169–183
McKeachie W (ed) (1994) Teaching tips, 9th edn. DC Heath, Lexington
Morgan AL (2003) Toward a global theory of mind: the potential benefits of presenting a range of IR theories through active learning. Int Stud Perspect 4:351–371
Newmann WW, Twigg JL (2000) Active engagement of the intro IR student: a simulation approach. PS Polit Sci Polit 33:835–842
Polajnar D, Polanjar J (2004) Teaching software engineering through real projects. In: Proceedings of the Western Canadian conference on computer education (WCCCE 2004). pp 83–90
Pomerantz A (2003) Who plays the client? Collaborating with theatre departments to enhance clinical psychology role playing exercises. J Clin Psychol 59:363–368
Poorman PB (2002) Biography and role-playing: fostering empathy in abnormal psychology. Teach Psychol 29:32–36
Powner LC, Allendoerfer MG (2008) Evaluating hypotheses about active learning. Int Stud Perspect 9:75–89
Raymond C (2010) Do role-playing simulations generate measurable and meaningful outcomes? A simulations effect on exam scores and teaching evaluations. Faculty and Staff—Articles and Papers. Paper 22
Regev G, Gause DC, Wegmann A (2008) Requirements engineering education in the 21st century, an experiential learning approach. In: Proceedings of the 16th IEEE international requirements engineering conference (RE’08). pp 85–94
Regev G, Gause DC, Wegmann A (2009) Experiential learning approach for requirements engineering education. Requir Eng 14:269–287
Rosca D (2000) An active/collaborative approach in teaching requirements engineering. In: Proceedings of the 30th annual frontiers in education conference. pp 9–12
Scott JM (2001) Changing perspective: teaching undergraduates about the new inequality. International Studies Perspectives 4:340–348
Shellman SM, Turan K (2006) Do simulations enhance student learning? An empirical evaluation of an IR simulation. J Polit Sci Educ 2:19–32
Silberman M (ed) (2007) The handbook of experiential learning. Wiley/Pfeiffer, San Francisco
Smith ET, Boyer MA (1996) Designing in-class simulations. PS Polit Sci Polit 29:690–694
Smith R, Gotel O (2007) RE-O-POLY: a game to introduce lightweight requirements engineering good practices. In: Proceedings of the 3rd international workshop on requirements engineering education and training (REET 2007). p 42
Svahnberg M, Gorschek T (2005) Multi-perspective requirements engineering education with focus on industry relevance. In: Proceedings of the 1st international workshop on requirements engineering education and training (REET 2005)
Thatcher DC (1990) Promoting learning through games and simulations. Simul Gaming 21:262–272
Tuya J, Garcia-Fanjul J (1999) Teaching requirements analysis by means of student collaboration. In: Proceedings of the 29th annual frontiers in education conference. pp 11–15
Wheeler SM (2006) Role-playing games and simulations for international issues courses. J Polit Sci Educ 2:331–347
Yusop N, Mehboob Z, Zowghi D (2007) The role of conducting stakeholder meetings in requirements engineering training. In: Proceedings of the 3rd international workshop on requirements engineering education and training (REET 2007). p 48
Zowghi D, Paryani S (2003) Teaching requirements engineering through role playing: lessons learnt. In: Proceedings of 11th IEEE international conference on requirements engineering (RE 2003). pp 233–241
Zowghi D (2009) Teaching requirements engineering to the Baha students in Iran who are denied of higher education. In: Proceedings of the 5th international workshop on requirements engineering education and training (REET 2009). pp 38–48
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Svensson, R.B., Regnell, B. Is role playing in Requirements Engineering Education increasing learning outcome?. Requirements Eng 22, 475–489 (2017). https://doi.org/10.1007/s00766-016-0248-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00766-016-0248-4