Keywords

1 Introduction

Serious Game (SG) design is often a pedagogical, design-driven top-down approach [32]. Insofar, SG designers, mindful of the game objectives, try to work along a scale of fun to simulation, while keeping the game’s requirements in mind. This then determines the pedagogical approach implemented, implying a top-down approach to the game’s design. This often leads to a disconnection of low-level game implementation aspects and high-level instructional design aspects of SGs, i.e. it is a challenge to manage the relation between the instructional design and actual game design implementation in an appropriate manner. Therefore, previous work identifies a lack of suitable methodologies [1], resulting in the development of new methodologies like Learning Mechanics-Game Mechanics (LM-GM) [2], focusing on the construct of SG mechanics as the mechanics supporting the required transition [3, 4]. This is also addressed by the Activity Theory-based Model for Serious (ATMSG) method [5] that considers several involved viewpoints and supports a systematic way of connecting learning objectives with game play objectives, as well as instructional aspects.

Developing SGs is not only challenging, but also costly and time consuming [1] and thus the idea of reusing components, assets and knowledge in design, development and implementation processes [6] is based upon the believe that this will not only lead to lower costs, but also higher quality, in terms of more efficient resources usage, enhanced error handling, higher usage of standards and less time consumption in the game design process [1, 6,7,8,9]. The idea of reusing software components is nothing new [7]: considering SGs, component such as design patterns, application frameworks, program libraries, program generators and so on, are suitable for re-use [10]. However, the challenge to connect instructional design and game design remains, and is often addressed by SG mechanics. In a previous project, some re-use principles were established [1]:

  • Passive components and assets that are reused as such, like the narrative. Yet, to enable successful reuse, such items should be accompanied by sufficient and easily accessible documentation that provides guidance based on previous experiences.

  • Active components and assets that evolve and are repurposed, like an activity to carry out using a specific game mechanics or something else that evolves or changes in the play. In this context version control issues should be given special attention.

As mentioned above most games are specifically designed to fit both a specific target group and a specific objective (often a learning objective [1]). However, while reusing components might be a good consideration from a quality and cost perspectives, this has consequences regarding newly introduced requirements and needs and their influence on the actual game design. Flexibility and adaptability of the re-used game (or game component) play a major role. In order to investigate this more, we have defined two research questions and used a case study to investigate this:

  1. 1.

    How does the reuse of game component influence the implementation of a target group’s needs?

  2. 2.

    How to include participants in the redesign process of a game scenario?

In order to address the research question, a single, empirical case study was conducted. The case study method facilitated an in-depth understanding of potential benefits and challenges of participatory design of serious games in the context of reuse [11]. According to Meredith [12], a case study is an suitable method in situations with a significant amount of unknown elements. Thus, considering the limited number of empirical studies regarding participatory design on reuse of SGs, this method was considered appropriate.

2 Sota

There are several frameworks that can be used to design new games, but few are useful as a design and analysis tool – an important characteristic for reuse [13,14,15,16]. As described in previous work [13, 14], and above, the reuse of existing games is often difficult: they are designed for a very specific purpose and the possibility of modding games is limited [14, 16], since they are often linked to different domains and thus, experiences [16]. Previous work shows, that it is necessary to use a manual process to investigate to what extent components can be changed and how this can be done: by using an authoring tool, changing the source code, adding plug & play components, etc. [13] and then create a “new game” scenario, thus going beyond mere content change. Furthermore, SGs are often developed using agile development methods [17], with regular testing and incremental improvements, in order to adjust the different game components to the game’s purpose. This is a tedious and costly process. While there are several suitable models for designing new games, few consider re-design. As mentioned above, models considering both analysis and design can address this issue. Two SG frameworks integrating these aspects are the LM-GM model [2,3,4] and the ATMSG model [5, 19].

The LM-GM model allows users to describe games on the basis of different pedagogical approaches. Learning mechanics (LM), including various pedagogical aspects, can be mapped to different game mechanics (GM). A list of learning and game mechanics can be found in [4]. Application in game analysis is quite straight forward: for each game situation the user identifies the GM and LMs and connects it to a certain level of Blooms revised taxonomy [4]. After that, in order to re-use a set of LM-GM, it is required to understand their relationship and the implementation, as well as, and that is the only tricky part, the dynamics in the game flow.

The ATMSG is a conceptual model that supports a systematic and detailed representation of educational SGs, depicting the ways that game elements are connected to each other throughout the game, and how these elements contribute to the achievement of the game’s desired pedagogical goals. It was designed by Carvalho [18] to fill the gap as presented in the introduction. It supports the user both in analysis of existing games, applicable for evaluating re-use potential (or to implement internal improvements), as well as in the design of new games. To achieve this, the ATMSG provides a structured way to connect several levels of the game, reaching from high-level game requirements, both educational and entertainment-related, down to the game mechanics. In re-use, it helps to decompose a game in order to understand the interaction, as well as the connection, of the game’s pedagogic and ‘fun’ elements on several levels. A very detailed description of the method can be found in [18].

Furthermore, in addition to the software patterns, there are also specific patterns that are related specifically to games. In their work Björk, Lundgren and Holopainen [33] investigates patterns of different games, and defined a set of core elements that needs to be included. Their intention was to create a tool that support creative work, which is essential in the game design process as well as useful for problem-solving. However, a main challenge in designing games by re-using existing games is, as addressed by our research questions, the balance between the realism of what the potential users need and the limitation of the existing software. A way of overcoming this challenge is to involve the users in the design process, so that his/her needs can be captured and matched against the limitation.

There are several different kinds of participatory methods, but all are characterized by the involvement of participants for solving a problem [20]. The level of involvement and engagement as well as the required knowledge of the participants may vary. But the main intention remains to involve stakeholders (i.e. the invited participants) and give them an opportunity to influence the decision-making process [21]. The type of methods may vary, but include methods such as brainstorming, surveys and questionnaires, tours, focus groups, participatory planning, and expert panels. This approach brings additional challenges, such as selecting the right level of participation in the design process, which has been discussed in several studies [22,23,24]. Additionally, [25, 26] list a poor connection between game and players, a poor representation of the real system by a game, a poor reach of the intended goals of the game and more as typical challenges for participatory design in serious games. According to [27], the required elements and the game play/game scenario need to be sufficiently realistic, and mirror the processes, the structure and the outcome of the corresponding real world problem adequately, but there is still a difficulty to transfer the real life experience to a game [28]. As [29] confirms, this may lead to low acceptance and validity of the game.

3 Experimental Set up (Case Study)

3.1 Existing Requirements

In the cross-institutional research project DigiLab4UFootnote 1, real laboratories are digitized to offer a hybrid IoT learning and research environment. The project consists of two German and one Italian Universities as well as a research institute and intends to expand even further to increase the laboratory variety. The goal of the case study is to establish a digital laboratory offering that can be used by any kind of students from bachelor to doctoral students. In order to realize the resource sharing of the different laboratory providers, the underlying trust relationship, that is required to share laboratory resources within the cooperation, has been examined. The laboratory providers need to know and trust on: “(1) states (conditions) of shareable assets in regard to capacity, presence and/or (idle time), capability; (2) previous experience in the sharing of same resource; (3) restrictions and compensation; (4) level of behavioral congruence of actors participating in the sharing; (5) regulatory issues and dispute resolution” [30]. Successful cooperation between different parties or organizations always depend on several aspects, e.g. personal characteristics, previous experience or behavior. These cooperative projects bring many advantages such as reduced cost or additional knowledge and resources. Nevertheless, involved parties might be driven not only by common but also individual goals, leading to opportunistic or unexpected behavior. The aim of the serious game is to simulate several scenarios and to let the player learn and raise awareness about various roles’ motives, actions and their consequences. Sharing resources might be beneficial but also stressful or arouse unrealistic expectations. Eventually, a player should be aware of interdependencies, gains and common pitfalls in cooperative projects. Additionally, this game is should be used to analyze player interactions and behavior to further investigate the trust model.

3.2 Description Existing Game Engine

The be.mog (BIBA Engine for Multi-player Online Games) engine, which has been chosen for reuse as discussed below, supports process-oriented scenarios. A process is divided into process steps, which are executed successively. The be.mog engine supports the following entities:

  • Games and Scenarios: A be.mog game is spread over one or more scenarios, which are played in a sequential order. These scenarios are equivalent to game levels. Associated to each scenario is a topic, which is essentially a description of the subject under consideration. In this example, there is only one scenario with the topic given above.

  • Players, Groups, Sub-Groups and Roles: In each scenario the players could be organized in groups and sub-groups, which might represent companies and their departments in real life. Groups and sub-groups have their own descriptions. Each player can have a different role in each sub-group, e.g. an employee or a group leader. Beside name, user identifier, password, and other relevant information, a characteristic role description is provided to the player. The case study features two groups (universities), where each has one sub-group (department).

  • Process and Process Steps: A specific process is associated to each group which needs to be followed by the players to play the game. The process is further divided into process steps which need to be completed in a sequential order. Each of these steps needs to be completed to conclude the overall process. A process step can be either completed by performing some action or by completing a set of documents.

  • Actions: Some process steps may be completed by applying an action, chosen out of a set of actions. Actions are always assigned to a specific player. The application of an action reveals further information for the player. Actions can only be applied by players.

  • Events: Events can only be set by a facilitator. The facilitator may choose an event from a predefined list of events and apply it to a specific group of players. The players are informed about the occurrence of the event and get further information about it. Events can be set in any of the process steps and may rewind to the process to one of the preceding steps.

  • Documents: Documents are associated to process steps and players. A document is a collection of document entries (fields), which can be edited by the player who owns the document. Each document entry has a type (e.g. text or numeric) and may have a predefined value and a target value. Documents might be visible from the beginning or they are created when specific process steps are completed. Players can work on documents while they are visible and until document completion. The associated process step is completed when all documents associated to this process step are completed. The owner of a document can manage the access rights of the document by providing view and edit rights to other players.

During the game, three performance indicators (related to a virtual time, costs, and quality) are updated upon the completion of each process step. Also, the occurrence of events can influence the values of these performance indicators. These indicators are used when playing against a given goal like reaching the fastest time, least cost, or highest quality. During game play, a facilitator is watching and supervising the players. The facilitator can observe the results of the players (the content of documents and actions applied by players so far) and intervene by setting some disruptive events, which may influence the direction of some of the players.

3.3 Approach

The approach employed in this case study consisted of five steps. First, exploratory workshops have been organized to initiate the development of different game scenarios and to identify game requirements. Second, requirements have been analyzed and mapped on existing be.mog entities. Third, one of the scenarios served as basis for game design, which was subsequently implemented with the be.mog engine. Fourth, the scenario has been tested with a small group of potential users (N = 4). And, fifth, an evaluation workshop has been carried out.

The exploratory workshops led to two main scenarios called “Remote Lab Lecture” and “New IOT Remote Lab”. Within the first scenario, Klaus is a lecturer from Koblenz who wants to use the Position Lab at the HFT Stuttgart to demonstrate a practical example during his lecture. He is already a user of the virtual labs network. He uses his account to book a slot suitable for his lecture. Lukas is the lab admin at the HFT Stuttgart and receives that request. He must prove whether the lab is available and whether Felix, a lab technician, can provide support for Klaus before and during that time slot. Eventually, Klaus can rate his experience with booking and using the HFT lab remotely on a scale of 0 to 5 stars.

Within the second scenario, Eric belongs to the academic staff at the University of Paris. Maja, his superior wants to integrate their IoT sensor lab into the Virtual labs network platform, to improve utilization. Eric contacts the support team. Paula receives his request and provides a form that she and Eric fill in together to sum up all necessary data for the University of Paris to join the project. The filled form is sent to Tom, a platform manager, who reviews the form. Tom and Maja discuss all critical points and prepare a contract. Eventually, Paula and Eric are organizing all further steps, while Magalie handles the technical set up of the integration.

The roles and steps of interactions have been identified together with their requirements. The identified game requirements were mapped to common game elements to create a list of elements that should be supported by a game that serves as a potential basis for reuse. The conclusion from this step was that the entities provided by be.mog are sufficient to allow the implementation of the scenarios as be.mog game scenarios. The scenario called “Remote Lab Lecture” has been refined and detailed information has been collected to implement it as a be.mog game scenario. This work included the detailed specification of the necessary entities (roles, organizations, groups, process steps, actions, documents, etc.). With this level of detail, the necessary data was provided for the be.mog engine and the scenario became executable (playable) as first prototype. Figure 1 provides an overview on the main entities of the first scenario. The solid lines represent the process flow, while the dotted lines represent dependencies between individual process steps.

Fig. 1.
figure 1

Organizations, departments, roles, and the process flow of “Remote Lab Lecture”

Full size image

The playable scenario has been tested with various users. This was organized within a workshop setting where the participants first played the game scenario, and second, provided open and unstructured feedback. These testing sessions were used to adapt and enhance some of the present entities for the second prototype.

Finally, a bigger workshop has been organized with 18 participants playing the game scenario and providing structured feedback through questionnaires concerning the game evaluation. In total, 18 questionnaires have been collected.

4 Results

In general, players considered the implemented game mechanics suitable. Remarks show that the use of roles as game mechanics was well understood, but the remarks also show that there is a need for elaborating these role descriptions in more detail in order to create a more realistic scenario. Conclusions about individual roles are difficult to draw. A further observation was, that early user involvement was appreciated, however, it was a challenge, that the team was changed and extended during the design process. Nevertheless, this is rather common in such research project, so that even if organizations are involved at an early stage, their members vary over time.

The answers of two open questions and their impact on the game mechanics of the be.mog engine and the content of the scenario (or story or its presentation) is analysed in detail below. The answers of the question “If you feel like the kind of game or the reused games are not an appropriate choice, please tell us why” and the resulting consequences are shown in Table 1.

Table 1. Answers and consequences of question “If you feel like the kind of game or the reused games are not an appropriate choice, please tell us why”
Full size table

Some players had difficulties to use the game engine at first. Even after presenting the user interface in detail, a demand for a tutorial remained. Some players claimed to have a better overview if they had insight into the processes of both participating organizations, but this was intentionally not provided to create a realistic setting, in which people would not know the processes of other cooperation partners. The options presented to players seemed to be too simplistic and not well explained. Therefore, an increase in options and better explanations will help to create a more realistic implementation of that game scenario. A further discussed feature is be.mog’s integrated chat function that supports communication between players. Some participants reported that this technique felt a bit outdated and asked for more modern communication means within the game. Other participants lamented the pace of the game (again, a feature closely aligned with similar real scenarios) and asked for reduced waiting times, which could be achieved by restructuring the processes of the scenario. Altogether, there is no need to change the game mechanics except the communications means, but there is potential in refining the story structure and its presentation.

The questionnaire provided a possibility for further comments, which are shown below in Table 2, including the resulting consequences.

Table 2. Answers and consequences of question “If you have any further comments, please let us know”
Full size table

Some participants criticized the responsiveness of the be.mog engine, which could be improved by an enhanced notification system and additional pop-up messages. Gender balance must be improved by introducing female roles, and again the “realism” of the game scenario has been criticized. Additionally, a reflection phase (on the pros and cons of cooperation) should be included and the introduction of push-events has been suggested. One participant provided a list of potential improvements. Amongst others, there was a demand for directly influencing the performance indicators (costs, time, and quality). This can be achieved by implementing another game engine entity called measures. Measures can be compared with events with the main difference that measures are under the control of the players, not the facilitator. Additionally, the concept of visualizing the next steps in the game can be enhanced. Altogether, there were three serious proposals for game mechanics enhancements: 1) Enhanced notification and pop-up messages, 2) push-events, and 3) measures, and some suggestions for story and presentation enhancements concerning better visualization and explanation, a reflection phase, gender balance, and increased “realism”.

5 Discussion

As described in Sect. 4, we started the involvement of the users at an early stage, before the question of whether to re-use existing an existing game or develop a new one had been decided. The starting point was a set of scenarios that had been developed based on an analysis of the project objectives and the consortium construction. The groups were asked to provide further input on two (out of four) of these scenarios. The matching process between user requirements and existing game scenario limitation was a manually process, as expected according to [13, 14]. For this step we used the ATMSG and, based on the results, took the decision that re-use of the be.mog structure would fit the purpose, though limiting the outcome. According to the re-design [15, 16] recommendations, this will allow a re-use of corresponding game mechanics, and this assumption was to a large extent confirmed by the case study participants. Of specific interest in the improvement is that 16 out of 18 find the use of interdependent steps and 13 out of 18 scenario and role description as mechanic suitable. These mechanics have the advantage that they can be easily changed or extended, so that the flexibility of the game scenario is high, and a customization is possible. A result that requires some more thoughts and investigation in order to improve is connected to resource management (3 of 18 did not find this suitable at all). This is an essential game mechanic for the scenario outcome, and thus it is vital for the game experience that it is perceived as at least neutral. A reflection on why we got these results, is connected to the focus in the underlying re-used scenario. In the design of those scenarios we so far have implemented using the be.mog engine for different purposes, we put much attention on these mechanics on role and scenario descriptions (i.e. the narrative setting) and the interconnection between the different work processes. Resource management has also been a part in these scenarios, but more as a help function (i.e. passive component) for the decision-making, it was never an active component nor did we ever do an alignment of this specific LM-GM relation in the existing game plays. It was required for being able to address the risk assessment, the ideation of new ideas and served as a boundary to make the scenario more realistic.

Furthermore, comparing two other results, appropriateness of re-using this game and the game experience (Fig. 2) shows a different picture: The figure below shows that the participants had a strong feeling of learning about risks and uncertainties, while not so much on benefits. This clearly indicates that we in the re-construction did not manage to decouple the LM-GM connection on risks and the uncertainty (a main objective of one of the game scenarios originally design [31] and re-connect that mechanic in such a way, that the emphasis on risks and benefits would be similar experience in the game play. Since this was actually given in the narrative, it is to assume that we need to analyze the constructive alignment in the first game in more detail, in order to achieve a proper decomposition and then make a new alignment of the LM-GM connection. A second observation which is also connected to the instructional design and the balancing of the game play is the realism, as mentioned by [20, 25, 27, 28]. Some of the participants reported, that they did find their first input sufficiently well covered, but still the realism was not assessed as sufficient. Based on the remarks, one reason for this was in the values of key performance indicators we used (time, costs, quality). For development reasons, and due to lack of real values from the processes (i.e. the time it takes for a certain process, the cost model behind each decision and each process etc.), the values of indicators were arbitrary and not realistic. From a game design perspective, i.e. for a software engineer to check and test whether it is functional or not, it is normally sufficient to implement the mechanic to see how it work, but for the game experience, this case study showed how big the influence of using the right values actually have on this user group (i.e. non-game designer or developers). For designing new games, we would still have used software engineers, developers and game designer for this phase of the development process. However, since we re-used a game engine, the game was playable (and bug-free) to a much larger extent than normally, and we assumed that it would be sufficient to just tell this user group (i.e. participant taking part in the project but having no competencies in game design) that the key performance indicator values would not reflect reality. This shows how important the alignment of abstraction level and which user groups we involve when, of high relevance is when using participatory approaches are.

Fig. 2.
figure 2

Game experience results

Full size image

Summarized, we can therefore state that the results of the case study indicate that the influence of the re-used components is high. Of specific attention for a successful re-usage is not the re-use of game mechanics but the relation of GM and LM and the dynamics. This further confirms the observation made in [4, 14,15,16] that it is essential to know how the first game designer thought when constructing the LM-GM relationship as well as the dynamic in the game play was constructed, while designing a new scenario. This can, to some extent, also answer why there is so little re-usage, even if analysis tools like LM-GM and ATMSG models provide good support, this study shows that it is necessary to know the reasoning behind the starting scenarios in order to understand specific results in the assessment of the new scenario. This either requires that the same designers are involved or that the documentation of the considerations behind a scenario needs to be very detailed. Therefore, it is expected that the re-use of a specific game mechanic is much higher than the reuse of the construct game mechanics-learning mechanic.

Based on the remarks related to the usage of participatory design methods, it can be concluded that in line with [27, 28], it is essential for a game scenario to involve the users early in the process in order to ensure the realism. However, as the usage of arbitrary values of specific key performance indicators show, that attention needs to be paid to when a stakeholder group is involved in the process, since the level of abstraction depends much upon their knowledge. It would probably therefore have been beneficial to ask the users for suitable values and implementing something similar also in the prototype, instead of only asking for the function (the key performance indicators). Consequently, the answer to research question 2 based on our case study results is that the stakeholders should be involved at an early stage, and then probably first when the scenario is sufficiently realistic. This will however need more verification, since we only had one scenario and 18 participants.

6 Conclusion and Next Steps

This paper uses a case study for investigating if participatory design approaches and agile software development methods are appropriate in terms of re-usage of game components or game scenarios.

Since this is just one case study, more studies will need to be carried out for making a clear statement, but in general, this study confirms the relevance of user involvement already at the conceptual level, before any decision on game design has been made. Regarding the re-usage, the study indicates that a thorough alignment of the input of future users (here the game scenarios they developed) with existing re-usable game components at an early stage has a very positive influence. However, when it comes to testing, the study indicates that the timing and the group selection needs to be better aligned to the real competencies of the user groups, since we see a clear discrepancy between what we wanted to test (function of the KPI as GM and design element) and what the participants focused on (realism), since this is imperative for the game experience.

In the next steps, we have therefore asked the participants to further elaborate on the scenarios and improve those as well as elaborate more on the indicators. This feedback will be integrated in an updated scenario and we will carry out a new set of experiments.