Keywords

1 Introduction

Gamification is a recent strategy used in several contexts where the human factor is a key for the success of the different initiatives. In particular, gamification is an alternative to increase the motivation and commitment levels in work teams. The propose of use gamification in software process improvement initiatives is achieving high levels of motivation to work with team members, integrate participants to increase technical knowledge and increase progress indicators of project teams [1]. However, not only is necessary procedures and methods that guide the gamification of a non-gamified context, because it is also important have mechanisms to verify if the principles and fundamental gamification elements are properly incorporated in a gamification environment for achieving the expected results.

In this paper, we present a proposal for the assessment of gamified environments to assure that gamified experiences meet the purposes, goals, principles and elements that are defined as gamification fundamental components. Our proposal is oriented towards the definition of an assessment framework for gamified environments that comprises the identification and adoption of gamification principles, as well as a design method for gamified activities.

Section 2 presents the main concepts that fundaments the proposal. In Sect. 3 the assessment framework proposal is described. This section presents each framework layer and its components in a detailed manner. In Sect. 4 an example of the use of the framework are described. Finally, in Sect. 5 the conclusions and future work are presented.

2 Background

2.1 Gamification

Gamification is a growing trend where the human factor is fundamental in terms of motivation and engagement of the work teams [2,3,4]. Such trend can promote in individuals’ social interaction, quality and productivity of his actions [5]. Gamification can be defined as: (i) an approach that involves the same psychological experiences of the games [4]; (ii) the application of the same principles and environments of the games, regardless of expected results [2] and (iii) a process implementing motivational affordances for obtain psychological and behavioral outcomes in individuals [6, 7].

The growing interest in gamification as a strategy for process improvement in organization is evident because: (i) the growth of computer game industry over the last two motivating individuals to play; (ii) the pervasiveness of social media and mobile and web-based technologies is changing the way how individuals interact inside organizations [8]; and (iii) organizations are continually looking new ways for influence the behavior of employees and clients.

However, there are a few well-established theoretical guidelines for design gamified experiences. This problem is the main motivation of this work oriented to propose a framework like a set of tools for create gamification environments guaranteeing the application of gamification principals and the achievement of the specific goals.

2.2 The MDE Framework

MDE Framework is an approach to design games emphasizing the need to understand games mechanics, dynamics and emotions for obtain an improvement in commitment levels from team members, employees and customers [9]. Figure 1 shows the framework elements: mechanics, dynamics and emotions:

Fig. 1.
figure 1

MDE framework

Full size image
  1. 1.

    Mechanics are related with the designers’ decisions for gamify a non-game context such as: goals, rules and the guidelines that stablish the boundaries of the gamified environment. Game mechanics are known before the gamified experience beginning and are the same for all game participants. However, mechanics are not enough to create a gamified environment allows to obtain behavior changes in participants. For this purpose, its necessary keep in mind the others two elements of MDE Framework described below.

  2. 2.

    Dynamics is a result of how the players follow the game mechanics. In other words, game dynamics emerge during play [10]. In the case of gamified environments, the interaction of team members can lead to dynamics like cooperation, competition or conspiracy.

  3. 3.

    Emotions refers to the mental affective state and reactions promoted in players when they participate in a gamified environment. In this sense, the main goal for gamification is create the player enjoyment for guaranteeing the continuity of the gamified experience. Some positive emotions promoted by games are excitement, surprise, wonder and amazement.

2.3 Gamification Principles

Oprescu et al. propose ten principles for implement gamification experiences [12]:

  1. 1.

    Orientation: the participant is the center of the experience for achieving engagement, sense of control and self-efficacy.

  2. 2.

    Persuasive elements: through game elements, we can obtain change initial behavior of participants.

  3. 3.

    Learning orientation: games can develop skills and motivational outcomes.

  4. 4.

    Achievement based on rewards: games could be oriented to a positive feedback related with goals achievement as a way to increase self-efficacy.

  5. 5.

    Y Generation adaptable: games allow Y generation individuals express their opinion in a fun, engaging and rewarding environment.

  6. 6.

    Amusement factors: games include humor, play and fun elements to increase participants’ motivation and satisfaction.

  7. 7.

    Transformative: gamification combines competition and collaboration and could integrate moral decisions to enhance the desired skills in participants.

  8. 8.

    Well-being oriented: games could focus on personal well-being.

  9. 9.

    Research generating: games could allow participants to collaborate in order to identify future improvements in the application area.

  10. 10.

    Knowledge-based: gamification could be based on knowledge, either as an outcome or as feedback.

2.4 Assessment Proposals

Gamification is a trend that receives increasing attention from researchers as well as practitioners. For this reason, exists different approaches to faces gamification projects [13] including the phases business modeling, design, implementation and monitoring and adaptation [14]. However, the activity less mature is the monitoring of the gamification designs. For this reason, Heilburnn et al. [14] suggest the use of gamification analytics tools to measure the success of gamification designs allowing to quantify and understand user behavior improvements [15]. Moreover, they propose the use of semi-structured interviews as a mechanism for obtain a direct feedback of user satisfaction and additional requirements for the gamified experiences. In [16] the authors identify relevant software solutions for measure the fulfillment of user requirements in gamification projects.

In such proposals, the assessment is centered in collecting data for software applications about user behavior using gamification analytics. However, our proposal is oriented to a generic gamification environment, supported of not by software artefacts. For this reason, the data of participants’ behavior will be collected directly for users through neuroscience equipment. This alternative will allow discover at first-hand how participants react to the different elements of the gamified environment.

3 Assessment Framework Proposal

Software process improvement is an approach used by software industries for increase the productivity and quality of their processes and products. One important issue is the social interaction among the team members is improve motivation and engagement levels in participants. This situation could be address-using gamification like a strategy to improve the social factors related to the software development process. In this sense is necessary to define methods, guidelines and frameworks to assurance the adequate implementing of the gamification strategy.

In general, the framework proposed in this work is a conceptual structure aims to serve as a support or guide for creating gamified environments focused on SPI.

Our framework is a layered structure indicating what kind of components can or should be built and how they would interrelate. The framework structure is the result of the analysis of different studies, cited in the previous section, that describes gamification elements, components and principles. Such framework is composed by layers containing basic components (fundamentals, methods, procedures) looking for obtain the expected results with the incorporation of gamified environments in SPI initiatives. Figure 2 shows the layers of proposed framework. The framework proposes the creation of gamification environments starting from the identification and adoption of gamification elements. The identification and adoption of the principles is related to the analysis and contextualization of useful fundamentals for software process improvement, although the framework is structured in a general way.

Fig. 2.
figure 2

Assessment framework for gamified environments

Full size image

The framework begins with a principles identification layer as a conceptual support of the principles adoption layer. The first layer allows to reflect the elements necessary to guarantee that gamification is using adequately as a strategy to improve the participants’ behavior. Starting with the gamified environment design, the validated assessment is carried out based on principles identified, adopted and reflected in the design.

Finally, the framework improves through constant feedback including the four-layer conforming it. In the next section, each framework layer is described indicating the level of progress reached in the project in the actual status of the research.

3.1 Framework Layers

The four-layer conforming the framework are: (1) principles identification that will contextualized the gamified environment; (2) principles adoption identified at the previous layer; (3) gamified environment design and (4) assessment of gamified environment validating its design from the principles adopted at previous layers.

The framework layered-structure is defined identified for each layer three key elements: (a) base component, (b) procedural component and (c) results component.

  • Base component of a layer includes the fundamental aspects for the use of the framework. Each layer will have a basic starting point, so that the framework user can establish the application basis of each layer.

  • Procedural component of a layer is equivalent to methods, procedures and guidelines allowing to obtain each layer goals and expected results.

  • Results component of a layer includes templates and standards for indicating to framework user the structure of expected results in each layer.

First Layer: Identification

This layer establishes a procedure for identifying, defining and contextualizing the gamification. The components of this layer are:

  • Base component: basic gamification principles for the framework. Such principles are described by Oprescu et al. [12] taking advance of their easy of adaptation in a gamification strategy.

  • Procedural component: procedure for recent studies selection where the gamification and basic components are defined (See Fig. 3). In different studies, gamification definition is accompanied by basic principles as a fundamental pillar for the gamified environment construction. For this reason, in this layer is recommended the identification of additional principles, when the base principles established in the layer base component are not sufficient or a specific change is required.

    Fig. 3.
    figure 3

    Identification layer procedural component

    Full size image

  • Results component: Table 1 shows the suggested template for this component construction starting from gamification principles definition established in the layer base component (Oprescu et al. [12] proposal).

    Table 1. Template for result component of Identification layer
    Full size table

Second Layer: Adoption

Set of rules based on software engineering experimentation [17]. We used recommendation and methods generated from software engineering experimentation area because is considered a paradigm to helps establish a scientific and engineering basis for the software engineering field. This paradigm helps to build, analyze and evaluate models of the software process and products [17].

The selection of the appropriate research method to clarify the research question is one of the first activities proposed in this area. Table 2 shows the different types of research questions.

Table 2. Research questions types [18]
Full size table

The components of this layer are described below:

  • Base component: basic gamification principles identified in the identification layer. This component includes the principles described in the suggested templated for the principles identification layer (Table 1). Besides, we suggest the use of the MDE framework for the construction of this component as a guideline for games design.

  • Procedural component: the adoption of gamification principles aims to complete gamification conceptual basics. This component is built from activities and procedure rules (Table 3), as a fundamental basics for the comparison of tabulated principles, implementing the set of rules for adoption layer construction.

    Table 3. Activities and rules of procedural component of adoption layer
    Full size table

  • Results component: The graph shown in Fig. 4 is an example of result of the adoption layer. In this component, the construction of a similar structure as a comparison map is expected as a result component of this layer. The adoption of gamification principles related to MDE framework for games design is achieved when a structure like the graph of Fig. 5 is established.

    Fig. 4.
    figure 4

    Layer result component for adoption layer

    Full size image
    Fig. 5.
    figure 5

    Gamified activities design method [20]

    Full size image

An alternative representation of the result layer-component is by means of the template shown in Table 4.

Table 4. Result layer-component template for adoption layer
Full size table

Third Layer: Design

Sequential route for makes more dynamic the process of a gamified environment construction. This layer is built from sequential steps for generating gamified learning activities in a systematic way. Although the method [19] shown in this layer is focused on gamification as teaching strategy, it is expected that the evolution of the framework proposed in this work will lead us to generate a refinement of the method for design gamified activities that conforms a gamification environment for software process improvement. The components of this layer are:

  • Base component: the template and the comparison map established in the adoption layer are the basics for the construction of the design layer.

  • Procedural component: a method for gamified activities design. Such method aims to guide the creation of gamified activities based on: (a) experience as immediate resource for learning, (b) gamification as a teaching strategy for dynamizing the students classroom work and increase the motivation levels in participants [19]. The method is a sequential route that allows to gamified activities designer obtain a set of activities for the teaching of a specific topic. Figure 5 shows the sequential steps for the construction of the design layer.

  • Results component: gamified activities. The results of the application of the procedural component must be a set of activities necessary to gamify an environment. The basic elements of a gamified environment are: goals, materials, roles, steps and rules [20].

Forth Layer: Assessment

Application and gamification data contain valuable information about users and their behavior. This data can be used to measure the success of gamification projects, to analyze user behavior, and to continuously improve gamification designs.

However, to support the process of gamification analysis, appropriate tool support is missing. Moreover, the specific requirements for methods and tools have not been studied yet and remain unclear [14]. Some proposals like Heilbrunn et al. work shows the results of semi-structured interviews for evaluate gamification tools facilitating the identification of relevant software solutions and assesses them with regards to their fulfillment of user requirements in the gamification analytics domain [16].

Such proposal of evaluation of gamification analytics tools considering that the design of a gamification project required the implementation of related activities, where the design is implemented as software artifacts and functionally tested [16].

Although the gamified environment assessment can be conceived from this principle and it is possible to use the tools evaluated in [14] and in [16], our proposal in this work will be focused on the capture and analysis of user data that allow to measure participants behavior in a gamified environment independent to the support of technology. In other words, the assessment to be propose will be independent to the generation of software artifacts as part of the gamification environment.

One of the first alternatives we intend to design as a part of the framework is the used of data captured with neuroscience equipment. The propose is incorporating in this assessment the concepts that neuroscientists are discovering about the ways in which humans react to interactive design elements.

As some initial components of this layer we propose the following alternatives:

  • Base component: basic elements of a gamified environment. The basis for the construction of this layer includes the basic elements of a gamified environment defined in the design layer [20] and software artefacts and requirements identified, if they exist as part of the gamification activities defined in the design.

  • Procedural component: methods of measurement and data collection. The procedure that will be designed as a procedural component of assessment layer will include methods for collect user data for assessing their behavior through neuroscience equipment and HCI design aspects and assessment methods in the case that the gamified environment elements include software artefacts of user requirements.

  • Results component: Data analysis facilitating the improvement of the gamified environment. The expected results in this layer will be data analysis because these elements could show the effectivity of gamification in terms of user satisfaction faces of the gamified environment.

4 Framework in Action

In this section, we present a brief example of the framework application for creating a gamified activity related with the defect tracking in software development process. For the first layer (Identification), using the base component consists of gamification principles, the results of this layer are presented in Table 5, where some principles and expected benefits are summarized.

Table 5. Results component for identification layer
Full size table

For the second layer (Adoption), using the principles identification (Table 5) and the activities and rules defined in the procedural component, the result is a relation between gamification principles and MDE framework elements as shown in Table 6.

Table 6. Relation between gamification principles and MDE Framework
Full size table

In relation with the Design Layer, in [20] the use of the method for design the gamified activity for defect tracking are described. This work includes generates the definition of basic elements of the gamified activity: goals, roles, steps, materials and rules. The next step consists on define the components of the Assessment layer and apply them to different gamified activities confirming a gamified environment. The purpose is verifying if such activities meet the goals and expected results of the gamification initiatives.

5 Conclusions and Future Work

This work proposes an assessment framework for gamified environments aims to facilitate the verification of principles and gamification elements identified and adopted as a part of the design of gamified activities includes in a gamified environment for software process improvement. The framework pretends to be a first guide to orientate from the contextualization of a gamification strategy for software process improvement initiatives to the assessment of the gamified environment.

The assessment goal is verifying that the gamification principles are fulfilled in the gamification environment. For the reason, the proposed framework pretends to include several assessment strategies related to data analysis for measure the participants behavior.

Although we are working on the construction and adaptation of the framework layers, there are some proposals for HCI methods and approaches with tools and technologies from neuroscience laboratories that can be part of the framework as an assessment strategy.

As a future work, we suggest: (1) analysis for mix HCI measures and approaches for assessing the users’ behavior of a gamified environment under the framework and (2) adaptation or refinement of the method for design gamified activities oriented to industrial contexts.