Keywords

1 Introduction

Recently a lot of attention has been devoted to the monitoring of the acquired competencies during a learning activity as well as to the possibility to learning by using simulation of difficult situations or scenarios. Independently by the context, usually simulation attempts to mimic real-life or hypothetical behavior to see how processes, systems or hardware devices can be improved and to predict their performance under different circumstances. Commonly, monitoring focuses on data collection and supervision of activities during the real-life execution of a process, systems or hardware components to ensure they are on-course and on-schedule in meeting the objectives and performance targets. Currently, inside the learning engineering area the use of Business Process Modeling Notation (BPMN) [1] makes easier the simulation and monitoring activities due to the possibility of exploiting concise definitions and taxonomies, and developing executable frameworks for overall management of the process itself. Indeed best practice of Business Process modeling lets the use of methods, techniques, and tools to support the design, enactment and analysis of the business process and to provide an excellent basis for simulation and monitoring purposes. Examples can be found even in different environments such for instance the clinical one, for assessing and managing the patient treatment, and the financial sector for verifying and checking the bank processes. In all these application contexts, a key role is played by the data collected during the business process execution or simulation, which lets the possibility of reasoning about and/or improving the overall performance of the business process itself.

In the specific area of learning management, simulation and monitoring enhances student’s learning and improves their knowledge; they are also very important for assessment of the teaching performance. Indeed different conceptual and mathematical models have been proposed for model-based learning and several type of simulations, including discrete event and continuous process simulations have been considered [2]. However, the main challenges of existing learning simulation and monitoring proposals are about collaborative simulation, gamification and the derived learning benefits. In particular, gamification is becoming one of the main challenges in the simulation activity, that can be incorporated with the aim of using game-based mechanisms and game thinking to engage, motivate action, promote learning and solve problems [3]. Moreover, rewarding strategies are encouraged in order to stimulate intrinsic motivations within the members of a community.

In this paper, we address model-based learning management through the evaluation of some performance indicators useful for learning assessment. We present a Simulation and Monitoring framework able to support collaboration and social interactions, as well as process visualization, monitoring and learning assessment. The proposed approach can be compared to a collaborative game where a team of players composed of one coach and any number of learners work together in order to achieve a common goal. The main objective is consequently to provide an easy to use and user-friendly environment for the learners in order to let them take part of the process when their turn comes, assuming different roles according to the content they have to learn. The principal contribution of this paper is the architecture of a framework for simulation and monitoring of model-based learning able to provide feedback for evaluating the learner competency and the collaborative learning activities. The proposed simulation and monitoring framework has been applied to a case study developed inside the Learn PAd project in the context of Marche Region public administration and important feedback and hints have been collected for the improvement of the framework itself over the Learn PAd project duration.

In the rest of the paper we first briefly introduce some background concepts and related work (Sect. 2), then is Sect. 3 we present the main components of the simulation and monitoring framework architecture whereas in Sect. 4 we describe its main functionalities. Finally, Sect. 5 shows the application of the proposed framework to a case study and conclusion concludes the paper.

2 Background and Related Work

The proposal of a simulation and monitoring framework for model-based learning originated in the context of the Model-Based Social Learning for Public Administrations (Learn PAd) European project [4] addressing the challenges set out in the “ICT-2013.8.2 Technology-enhanced learning” work programme. Learn PAd project envisions an innovative holistic e-learning platform for Public Administrations (PAs) that enables process-driven learning and fosters cooperation and knowledge sharing. The main Learn PAd objectives include: (i) a new concept of model-based e-learning (both process and knowledge); (ii) an open and collaborative e-learning content management; (iii) an automatic, learner-specific and collaborative content quality assessment; and finally (iv) an automatic model-driven simulation-based learning and assessment. The developed Learn PAd platform will support an informative learning approach based on enriched BP models, as well as a procedural learning approach based on simulation and monitoring that will allow users to learn by doing.

Recently other EU funded projects of 6th and 7th framework programmes have been financed in the area of Technology Enhanced Learning. Among the relevant ones there are: (i) MATURE [5] which interlinks individual learning processes in a knowledge maturing process. In particular, the focus has been in the maturing process and in building tools and services to reduce maturing barriers, to embed learning more seamlessly in work processes and knowledge management systems; (ii) Mirror [6] which delivered a set of real-time, interoperable learning applications, based on a conceptual model of holistic continuous learning by reflection. The project incorporates in particular (collaborative) knowledge construction and creative problem solving and innovation; (iii) Target [7] which is based on a gaming activity so to deal with complex situations and results in experiences that are gradually honed into knowledge; (iv) Prolix [8] which aligned learning with business processes in order to enable organizations to faster improve the competencies of their employees according to continuous changes of work requirements. The solution to develop includes also workflows for competence building, simulations, and games for process-oriented learning and information exchange.

Considering the industrial and research learning context, BP simulation approaches are very popular since learners prefer simulation exercises to either lectures or discussions [9]. Simulations have been used to teach procedural skills and for training of software applications and industrial control operations as well as for learning domain specific concepts and knowledge, such as business management strategies [10]. Nowadays, more attention is given to business process oriented analysis and simulation [11]. Studies have shown that the global purpose of these existing business process simulation platforms is to evaluate BPs and redesign them, whereas in the last years simulation/gaming is establishing as a discipline [12]. However, these platforms present several shortcomings regarding their applicability to a collaborative learning approach. Namely, no existing platform regroups all of the main functionalities of a learning simulation solution such as facilities for providing a controlled and flexible simulated environment (for example allowing to switch between possible outcomes of a task, in order to explore the different paths of a process), good visualization and monitoring of a process execution flow (in order both to assist and evaluate the learners) [12]. The main challenges of a learning simulation are about collaborative simulation and the derived learning benefits. To answer all of these concerns a new learning simulation and monitoring framework is designed in this paper, providing a flexible simulation framework with a strong support for collaboration and social interactions, as well as process visualization, monitoring and learners assessment.

Concerning monitoring, existing works [13] combine modeling and monitoring facilities of business process. PROMO [13] allows to model, monitor and analyze business process. It provides an editor for the definition of interesting KPIs (Key Performance Indicator) to be monitored as well as facilities for specifying aggregation and monitoring rules. Our proposal is different since it addresses a flexible, adaptable and dynamic monitoring infrastructure that is independent from any specific business process modeling notation and execution engine. Other approaches [14] focus on monitoring business constraints at runtime by means of temporal logic and colored automata. They allow continuous compliance with respect to predefined business process constraint model and recovery after the first violation. Differently from these approaches, the proposed solution does not allow to take counter measures for recovering from violation of defined performance constraints. Moreover, in our solution these constraints are not specified in the business process but they are dynamically defined as monitoring proprieties that can be applied to different business process notations. In the context of learning, monitoring solutions can be used for providing feedback on training sessions and allow KPI evaluation. Some learning systems such as that in [15] propose customized learning paths that learners can follow according to their knowledge, learning requirements or learning disability. Changing and management of learning pathways as well as adaptation of learning material are made according to the monitored data. However, contemporary Learning Content Management Systems (LCMS) provide rather basic feedback and monitoring facilities about the learning process, such as simple statistics on technology usage or low-level data on students activities (e.g., page view). Some tools have been developed for providing feedback on the learning tasks by the analysis of the user tracking data and monitoring of the simulation activity. The authors of [16], for instance, propose LOCO-Analyst, an educational tool aimed at providing educators with feedback on the relevant aspects of the learning process taking place in a web-based learning environment such as the usage and the comprehensibility of the learning content or contextualized social interactions among students (i.e., social networking). The main goal of these tools is to support educators for creating courses, viewing the feedback on those courses, and modifying the courses accordingly. Differently from these solutions, other proposals [17, 18] focus on model-based learning and monitoring of business process execution. Specifically, [17] presents a flexible and adaptable monitoring infrastructure for business process execution and a critical comparison of the proposed framework with closest related works whereas [18] presents an integrated framework that allows modeling, execution and analysis of business process based on a flexible and adaptable monitoring infrastructure. The main advantage of this last solution is that it is independent from any specific business process modeling notation and execution engine and allows for the definition and evaluation of user-specific KPI measures. The monitoring framework presented in this paper has been inspired by the monitoring architecture presented in [17, 18]. It includes new components specifically devoted to the computation of the evaluation scores useful for the learning assessment.

Fig. 1.
figure 1

Simulation Framework Architecture.

Full size image
Fig. 2.
figure 2

Monitoring Framework Architecture.

Full size image

3 Simulation and Monitoring Framework Architecture

Extending the preliminary version of [19] in this section, we describe the high level architecture of the proposed Simulation and Monitoring framework, its main components, their purpose, the interfaces they expose, and how they interact with each others. In particular, as depicted in Fig. 1 each component is exposed as a service and provides an API as a unique point of access. Inside the Learn PAd infrastructure, the proposed simulation framework interacts with the Learn PAd components by means of the Learn PAd Core Platform and specifically through the Bridge and the Core Facade interfaces. Moreover, in the Learn PAd vision two levels of learners have been considered: the civil servant who is the standard learner, and the civil servant coordinator who is a generalization of the civil servant who is in charge to activate and manage a simulation session.

The simulation framework components are:

SimulationGUI: it is in charge of the interactions between learners and simulator’s components. It provides different facilities that are: (i) Chat areas that represents a space for learners to chat either one by one, one with the group of civil servants or with experts connected to the current simulation; (ii) Notification area which provides notifications to the learner; (iii) User input/output panel: this area contains forms for learners interactions; (iv) Context area: provides the documents related to the simulation, additional information or links to material that may be useful during the simulation activity; (v) Simulation teams members: learners involved in the simulation have a special placeholder so to distinguish them from the coordinator; (vi) Contextual search: allows to search among different kinds of information depending on the current displayed layout (users, processes, simulations, etc.); (vii) Simulation lifecycle menu: it allows the learner to choose among different views that are: Business process view allows to obtain a graphical representation of the current BP; Play: allows to run a simulation as an instance of a given BP; Save: allow to save the current simulation and restart it from the point in which it has been saved; Pause allow to pause the running of the current simulation; Stop: allow to stop the current simulation and exit; Coordinator: provides the name of the coordinator of the current session and allows him/her to modify it. Only the civil servant coordinator has the possibility to designate another civil servant to be the new coordinator of the simulation session; Stats&Logs: using an analytic dashboard, it allows the learner to display statistics and logs about all activities carried out during the simulation.

PersistenceLayer: it stores the status of the simulation at each step (i.e. BP executed task) in order to give to the civil servant the ability to stop it and restart when needed. Its main sub-components are: (i) the Logger that is in charge of storing time-stamped event data coming from the simulation engine; (ii) the BPStateStorage that allows to store/retrieve/delete/update the state of a given simulation associated to a BP; (iii) the TestDataRepository that collects the historical data that relate to the simulations executions.

RobotFramework: it allows to simulate the behavior of civil servants by means of robots. The Robots are implemented on the basis of the availability of historical data, i.e. the data saved in the TestDataRepository during a previous simulation session and provided by an expert who takes the role of the civil servant.

SimulationEngine: this is the core component of the simulation framework. It enacts business processes and links activities with corresponding civil servants or robots.

Monitoring: it collects the events occurred during the simulation and infers rules related to the business process execution.

Communication Middleware: it provides event-based communication facilities between the simulation components according to the publish/subscribe paradigm.

UserFacade: it is in charge of encapsulating real or simulated civil servants (i.e. robots) in order to make the learner interaction transparent to the other components of the architecture.

In the following more details about the simulation engine and monitor components are provided. More details about the simulation and monitoring design are in [20].

3.1 Simulation Engine

Simulation engine takes in charge the simulation of a given business process instance. It takes the form of an orchestration engine that invokes treatments associated to each activity of the current process. Such workflow may involve multiple civil servants taking different roles that may be present or not. For those that are not available, robots are used in order to mimic their behavior. A simulation manager is provided in order to manage BP lifecycle according to the current context (create, stop, resume, kill, etc.). Business processes are made of two kinds of activities: (i) Human activities involve civil servants who should provide information in order to complete the task. The concept of human activity is used to specify work which has to be accomplished by people; (ii) Mocked activities involve robots to compute the treatment associated to the activity. When the simulation engine invokes a human activity the corresponding civil servant is asked to provide input through a form. Those forms are managed by a form engine that delegates task to a robot if necessary. All the state information necessary to restart a specific simulation are stored “on the fly”. The civil servant may decide to freeze a running simulation, to store it, to backtrack to a previous stored state and to logout. He/she will be able to resume it later.

Business Process orchestrator takes in charge the step by step execution of a given BP instance. Such BP instance is made of a BPMN description enriched with necessary run-time information such as end-points of software applications mocks, user id, etc. The BP engine is connected with the Forms Engine in order to take in charge users and robots input/output. Different solutions for the business process execution engine are: Activiti [21], Camunda [22] and jBPM [23]. In this paper, we rely on Activiti [21]. In order to collect inputs from learners during a simulation session, a form engine has been defined so to design and run the proper corresponding forms. Forms Engine allows dynamic forms creation and complex forms processing for web applications. The processing of a form involves the verification of the input data, calculation of the input based on the information from other input fields as well as dynamic activation or hiding of the data fields depending on the user input. Inside our solution the javascript Form editor, called FormaaS, has been adopted. It allows to design and run javascript forms and to quickly define forms and executable code.

3.2 Monitoring

The simulation framework is equipped with a monitoring facility that allows to provide feedback on the business process execution and learning activities. Figure 2 shows the architecture of the proposed monitoring infrastructure. The design of this monitoring infrastructure has been inspired by [17].

For aim of readability, we list below the monitoring components presented in [17] and refer to [17] for the complete description of their functionalities:

  • Complex Event Processor (CEP). It is the rule engine, which analyzes the events, generated by the business process execution. Several rule engines can be used for this task like Drools Fusion, VisiRule, RuleML. Our instance is realized using Drools Fusion [24], that is able to detect patterns and monitor the business process performance metrics.

  • BPMN explorer. It is in charge to explore and save all the possible entities (Activity Entity, Sequence Flow Entity, Path Entity) reachable on a BPMN. Specifically, the extracted paths will be provided to the Rules Manager that through the Rules Generator will create, using the templates of rules stored into the Template Manager, a set of rules that aims to check the KPI defined on the business process.

  • Rules Generator. It is the component in charge to generate the rules needed for the monitoring of the business process execution and the assessment of the performance metrics. It uses the templates stored into the Rules Template Manager. These rules are generated according to the specific performance metrics to be assessed. A generic rule consists of two main parts: in the first part the events to be matched are specified; the second part includes the events/actions to be notified after the rules evaluation.

  • Rules Template Manager. It is an archive of predetermined rules templates that will be instantiated by the Rules Generator. A rule template is a rule skeleton, the specification of which has to be completed by instantiating a set of template-dependent placeholders. The instantiation will refer to appropriate values inferred from the specific performance metrics to be assessed. Once the synthesis of the new set of rules is completed, the new rules are loaded by the Rule Generator into the Rules Template Manager.

  • Rules Manager. The complex event detection process depends directly from the operation done by the Rules Manager component which is in charge to load and unload a set of rules into the complex event processor and fire it when needed.

  • Response Dispatcher. It is a registry that keeps track of the requests for monitoring sent to the monitoring infrastructure.

In this section a refined and complete design of the monitoring infrastructure is presented as depicted in Fig. 2. It includes three new components (shown in pink in Fig. 2) that are:

  • DBController. This component has been introduced to satisfy the Learn PAd requirements of having storage of simulation executions data. Specifically, the DB Controller manages the updating of the civil servant score during a simulation or the retrieval of historical data concerning the assessment level of the civil servants. The DB Controller interacts with the Learner Assessment Manager to get the different evaluation scores that will be defined in Sect. 3.2.

  • Learner Assessment Manager. It evaluates the learner activities and it is in charge to calculate the different scores. More details about this component are in Sect. 3.2.

  • Monitoring Manager component. It is the orchestrator of the overall Monitoring Infrastructure. It interacts with the Learn PAd Core Platform through the REST interfaces (core facade and bridge interface) and is in charge to query the Rules Manager. It also interacts with the BPMN Explorer and the Rules Generator. This component initializes the overall monitoring infrastructure allocating resources, instantiating the Complex Event Processor and instrumenting channel on which events coming from the simulation engine will flow.

Learner Assessment Manager. During learning simulation, it is important to asses learning activities as well as to visualize to the civil servants their success incrementally by displaying the achieved evaluation scores. To this end, the proposed simulation and monitoring component integrates a scoring mechanism in order to generate ranking of the civil servants and data useful for rewarding. The Learner Assessment Manager component evaluates the learner activities and is in charge to calculate different scores useful for the civil servant assessment. In addition, independently from any ongoing simulation, this component is in charge of retrieving the data necessary for the score evaluation and updating them on a database. Data collected during monitoring of business process execution can be used for providing feedback for the continuous tracking of the process behavior and measurement of learning-specific goals. All scores computed by the Learner Assessment Manager are then stored in the DB by the interaction with the DB Controller component. The evaluation scores computed by the Learner Assessment Manager relate both to the simulation of a session of the business process (session score(s)) and to the simulation of the overall business process (Business Process scores). Specifically, we define the session score(s) and Business Process score(s) as detailed below.

Session Scores. The civil servant may simulate different learning sessions on the same business process, each one referring to a (different) path. During a simulation session the Learner Assessment Manager computes the following scores:

  • The session score (called session_score), i.e. the ongoing session score of each participating civil servant.

  • An assessment value (called absolute_session_score) useful as boundary value for the session score.

Specifically, the session score is calculated using a weighted sum of scores attributed to the civil servant for each task of the Business Process realized during the simulation. Considering n the number of tasks executed by the civil servant during the learning session simulation and P the weight of the task, the session score is computed as follows:

$$ session\_score = \sum _{i=1}^{n} task\_score_{i} P_{i} $$

Each task of the Business Process is associated with a weight specified as a metadata. These metadata are attributed in the Business Process definition and defined by the modeler. The calculation of the score’s task is based on several criteria, namely number of attempts, Success/Fail and finally some pre-defined performance indicators named KPI (e.g. response time). The formula below allows calculating this score:

$$\begin{aligned} \begin{aligned} task\_score = success *(\frac{1}{nb\_attempts} + \\ \sum _{i=1}^{k}\frac{expected\_KPI\_value_i}{observed\_KPI\_value_i}) \end{aligned} \end{aligned}$$

where k is the number of KPI considered in the evaluation of the civil servants performances and success is a Boolean. For what concerns the boundary values useful for the learning assessment, the Learner Assessment Manager can provide the absolute_session_score, which represents the maximum score that could be assigned to the civil servant during a simulation session. Supposing that the maximum obtained value of the task_score is equal to k+1, the absolute_session_score is computed as:

$$ absolute\_session\_score = \sum _{i=1}^{n} (k+1) P_{i} $$

This absolute_session_score computes an accuracy measure of the session_score. A session_score value closer to the absolute_session_score represents a better performance of the civil servant for the considered simulation session.

Business Process Scores. During the learning simulation, the civil servant can execute different learning sessions on the same Business Process, each one referring to a different path. Therefore, the cumulative score obtained by the civil servant on the executed sessions is a good indicator of the knowledge of the civil servant about the overall Business Process. The learner assessment manager is able to compute the following scores related to the business process:

  • Business Process Score (called bp_score), i.e. the cumulative score obtained by the civil servant after the execution of different simulation sessions on the same business process. It represents the degree of acquired knowledge of the Business Process activities obtained by the civil servant.

  • Two assessment values (called relative_bp_score and absolute_bp_score) used as boundary values for the bp_score to evaluate the acquired civil servant competency on the executed business process. Specifically, the relative_bp_score is the maximum score that the civil servant can obtain on the set of simulated paths whereas the absolute_bp_score is the maximum score that the civil servant can obtain on all the possible paths of the business process.

  • A business process coverage percentage (called bp_coverage), i.e. the percentage of different learning sessions (paths) executed by the civil servant during the simulation of a business process. It represents the completeness of the civil servant knowledge about the overall business process.

In the following we provide more details about the above-mentioned scores. The bp_score is computed as the sum of the maximum values of session_score(s) obtained by the civil servant during the simulation of a set of different k paths (over the overall number of paths) on a business process, according to the following formula:

$$ bp\_score = \sum _{i=1}^{k} \max (session\_score_i) $$

Considering a bp_score and the set of k paths to which the bp_score is related to, the relative_bp_score is the boundary value representing the maximum score that the civil servant can obtain on the set of k paths. It is computed as the sum of the absolute_session_score according to the following formula:

$$ relative\_bp\_score = \sum _{i=1}^{k} absolute\_session\_score_i $$

Considering all paths of a business process to which a bp_score is related to, the absolute_bp_score is an additional boundary value representing the maximum score that the civil servant can reach. It is computed as the sum of the absolute_bp_score for all the paths of the business process according to the following formula:

$$ absolute\_bp\_score = \sum _{i=1}^{\# path} absolute\_session\_score_i $$

The more the bp_score is close to the relative_bp_score the more the civil servant reaches the maximum cumulative learning performance on the different simulated sessions. The more the values of bp_score are close to the absolute_bp_score the more the civil servant knowledge about the overall business process is complete.

Finally, the bp_coverage value is an additional measure for evaluating the completeness of the civil servant knowledge about the overall business process. It is computed as the percentage of different paths (k), executed by the civil servant during the simulation of a business process, over the paths cardinality as in the following:

$$ bp\_coverage = \frac{k}{\# path} $$

When the civil servant executes all paths of the business process, the computed bp_coverage is 1. A bp_coverage value closer to 1 represents a better performance of the civil servant for the considered business process simulation.

4 Functional Specification of the Learning Simulation and Monitoring Framework

The Simulation and Monitoring framework provides the subsystem where learners can simulate the business process interactively and is used by one or multiple civil servant(s) in order to learn processes. As mentioned in Sect. 3, the Simulation and Monitoring framework distinguishes between the two following actors: the civil servant coordinator who is in charge of starting a simulation session and the civil servant who represents a generic participant to a simulation session. In particular, the civil servant coordinator can request to start a new simulation execution of a Public Administration business process or he/she can manage an ongoing one by for instance inviting/cancelling other civil servants. The civil servant coordinator can also restart/stop a current simulation session and redefine a new coordinator. On its turn, each civil servant has different possibilities like for instance joining, disconnecting or pausing a simulation session, chatting, asking for evaluation/help, or managing his/her own profile.

The Simulation and Monitoring framework functionalities have been split into three different phases: (i) Initialization in which the simulation framework is set up; (ii) Activation in which the participants to the simulation are invited; (iii) Execution in which the participants effectively collaborate each other during a learning session. During the Activation phase, the civil servant can select the type of simulation he/she wants to execute. Specifically, three different types of simulation are provided:

Individual Simulation. The civil servant decides to execute the simulation without interacting with other human participants. In this case the other participants are emulated by means of Robots (see Sect. 3 for more details). The creation of robots instances is performed before the simulation execution.

Collaborative Simulation. This option of simulation involves the collaboration of several human participants (no robots instances are involved). During the collaborative simulation, users can interact between them using chat instruments. This will improve performances of the overall learning session due to the possibility to rapidly share experience between human participants. This kind of simulation can be considered the most interesting from the learning point of view, because cooperation can make learning procedures more intensive and productive. Diversities will raise up and the opportunity to reflect upon encountered issues will help learners to improve their knowledge and better understand the problem. For activating a simulation, the system requires that all the civil servants involved have joined the session in order to provide an online collaborative environment. Moreover, the Simulation and Monitoring framework also supports asynchronous tasks execution among simulation participants. If a civil servant does not satisfy the simulation requirements or time constraints, the civil servant coordinator may decide either to kick the civil servant, or to swap him with another one among those available, or replace him with a Robot.

Mixed Simulation. This type of simulation requires the participation of both humans and robots. This usually happens when there are not enough civil servants to cover all the necessary roles to execute a BP or if one or more civil servants leave the ongoing simulation (disconnection or kick). The activation of a mixed simulation can be done only if the following two constraints are met: (i) the required instances of robots are ready; (ii) all the invited civil servants have completed the connection procedures.

Both gamification and serious game concepts are also included in the proposed Simulation and Monitoring framework so to engage civil servants during training tasks and activities to be learned. Specifically, two main gamification elements are included in the proposed simulation and monitoring framework for educational purposes: (i) progression that allows the learner to see success visualized incrementally by the achieved evaluation scores; (ii) virtual rewards that allow learner who satisfies some conditions to be automatically awarded by the platform with a specific certificate that gives to him/her additional rights. For more details about the gamification model used in the proposed simulation and monitoring framework we refer to [25]. During the different types of simulation, the monitoring component checks if execution patterns will be respected during the simulation of a business process. In order to do that, the simulation engine interacts with the monitoring component through a pre-fixed set of messages specifying the set of events, detected failures and time values useful for evaluating the learner’s competency and assessing non-functional properties such as the overall simulation time completion.

5 Learn PAd Simulation and Monitoring Framework: The Application to Learn PAd Case Study

In this section, we show the application of the proposed Simulation and Monitoring framework to a case study developed inside the Learn PAd project with the collaboration of SUAP (Sportello Unico per le Attività produttive) officers from both Public Administrations Senigallia and Monti Azzurri. The scenario refers to the activities that the Italian Public Administrations have to put in place in order to permit to entrepreneurs to set up a new company. In particular, the case study is focused on the Titolo Unico process, i.e. the standard Italian procedure to be applied so to start a business activityFootnote 1.

Fig. 3.
figure 3

Titolo Unico Business Process.

Full size image

In this case, the entrepreneur notifies to the municipality and third parties organizations the starting of a commercial business activity and self-certifies all the required documentation. Then the entrepreneur has to wait for a decision taken by the SUAP office before to really start the activity. If necessary, SUAP office could require document integration or organize specific conference, called Service Conference, for critical decisions. In this case the participants (municipality offices, third parties administrations, and entrepreneur) discuss about the specific situation and decide if the application is acceptable or not. In general, the whole process has to be performed within 60 days: within 30 days the regularity of the application must be verified and in the remaining 30 days the decision must be prepared. If a conference has to be conducted, process duration is extended to 120 days. That is after the 30 days for verifying the regularity of the application, 60 more days are available for activating and performing the conference and for reaching a common decision. A simplified BPM representation of the described process is depicted in Fig. 3, in which sub-processes are not presented.

Using the proposed Simulation and Monitoring framework, the Marche Region personnel has the possibility to learn the steps necessary to organize a Service Conference and practice with the documentations and several criticalities and exceptions that could be encountered during the Titolo Unico process.

Fig. 4.
figure 4

LearnPAd Platform.

Full size image

In the Learn PAd project the Simulation and Monitoring framework has been integrated in the more complex project platform as depicted in Fig. 4; however it can also be executed independently. It is out of the scope of this paper to provide a complete description of the Learn PAd platform; however in this specific case the Simulation and Monitoring framework interacts with the other components of the platform by sending through the Learn PAd Core component the simulation events and basic KPI evaluations using a REST event API.

In particular, the simulator interacts through the Learn PAd Core with the Recommender component which is in charge of providing recommendations for individual learners, suggesting learning material or activities to improve bad learner’s performance scores. Moreover, recommendations can also be made for entire organizational units like the SUAP office or the whole Public Administration. They may refer to organizational activities recommended for improving a bad KPI value. In this case study, because the Simulation and Monitoring framework is integrated in Lean PAd, it is automatically instantiated as soon as the platform is executed without requiring additional actions.

In the Titolo Unico process considered in this case study the main actors involved in the simulation process are: (i) the learner, i.e. a Regione Marche employee, who would practice on this BP; (ii) an expert called Expert1 who is a Regione Marche domain expert employee who can provide suggestions or recommendations trough the Recommender; (iii) a second expert called Expert2 who is again a Regione Marche domain expert employee and who actively participates to the simulation.

Fig. 5.
figure 5

Create a simulation session of a process.

Full size image

Interacting with the GUI the learner who wants to start a learning activity, can create a new simulation session of selected BP process (see Fig. 5). In this case, the Titolo Unico BP is by construction a single-user process, however collaborative and mixed simulation can also be executed by selecting other available BPs. Thus the learner logs into the Learn PAd platform and starts running a stand-alone simulation. For aim of simplicity the duration time of each task has been reduced in proportion so to make the overall simulation completion time within the range of minutes. The learner by a browsing mode decides to start a new simulation session by clicking on the “simulate” button shown in the process landing page (i.e. the summary page of a BPMN, where a picture of a process is also shown). In particular, the user can configure the simulation session selection setting the different parameters of the process (see Fig. 6). Depending on the selected BP, the framework is also configured with a set of test application forms, along with their associated metadata info, which are completely separated from the data of the historical cases provided to the Recommender.

When a simulation starts, the Simulator and Monitoring framework sends “simulation session start” event with the associated metadata to the Learn PAd platform. Then the learner can execute all tasks of the business process. In particular, the monitoring sub-component collects data of interest (as detailed in Sect. 3.2) and updates the internal database accordingly. Once a task is completed, the associated scores are computed and updated. If all the inputs provided by the user during the task simulation have been evaluated correct, the framework indicates that the task has been validated, and will display new tasks corresponding to the continuation of the process (Fig. 7). Otherwise, the simulator will indicate that the submission is incorrect. In such a way the Simulation and Monitoring framework drives the learner through the process and assesses his/her activities with respect to what it is expected (possibly using previous correct runs of the same activities).

Fig. 6.
figure 6

Configuration of a simulation session.

Full size image
Fig. 7.
figure 7

Simulation Execution.

Full size image

Once the simulation session is terminated, the final scores are shown through the interface as reported in Fig. 8.

Fig. 8.
figure 8

Simulation session completion.

Full size image

In the following, referring to Fig. 3, we describe in detail which are the tasks of the business process executed by the learner during the simulation and for each task the input and output data. When the simulation starts, the task “Receive Instance” is executed using the available robots and metadata because it is not in charge of learner. It passes directly to the execution of the task “Assess Application”. The input data for this task is the pdf file with the application data. The output of this task is a statement saying whether the application is accepted or not, the values of relevant KPIs for this task, and in case the application is not accepted, a motivation for rejecting the application. In this task, the learner can ask to the Recommender which provides feedbacks according to use-case formatted data.

The learner executes then the task “Send Communications of non-admissible Instance to Third Parties”. The input data for this task is the pdf file with the application data. The output of this task consists of the list of the third parties involved and for each third party a referent person and a text message. During the execution of this task the learner can ask the Recommender which gives feedbacks about similar cases and third parties involved.

The learner executes the “Check Integration” task. The input data for this task is the pdf file with the application data. The output of this task is a report on the application data and decision about whether arrange or not the conference of services, and if yes the motivation. Also in this case the learner can ask the Recommender which gives feedbacks about similar cases.

The learner executes the task “Active Service Conference” that is included in the sub-process, the input data for this task is again the pdf file with the application data. The output of the execution of this task is a list of involved third parties. Also in this case the learner can ask the Recommender which gives feedbacks about similar case. Note that ranking of similar cases may change.

The learner then executes the task “Send Authorization Document”. In this case, the input data for this task consists of the pdf file with the application data and a document containing the minutes of the service conference. The output of the execution of this task is a summary of the service conference, a document specifying if an integration is needed or not. In case an integration is needed a message is sent to the sender of the application including integration requirement. Also in this case, the learner can ask the Recommender which gives feedbacks about similar cases.

At the end of each task and of the overall simulation the associated scores are shown to the learner and the validation of the performed activity provided.

During this first validation, the Simulation and Monitoring framework has been used by different end-users inside the Italian Public Administration and comments and suggestions have been collected. If from one side all users agreed that the framework represents a very good means for improving the understanding and practice of the administrative process, from the other side requests for improvements have been collected. This meanly concerns the usability of the framework especially in case of collaborative simulation as well as score visualization and management. This validation provides a positive assessment of the Simulation and Monitoring framework and a very important starting point for the next release of the learning system.

6 Conclusions

The paper presented a simulation and monitoring framework for assessing the acquired competencies during a learning activity as well as simulating difficult situations or scenarios. In particular a detailed description of the framework components and main functionalities is provided. Differently from the other simulation environment the proposal of this paper supports collaboration and social interactions, as well as process visualization, monitoring of learning activities and assessment. Specific attention has been devoted to the possibility of gamification during the simulation of the business process to be learned, so to engage more users while training them and improve their knowledge. A set of evaluation scores for assessing the leaning activity has also been proposed. They refer to both the simulation sessions and the overall business process simulation; relative values are automatically computed and updated by the proposed simulation and monitoring framework during each execution.

A version of the proposed framework, integrated into the generic architecture of the Learn PAd project, has been presented and used for the learning assessment activity of a case study developed inside the context of Marche Region Public administration. Preliminary results collected evidenced its importance in improving the understanding and practice of the administrative process, as well as the possibility of executing collaborative simulation and providing learners assessment. Moreover, this real case study also provided important feedbacks for the improvement and extension of the framework itself during the project duration. Future works include: (i) improvement of some parts of the architecture, such as the Test Data Repository and Robot; (ii) integration of usability concepts as well as the improvement of evaluation score visualization and management; (iii) enriching the set of learner’s evaluation scores considering for instance the number of errors made during the execution of a path; and finally (iv) evaluation of the industrial significance and benefits of the proposed framework in different application areas of technology enhanced learning.