Keywords

1 Introduction

Electronic systems are becoming more and more complex and software intensive. This situation calls for modern software and systems engineering practices in order to keep high productivity and quality levels. In the last decade, the ecosystem around Model-Driven Engineering (MDE) has flourished, providing developers with a plethora of tools. However, these tools need to be further developed to scale up for real-world industrial applications. They also need to be enhanced in order to provide advantages at runtime as well. This represents a real opportunity for achieving a complete continuous systems engineering lifecycle, thus connecting together the design and runtime phases [1, 2].

The MegaM@Rt2 project’s main goal is to create a framework incorporating methods and tools for the continuous development and runtime support of complex software-intensive systems. Our current architecture vision and development over the MegaM@Rt2 framework integrate three main complementary big capabilities: systems design engineering, runtime analysis, and global model & traceability management. The project is organized around the research work and related technical developments concerning the tool sets supporting those capabilities.

The research topics include holistic Systems Engineering covering design, verification and validation; Runtime Analysis dealing with monitoring, online testing and verification as well as models@runtime techniques; and so-called Mega-Modelling, i.e. large-scale model and traceability management. The framework is under evaluation by 9 industrial case studies ranging from transportation - avionics, railway, automotive, traffic monitoring; and telecommunications - short range communications, base transceiver stations; to logistics - indoor positioning, smart warehouses domains. Among the partners providing use cases in the project, we can cite Thales, Volvo Construction Equipment, Bombardier Transportation and Nokia. These organizations have different product management and engineering practices, as well as regulatory and legal constraints. This results in a large and complex catalog of requirements to be realized by the architecture building blocks at different levels of abstraction. Thus, the development of the MegaM@Rt2 framework is based on a feature-intensive architecture and on a related implementation roadmap that is kept up-to-date. A comprehensive set of the project information, as well as the published deliverables, are all publicly available from the project web site [3].

In this paper, we present the main project research and technological results after two years, and outline the outstanding challenges and further work. To this intent, the rest of the paper is structured as follows. Section 2 briefly describes the MegaM@Rt2 overall approach. Then Sect. 3 focuses on the three complementary tools sets that are designed and developed in the project to support this MegaM@Rt2 approach in practice. We notably insist on the main related research achievements we obtained so far, as well as on still open research and technical challenges. Finally, Sect. 4 concludes by summarizing the main results from this first phase of MegaM@Rt2 and by opening on some future work to come during the second phase of the project.

2 Description of the Overall Approach

As stated in the Electronic Components and Systems for European Leadership program’s Multi-Annual Strategic Plan [4], design methods and related technologies should fully support the constant technology push and corresponding new user/society demands of products/services based on more and more complex Electronic Components and Systems (ECS). This is particularly true in the context of the involved software components relying on hardware configurations and their interactions e.g. with their underlying environment, being very often numerous, complex, heterogeneous and strongly interrelated. In the past, Model-Based Engineering principles and techniques have already shown promising capabilities that have been experimented in such context. However, they have generally failed in terms of (1) scalability to support real-world scenarios implied by the full deployment and use of complex ECS and (2) efficient traceability, integration and communication between two fundamental system levels which are design time and runtime, notably as far as non-functional properties and their verification & validation aspects are concerned.

As a consequence, the overall idea of MegaM@Rt2 is to scale up the use of model-based techniques by offering scalable methods and related tools interacting between both design time and runtime, as well as to validate the designed and developed approach in concrete industrial cases involving complex ECS. To this intent, MegaM@Rt2 proposes an overall model-based approach combining existing techniques to be enhanced when relevant, and novel ones to be developed when needed. A fundamental challenge notably resides in providing efficient traceability support between the two levels i.e. from design models to runtime ones and back. In parallel to these, modern large-scale industrial software engineering processes require thorough configuration and model governance to provide the promised productivity gains. Thus, a scalable mega-modelling approach is being designed and will be deployed to manage all the involved artifacts e.g. the many different models, corresponding workflows, configurations, etc. and to better tackle their large diversity in terms of nature, number, size, complexity, etc.

To cover all these topics and deal with the complete value chain, MegaM@Rt2 brings together prominent tool developers and vendors and research organisations with state-of-the-art methods and tools that are validated in highly relevant European industry case studies. The end users from the space, naval, railway, smart grid, smart warehouse and telecom industry domains are driving the project by providing real-world requirements and case studies as well as by validating and endorsing the MegaM@Rt2 results.

Figure 1 provides an overview of the MegaM@Rt2 global approach and emphasizes its key principles and concepts. Industries apply a set of current engineering practices based on SysML, AADL, EAST_ADL, but also Matlab/Simulink, and Method B, each one producing specific design models, requirement specifications and resulting software and hardware artefacts. MegaM@Rt2 suggests to integrate those artefacts into a global system model providing a complete view of the Cyber-Physical System (CPS), and detailing the component, behaviour and desired quality properties of the system. These properties are then an object of exhaustive continuous testing and monitoring in the runtime environment to detect deviations in real-time, thanks to the configuration of the target platform and the injection of probes in the software. The detected deviations plus all the traces information collected in the process are analyzed to detect the impacted components in the integrated view of system models. When possible, automatic repairing suggestions are provided to correct the issue and reconfigure or redeploy the system to start the next iteration of the continuous integration process. This approach was further developed in [5] where we defined the specific tool sets - their requirements and features as well as outlined integration means.

Fig. 1.
figure 1

Overall conceptual architecture of the MegaM@Rt project.

Full size image

The methods and tools provided by MegaM@Rt2 are evaluated and applied in several industrial case studies. Each individual case study defines a set of key performance indicators (KPIs) that are used to evaluate the improvement that the new technologies provide. The case study specific KPIs are aggregated into project level KPIs which provide a quantitative evaluation of the project goals.

The project has set challenging goals in terms of KPIs such as:

  • Reduction of design time/design effort in the range of 10%–50% by design artefacts reuse.

  • Reduction of validation effort in the range of 10%–30% by automated trace collection and analysis.

  • Reduction 10%–50% in time/effort required for managing and handling all the involved models (e.g. time for model retrieval and access).

  • Reduction 10%–50% in time/effort required for tracing and handling all the involved models at design and runtime levels (e.g. creation of and access to relations between system and traces models).

The above-mentioned KPIs are measured through out the project industrial case studies. At the current stage the first evaluation phase has finished. The next sections present the preliminary results.

3 Results of the First Evaluation Phase and Outstanding Research Challenges

At the time of writing this paper, the MegaM@Rt2 project has entered its second half. In the following sections, we provide an overview of the current achievements of the project by focusing on research work and the corresponding results that we have already obtained.

3.1 The MegaM@Rt2 System Engineering Tool Set

This tool set aims to support system design activities. It has been architected around three main topics: (i) requirements analysis & specification, (ii) system modeling and (iii) model verification & validation. The approach integrates up to 20 different open source tools, mainly Eclipse-based, such as Modelio, developed by the consortium research partners. These tools support a variety of current engineering practices based on standard modelling languages, profiles and extensions like: UML, SysML, MARTE, AADL, EAST_ADL, etc. The framework is designed to integrate additional “external” tools like Matlab/Simulink, AUTOSAR, Modelica and others, based on specific needs of the industrial partners.

Different techniques have been adopted to ensure the correctness of system models, either in terms of verification of languages syntactic paradigms (e.g. using SAT- and CP-solver technologies) and in terms of functional and non-functional validation of system artefacts with respect of given requirements (e.g. through model simulation, model testing, machine learning technique, etc.). For example, [6] proposes a framework to reason about the satisfiability of class models described using the Unified Modeling Language (UML). It allows to identify possible design flaws as early as possible in the software development cycle, by annotating UML Class Diagrams with Object Constraint Language (OCL) invariants. Then, the Constraint Logic Programming (CLP) paradigm allows to reason about UML Class Diagrams modeling foundations thanks to a translation to Formula.

Several other research areas have been investigating. For instance, the current trend on Internet-of-Things Systems of Systems (IoT-SoS) implies significant evolution of modeling, analysis and design approaches [7].

Separation of concerns is one of the fundamental principles allowing to build well-structured software and improving its maintainability/evolutivity. Executable models are good candidates to capture the behavior of a software-intensive system using separation of concerns approach. In [8], Domains Specific Languages (DSL) have been exploited to create executable models when business operations are tied to specific technological platforms. This method is applied both at design-time for creation of executable models with EMF and at run-time by monitoring operation calls from the deployed execution engine.

Another aspect investigated in [9, 10] is the availability of platform-independent SW models and HW synthesis tools able to automatically produce efficient implementations based on performance predictions of the system model and this on many different distributed and parallel computing resources.

Safety critical systems, e.g. as proposed by Bombardier Transportation and ClearSy in the project, require specific support for safety analysis, assessment and certification. The contract-based approach is adopted by some of the framework tools and is presented and discussed in [11]. It is based on finding static schedules relying on contracts and using this information in the verification process to reduce the number of invariant annotations needed. Moreover, contracts can be used to make compile-time scheduling decisions, improving runtime performance.

A complementary research area is related to the application of the Aspect-Oriented Methodologies focusing on the reduction of the modeling and verification effort by applying aspect-oriented principles in model construction [12]. The industrial partners have a preference for more classical and consolidated methodologies. However, such capabilities are still available for possible future applications in case needed.

In the general case, a main achievement is the ongoing contribution to the MARTE standard, as presented in [13] and responding to the Request for Information issued by the OMG for a new MARTE 2.0. Partners proposals have been collected in an initial survey, then an answer to the RFI has been prepared and sent back.

Finally, the last project period will focus on the exploitation, at design time, of the runtime trace collection and analysis capability, in order to address possible model refinements in the context of feedback loops. To this intent, the most promising approach is the one provided by PADRE tool on performance anti-pattern detection and model refactoring [14, 15]. Finally, as a part of an effort to automate system engineering, [16] provides a systematic mapping study on published tools and approaches that can be used for generating API documentation, or for assisting in the API documentation process. the paper presents an overview of what kind of tools have been developed, what kind of documentation they generate, and what sources the documentation approaches require.

3.2 The MegaM@Rt2 Runtime (Trace) Analysis Tool Set

This tool set aims to define new methods and tools for creating and managing models at runtime verification and testing, including automated runtime testing and monitoring as well as a model-based log collection and analysis infrastructure supported by tools such as PauWare or CertifyIt. This runtime tool set integrates 24 tools that further propose automated code generation, model execution as a part of a system, runtime verification and online testing, such as CompleteTest, JTL, PauWare, Smartesting tools, AIPHS, Comformiq Designer, Modelio, etc. These tools within the MegaM@Rt2 approach integrate with the analysis tools. The main ongoing activity is related to establishing a smooth connection with the analysis tools, that will allow user-friendly and simple inclusion within the continuous development process, addressed in MegaM@Rt2.

Several results have been published. In the context of testing and test generation several papers address test generation using UPPAAL model checker and its extensions. For instance, [17] outlines a method for testing energy consumption in embedded systems using energy-related mutants for EAST-ADL architectural models, which are converted to UPPAAL Timed Automata and used for test generation UPPAAL Statistical Model Checker (SMC). A complementary approach is presented in [18], where we show how architectural models described in the EAST-ADL architectural language can also be used for testing the energy consumption of embedded systems, after transforming them into networks of formal models called priced timed automata. A mutation testing approach for UPPAAL TA has been proposed in [19] to mutate UPPAAL-TA models and use them for generating tests used for evaluating security vulnerabilities of web services. Last but not least, in order to enable the analysis of failed traces and quick fault localization, [20] proposes an approach that converts concrete test sequences generated and executed by Uppaal Tron against the system under test into symbolic traces that can be imported in the Uppaal tool and visualized in the Uppaal simulator.

In the same context of testing, [21] presents an approach for testing of software intensive safety-critical products to validate the hardware-in-the-loop simulation of a safety-critical system, by executing test cases both in the control setting (lab) and on the real product (train). The process is intended to be used when certifying the simulation which is a necessary step in order to certify the complete system. In addition, in [22], the authors propose an extension of base-choice criterion used for testing software-based on its nominal choice of input parameters, which takes into account time as another parameter when generating and executing tests by defining the timed base-choice coverage criterion. In [23], the authors conducted a comparative study on the cost and effectiveness of tests that are manually written versus those that are automatically generated in the field of industrial control software, where strict requirements on both specification-based testing and code coverage typically are met with rigorous manual testing.

In order to explore the performance of deployed systems at runtime, [24] suggests a performance space exploration approach for inferring the worst-case user scenario in a given workload model. The goal of this work is to detect which configuration of the load model has the potential to create the highest resource utilization on the system under test with respect to a given resource so that performance tests can be run with that configuration. An exact and an approximate method are suggested and compared.

Finally, in [25] we propose a marker design and an algorithm to detect the markers under different ambient conditions, with a long range to be executed on embedded systems with low computational requirements. The proposed method reduces the existing problems in the state-of-the-art related to the use of different environments and conditions such as different distances or different illumination.

3.3 The MegaM@Rt2 Model and Traceability Management (MTM) Tool Set

The Model and Traceability Management (MTM) tool set aims at providing generic global model management and traceability capabilities, with a focus on the dedicated support for creating and using feedback loops between design-time and runtime models in the context of complex CPSs engineering. To this intent, the MTM tool set is composed of 5 different complementary tools supporting the Eclipse [26] and Modelio [27] technical modeling environments. These tools provide support for storing and handling large EMF models (NeoEMF) [28], building and handling views integrating different EMF models (EMF Views) [29], keeping consistency and traceability between different EMF models (JTL) [30], detecting and refactoring performance antipatterns (PADRE) [14] or organizing and managing Modelio-based models and their relationships (Modelio Constellation) [31]. In all cases, their main objective is notably to leverage the different kinds of models resulting from the System Engineering and Runtime Analysis tool sets, in order to handle and reuse these models altogether in a coherent way as part of the continuous CPS engineering approach promoted by MegaM@Rt2. During the first phase of the MegaM@Rt2 project, a significant research effort has been conducted by the involved partners in order to provide these fundamental capabilities via the various tools of the MTM tool set. We summarize significant related research achievements in what follows.

On one hand, we have worked on improving the general support for backward traceability and change propagation between different kinds of models thanks to the JTL tool [32]. We then used such a support in order to provide change propagation capabilities at architectural (design) model-level, and illustrated it in a software availability context [33]. We also used this same support in order to automate performance improvements via the detection of architectural antipatterns using PADRE and thanks to traceability with corresponding runtime data [15] (cf. also Sect. 3.1).

On the other hand, we have obtained interesting results in the model view area [34]. Notably, we have worked on supporting the creation and handling of scalable model views combining different large-scale models together (including design and runtime ones) via traceability links [35]. To this intent, we worked on providing the required infrastructure to store, handle and trace efficiently very large models. This has been implemented in practice by leveraging the EMF Views and NeoEMF tools from the MTM tool set. This was a required achievement in order to be able to implement runtime-to-design time feedback loops, which is one of the longer-term objectives of MegaM@Rt2.

Interestingly, based on the two complementary efforts above-mentioned, we have then been able to apply our model view approach - EMF Views, in combination with our traceability capabilities in JTL, in order to provide a first concrete instantiation of the MegaM@Rt2 runtime-to-design feedback loop in the context of a safety-critical system from our partner ClearSy [36]. In the second phase of the project, we plan to work on more practical instantiations of such a feedback loop by relying on tools from the MTM tool set.

Nevertheless, there are still open challenges in these promising research areas. We have already been able to discuss that within the Modeling community when organizing and running the first edition of the International Workshop on Model-Driven Engineering for Design-Runtime Interaction in Complex Systems (MDE@DeRun 2018), co-located with STAF 2018 in Toulouse, France [37]. Notably, we identified challenges related to the particularities of design-runtime traceability: e.g. which semantics has to be given to the traceability information, in which contexts and how? We also identified questions related to the analysis of the traced runtime information: e.g. what kind of runtime data is actually needed, in which contexts and how to collect it properly? Finally, we identified issues related to the overall objectives of such a design-runtime traceability: e.g. which engineering purposes or activities do we intend to address or cover thanks to such feedback loops?

3.4 Case Study Evaluation

A total of nine industrial case studies are used in the project in order to evaluate the MegaM@Rt2 framework in practice. To provide measurable evidence on the extent to which the framework fits and provides benefits to the industrial development process, each case study defined a set of Key Performance Indicators (KPIs) that have been measured at baseline (i.e. when the project started) and have been/will be measured again after each of the two development phases of the project (i.e. at month 24 and month 36 respectively). At the time of writing this paper, the first evaluation phase has recently finished (at month 24).

During Phase 1, the case study providers have evaluated different scenarios using the tools and technologies offered by the different tool sets previously presented in this Sect. 3. The evaluation in terms of scenarios has put the focus on the benefits that MegaM@Rt2 is expected to bring: (1) They allow to better understand the aspects that the case study providers found most important for their industrial activities and (2) They structure and organize the tools’ verification and validation, which are based on the requirements and the KPIs defined by MegaM@Rt2. The case study providers made some changes in the choice of the best scenario to validate a tool/technology and, conversely, in the judgement of the best way for using a tool in a certain scenario; many problems were encountered but solved thanks to the collaboration that the tool providers fully offered.

The details on the case study results are provided in deliverable D5.5 as available from the project website [38]. It is important to mention that case studies measure differently the KPIs depending on their respective contexts and designed experiments. Nevertheless, it is noteworthy to point out that the case studies succeeded to demonstrate improvements significantly above targets, in particular, in:

  • Time required for identification of design problems;

  • Time/effort for requirements validation;

  • Productivity improvements;

  • Cost savings for development and maintenance of large complex systems.

The project has also already demonstrated values close to the targets for the following set of KPIs:

  • Reduction of validation effort by automated trace collection and analysis;

  • Reduction in time/effort required for tracing and handling all involved models at design and runtime levels;

  • Quality improvement by improving predictability and conformance to specifications.

4 Conclusions and Future Work

With a total of 40 deliverables and multiple tools provided via three complementary tool sets [39], MegaM@Rt2 aims at improving the productivity and quality of the system development and at reducing the time-to-market for complex systems, as well as to reinforce the European scientific and technological leadership and competitiveness of the European market.

The project has already delivered a significant number of research approaches, technical tools and methods spanning from system-level modeling to runtime analysis and global traceability and model management. While the results are globally evaluated as substantial, we face several open challenges towards our goal for scalable and traceable model-driven engineering applicable to a variety of industrial domains. The first phase of the project put in place the baseline methods that were assessed in the industrial settings. We demonstrated the opportunities brought by the global traceability and model management technologies resulting from research activities. In the meantime, we identified several further challenges. One of them is the need for a common runtime trace format, i.e. a shared representation for different types of runtime (meta)data. Another challenge is the need for more automated inference methods that could systematically relate these runtime traces (uniformly represented/modeled) to the corresponding system design artifacts.

Therefore, during its last period, the project plans to concentrate on those open areas by scheduling dedicated activities such as hackathons, demonstration session and workshops. We would like to engage the project community, both case study providers and technology providers to focus on a common agenda that would push the state-of-the-art further (in these areas, but also in others of interest to the project). Moreover, we plan activities to create awareness about the approaches and technologies developed in the project, which have already been adopted and endorsed by the industrial partners. Finally, an important aspect is about planning and preparing for the sustainability of the project results by creating an ecosystem for all the tools and methods composing the MegaM@Rt2 framework.