Keywords

1 Introduction

Modeling plays an important role in representing and supporting complex human design activities [16]. Ontology-Driven Conceptual Modeling (ODCM), for instance, creates concrete artifacts representing conceptualizations of particular domains that support the understanding and communication among stakeholders. More precisely, ODCM involves the use of ontological theories to develop engineering artifacts visioning the improvement of conceptual modeling [2]. For example, the development of new conceptual modeling languages, the improvement of existing languages by adding structuring rules, and the proposition of conceptual modeling patterns and anti-patterns [16]. The use of ODCM can lead to various system engineering benefits such as increased reusability and reliability, sophisticated representation of the domain being modeled, and enhanced domain understanding among its modelers and users [16].

Nevertheless, the development, management, and usage of these artifacts require investments of resources, such as money, time, and workforce. Often, stakeholders neglected the trade-off analysis of the benefits and investments in ODCM experiences because of the difficulties in identifying and quantifying the inputs and outputs involved in it. By modeling experience, we mean the modeling initiatives that involve agents, events, objects, and goals related to the creation, use, and transference of models. Therefore, the analysis of the benefits and investments in ODCM experiences goes beyond the model, it is also associated with the modeling language, the modeling goals, the model designers, and so on. Although some authors had identified the need to reason about this cost-benefit relationship [4, 7] - called return on modeling effort (RoME) [11] -, very little work has been conducted to define and explain it.

Due to the involved investments, modeling initiatives should be done to solve specific problems and offering potential returns. Thus, before developing a method to assess ODCM experiences, we first need to reason about the value of these experiences. According to Sales et al. [13], “the value of a thing emerges from how well its affordances match the goals/needs of a given agent in a given context.” Therefore, value is a composition of benefits, which emerge from goal satisfaction, and sacrifices, which emerge from goal dissatisfaction [13]. One way to analyze the benefits and sacrifices related to an ODCM experience is through a quality assessment of the models it produces. This can be done via an analysis of the quality dimensions attended (benefits) or not (sacrifices) by the model and its process of development. Despite its importance, the notion of quality in domain modeling is still immature [10], as well as its identification and evaluation.

One example of how quality assessment can be used to analyze the value and the RoME of ODCM experiences is through domain debt (DD). DD is a new notion proposed by Störrle and Ciolkowski [15] in 2019 that means “the misrepresentation of the application domain by an actual system”. One reason for this flaw in the system in representing the domain can be the poor quality of the model. Domain debt caused by poor ODCM can require changes in the model that can affect other parts of the system, causing problems that are difficult and costly to solve. The efforts and investments made to solve these problems will directly affect the value and the RoME of the ODCM experience.

In this context, the core objective of this research is to develop a method to identify when worth investing in ODCM experiences based on an analysis of value. This new method can enhance enterprises’ decision-making processes by offering means to stakeholders to better assess and manage their investments in modeling experiences. To propose the method, we will develop and correlate knowledge and artifacts regarding the value of the modeling process and product, the return on modeling effort, and domain debt.

2 Related Work

Return of Modeling Effort - Guizzardi and Proper identified the need to more explicitly determine the purpose for modeling as well as to reason about RoME [4]. They proposed a taxonomy of modeling-related goals to reason about the purpose for which a model may be created in the context of enterprises. To develop the method to measure RoME, this research will also be based on the methods published in the field of return on investment (ROI) in modeling initiatives. For instance, modeling and simulation [12], data modeling [5], and building information modeling [3].

Quality Evaluation of ODCM - Some frameworks available in the literature address quality in the process and product of modeling. Two fit better with the purpose of this study, the Semiotic Quality (SEQUAL) proposed by Krogstie [8] for the evaluation of the quality of conceptual data models, and the Conceptual Modeling Quality Framework (CMQF) proposed by Nelson et al. [10] in defining the quality attributes of enterprise architecture models. However, both frameworks should be adapted to be used in the quality evaluation of ODCM.

Domain Debt - The term domain debt (DD) was coined in 2019 by Störrle and Ciolkowski [15] to represent technical debts (TD) related to domain-oriented design. In his book [14], Sterling detailed explains software debt, its causes, quality impacts, and management. Kruchten et al. [9] also reasoned about TD and its practices pointing to the need for more tools and methods to identify and manage different types of technical debts. Alves et al. [1] identified thirteen types of technical debt in their ontology of terms on technical debt. However, none of them were related to domain modeling debt.

3 Research Questions

The main question this research aims to answer is “When is it worth investing in ontology-driven conceptual modeling?”. Three subjects will be analyzed to answer this question: Quality Evaluation of ODCM, RoME, and DD. The knowledge of each subject will be developed and addressed according to the sub-questions presented below.

  • QR1: How do measure the return on modeling effort of ODCM experiences?

  • QR2: How to evaluate quality in ontology-driven conceptual models to assess the value of ODCM experiences?

  • QR3: How domain debt can be used to identify and quantify the value of ODCM experiences?

4 Research Methodology and Outputs

This research follows a Design Science Methodology [6] since it aims to create novel artifacts in the form of models and methods that will support people in addressing specific problems. The research tasks to be developed will follow the process detailed in Fig. 1. As shown in the figure, TSK 1.1 and TLS 1.2 have already been developed.

Fig. 1.
figure 1

Methodology detailing.

Full size image

Our first step was to conduct a literature review to analyze what is available about RoME and how it can be related to return on investment (TSK 1.1). In sequence, we developed an analysis of RoME based on the Common Ontology of Value and Risk (COVER) [13] to understand the value and value ascription of modeling experiences. Both analyses were combined with the taxonomy of modeling-related goals proposed by Guizzardi and Proper [4] to develop a study regarding the value, goals, and affordances that motivate modeling experiences (TSK 1.2). Our next step is to develop and apply an online survey to practitioners of conceptual modeling to analyze the state of the practice in domain modeling experiences (TSK 1.3). The results of the literature review, the value-based analysis, and the survey will generate the inputs to propose an ontology of the core components of RoME (RS1).

To understand how to evaluate quality in ontology-driven conceptual models we will develop a systematic mapping of the literature on quality in ontologies and conceptual modeling (TSK 2.1). In sequence, the plan is to develop a Delphi Study to get consensus among experts about the quality dimensions to evaluate ODCM (TSK 2.2). The quality dimensions to evaluate ODCM defined in the Delphi study will be used to adapt and complete the two existing frameworks: SEQUAL [8] and CMQF [10]. Then, a new framework will be proposed to assess the quality of the product and process of ODCM. The resulting framework (RS2) will be used as an input to study the relationship between domain debt and domain model quality.

We also plan to do a literature review of domain debt and the cost of technical debt (TSK 3.1). Then, we will develop and apply another online survey to people involved in projects that use domain models to analyze the state of the practice of domain debt (TSK 3.2). The last activity is to understand how domain debt affects the RoME of ODCM experiences. Therefore, we will develop an empirical study of repository mining to analyze the consequences of domain changes in real-world projects (TSK 3.3). The goal is to analyze the changes in an ontology-driven conceptual model related to domain debts and their impact on the RoME of the modeling experience. The development of these three tasks will result in a method to assess the consequences of DD (RS3).

Aiming to achieve the main objective of this research project, the final analysis will combine the outputs delivered throughout the project to propose a method for measuring RoME of ODCM experiences (RSF). To do so, first, we will correlate the framework on ODCM quality evaluation and the method to assess the consequences of domain debt to map the relationship between domain model quality and domain debt (TSK 4.1). In sequence, we will combine the knowledge gained about domain model quality and value, domain debt consequences and cost, and RoME’s components to refine, evaluate, and proposed a method for measuring the RoME of ODCM experiences (TSK 4.2).

5 Expected Results and Evaluation

RS1: An Ontology of the Core Components of RoME - The ontology will be a representation of the RoME domain encompassing the modeling experience and the modeling value ascription. It will be developed based on theoretical and empirical studies and validated by domain modeling specialists. We also plan to specialize the ontology using real-world examples.

RS2: A Framework to Evaluate Quality in ODCM - The framework will be a tool one can use to evaluate the quality of ODCM according to specific quality dimensions. By correlation, it can also assess the value of ODCM. The quality framework and its dimensions will be evaluated and validated through a Delphi Study with experts in ODCM. After the validation, we will apply it as a tool to measure the value of an ODCM experience in a case study developed later in the research.

RS3: A Method to Assess the Consequence of Domain Debt - This method will help identify the consequences of changes in the domain representation due to problems or mistakes in its modeling. It will consider the artifacts that depend directly and indirectly on the domain entities to estimate the effort needed to repay a debt. The method will be evaluated through a repository mining study in which the changes in the domain representation of a project will be analyzed and categorized. The ones related to problems or mistakes in the ODCM will be further investigated in a way that their consequences could be identified and, if possible, quantified.

RSF: A Method to Measure RoME of ODCM Experience - It will be the final and validated method to measure RoME of ODCM experiences; more than a formula, a complete value-based analysis method one can apply to identify when it is worth investing in ODCM experiences. This final method will be evaluated and validated through a case study in the same project used in the repository mining study. It will be a complete analysis of the investments, costs, and quality of an ODCM experience.