Keywords

1 Introduction

Ontology engineering is a field that concerns the ontology development process (i.e., planning, designing, building, and maintaining) and studies methodologies, methods, tools and languages to produce a well-engineered ontology [1, 2]. However, it is debatable whether a well-engineered ontology is fit for the intended purpose, in other terms, it can be questioned whether a well-engineered ontology meets the requirements of the considered use case. This can be ensured only through a rigorous quality evaluation across the ontology development. However, the existing well-known ontology methodologies such as Uschold and King’s method [3] TOVE [4], METHONTLOGY and On-to-Knowledge, have identified ontology evaluation as a phase of a methodology rather than considering it as an ongoing task which begins in the early stages of the developmental process. Moreover, these methodologies have not provided a detailed description of the quality evaluation. As a result, the attention to be provided for quality evaluation of ontologies got limited. Consequently, quality problems could not be detected until the ontology is used in an operational environment. Thus, in turn, it can adversely affect the overall quality of a system in which the ontology is integrated. Moreover, solving those quality problems requires an extra effort and not only that, it can be expensive and time-consuming. Thus, quality evaluation of ontologies cannot be underestimated.

As explained in [5, 6], quality evaluation requires a proper understanding of the intended purpose of the ontology and the domain to be modeled. Moreover, it is an iterative process that should be initiated at the requirements analysis phase and should be carried out in parallel with the development of the ontology. In analyzing the existing works, it has been identified that these notions have not been considered. Instead, the studies have discussed several other aspects such as quality scopes, evaluation levels, quality characteristics, tools, stages, approaches and techniques (see Fig. 1). To this end, we proposed an iterative methodology for ontology quality evaluation by analyzing quality theories in software engineering and also based upon the experience gained through ontology development and evaluation. Moreover, we made an effort to map the existing quality theories with the relevant steps of the proposed methodology (see Sect. 3). This, in turn, provides a better understanding of how the quality theories are getting associated with the quality evaluation process. Accordingly, the paper has been structured as follows.

Section 2 discusses the quality aspects: levels, characteristics, scopes, techniques, stages, tools and approaches considered in the previous works. Section 3 presents the proposed methodology for ontology quality evaluation. This has been further exemplified in Sect. 4. Finally, Sect. 5 concludes the discussed methodology by highlighting the future work.

Fig. 1.
figure 1

Overview of ontology quality evaluation.

Full size image

2 Overview of the Ontology Quality

Ontology quality has been discussed under several aspects such as (see Fig. 1):

  • Levels of ontologies: syntactic, semantic, hierarchy/taxonomy, architecture/structure, lexicon and context [7, 8]

  • Characteristics (i.e., criteria, principles): consistency, conciseness, completeness, accuracy, adaptability, clarity, etc. [8,9,10]

  • Scopes: conceptualization/structure, domain and technical/application [11, 12]

  • Evaluation aspects: structural intrinsic, domain intrinsic, domain extrinsic and application extrinsic [5, 6]

  • Techniques: data-driven, golden standard-based, human-based and application-based [7, 8, 11]

  • Stages in the ontology life cycle: analysis, design (i.e., conceptualization, formalization), implementation, deployment and maintenance [5]

  • Tools/methods [6, 13, 14]

  • Approaches/methodologies [10, 15, 16]

The following sub-sections discuss the aforementioned aspects in detail.

2.1 Ontology Quality Evaluation Levels

In evaluating an ontology, different levels namely syntactic, hierarchical, semantic, lexicon, architecture and context have been taken into account rather than evaluating ontology as a whole component [7]. For instance, under the syntactic level, ontology is evaluated to confirm whether it complies with the specification and syntax defined in the representation language. Under the hierarchy level, the properties related to the taxonomic structure (i.e., subsumption relationship) of an ontology are evaluated. Thus, it is also known as the taxonomy level evaluation. OntoClean [17] is one of the well-known methodologies that supports assessing the taxonomic properties such as essence, identity, and unity of an ontology. Under the semantic level, it is checked whether the ontology content is coherent with the domain knowledge that has been used to model the ontology. Typically, precision and recall measures have been used to assess the characteristics (i.e., semantic consistency, domain coverage and conciseness) related to the domain coherency [10, 15, 18]. In addition to that, the ontology taxonomic measures such as maximum depth, maximum breadth and structural variance have been adopted to assess the semantic level of an ontology [19, 20]. Under the lexicon level of an ontology, the vocabulary used to identify the ontology components (i.e., concepts, relationships, attributes, and individuals) is evaluated. The pre-defined representation principles are evaluated under the architecture level and the practical usefulness in the operational environment is evaluated under the context level [7, 8].

2.2 Ontology Dimensions, Characteristics and Measures

Various characteristics/attributes and measures have been proposed for ontology evaluation. Initially, the author in [3, 9] proposed five characteristics such as clarity, coherence, expandability, minimum encoding bias and minimal ontological commitment for ontology evaluation. Thereafter, the author in [8] has proposed another set of characteristics by associating them with the ontology evaluation levels (see Table 1). For instance, the characteristic: correctness has been defined to evaluate the lexicon and syntactic levels of ontologies. Later, the researchers in [21] have proposed a set of quantitative attributes to assess the quality of an ontology. This set of attributes can also be utilized to select a suitable ontology from the ontology repositories for the intended purpose. Based on that, the tools: OntoQA [21] and OntoMetrics [22] have been developed which support ontology developers to easily assess ontologies by avoiding the cost of manual evaluation.

Meanwhile, the semiotic metric suit has been proposed that consists of ten attributes grouped into four dimensions, namely: syntactic, semantic, pragmatic and social. Moreover, a set of measures to evaluate each attribute has been provided through that metric suit. Another comprehensive set of measures has also been introduced for ontology quality assessment in [23]. To this end, the attributes have been classified into three dimensions, namely: structural, functional and usability-related. Similarly, many researchers have proposed different characteristics, attributes and measures by grouping them into different dimensions (see Table 1). Therefore, we had to perform a comprehensive literature analysis to identify a set of characteristics that can be used for ontology quality evaluation [15, 24, 25]. Consequently, we proposed a quality model which consists of nineteen characteristics associated with different evaluation scopes [25]. Moreover, when analyzing the existing works, several discrepancies associated with the definitions provided for the characteristics, attributes and measures were observed. To this end, we made an effort to provide definitions for each characteristic identified through the literature that has been further discussed in [24, 25].

Table 1. Ontology quality characteristics.
Full size table

2.3 Ontology Evaluation Scopes and Aspects

The researchers have discussed different scopes/aspects to be considered in quality evaluation. The authors in [11, 12] have presented the three main scopes, namely: conceptual scope, domain scope and technical scope. Conceptual scope evaluates whether the concepts associated with the taxonomy of ontologies are well represented. When comparing with the ontology levels, it can be understood that the quality of the taxonomic level is considered in this scope. The domain scope evaluates whether the ontology represents the domain knowledge required to accomplish the specified tasks in the considered use case. To this end, the authors in [12] have shown that the domain scope considers the evaluation of the lexicon and architectural levels. Moreover, the technical scope evaluates whether the ontology meets the specified application requirements which are required for ontology integration and application in practice [5, 7].

Similar to the evaluation scopes described just above, the ontology summit has also proposed a set of scopes (i.e., evaluation aspects) to be considered in evaluating the quality of ontologies [6]. However, they have defined four main scopes, namely: structural intrinsic, domain intrinsic, domain extrinsic and application extrinsic.

The domain extrinsic and application extrinsic consider the evaluation of domain requirements and application requirements of an ontology which are specifically needed for a particular application respectively. These evaluations are very much similar to the “black-box” or “task-based” testing in software engineering [29]. Thus, the quality assessment is performed without peering at the ontology structure and the content. This has also been defined as ontology validation in [5, 8]. In general, the quality under the extrinsic aspect is determined by analyzing the correctness of the answers provided to competency questions with the involvement of users and domain experts [4].

The structural intrinsic and domain intrinsic scopes focus on the ontology structural and content quality evaluation respectively. In the structural intrinsic scope, the focus is given to the syntactic and structural requirements (i.e., syntactic, structural, architectural levels) which involve the specified conceptualization such as language compliance, conceptual complexity, and logical consistency. In the domain intrinsic aspect, quality is evaluated with reference to the domain knowledge which is used for the knowledge representation. At this stage (i.e., in the intrinsic aspect), mainly, the evaluation is performed by ontology engineers by considering the ontology as an isolated component separated from the system [5]. Somehow, ontology engineers need to obtain the assistance of domain experts when evaluating the domain intrinsic aspect as it requires some domain understanding. As we understood through the literature review, the semantic, vocabulary and architectural levels of ontologies are evaluated under the domain intrinsic aspect. Furthermore, the quality evaluation performed under the structural and domain intrinsic aspects are similar to the paradigm of “white-box” testing in software engineering [29]. Thus, the verification is being done under the intrinsic scope to ensure whether the ontology is built in the right way [8, 29].

2.4 Ontology Quality Evaluation Techniques

There are various techniques for carrying out a quality evaluation. The most commonly discussed techniques are human-based evaluation, data-driven evaluation, golden standard-based evaluation and application-based evaluation [7]. These techniques can be selected based upon the purpose, characteristics, scope, and/or levels to be evaluated of an ontology [24, 30]. For instance, the human-based technique is performing the quality assessment with the intervention of domain experts and/or users, and all the ontology levels can be assessed using this technique. In contrast to that, the data-driven evaluation assesses the ontology using a valid corpus and it is more suitable for ontology evaluation when it is difficult to acquire domain experts. However, this technique typically is used to evaluate the vocabulary and semantic levels of an ontology. When considering the golden-standard-based technique, it uses a standard ontology for the quality assessment and also it can be used in evaluating the levels: vocabulary, structure and semantics. However, the main difficulty of this technique is to find a standard ontology that has the quality at an acceptable level for the specified use case. The application-based technique is used to evaluate an ontology in an operational environment after it is integrated into the application. Thus, this technique can be adopted to evaluate the practical usefulness of an ontology and therefore, is suitable for assessing the context level of an ontology.

2.5 Ontology Evaluation Across the Ontology Life Cycle

Ontology quality could also depend on the success of activities that are followed to develop an ontology. For instance, if the requirement specification of an ontology is poorly performed at the requirement development phase, then, the resulting ontology will not succeed in providing knowledge to the specified tasks. To this end, the authors in [5] have introduced a comprehensive set of criteria that can be used to evaluate the activities performed under each stage (i.e., requirement development, ontological analysis, ontology design, system design, ontology development & reuse, system development & integration, deployment and maintenance) of the ontology development.

In addition to that, the author in [31] has performed a comprehensive survey and analyzed quality criteria that can be used to evaluate ontologies at the design and implementation stages of the development. Consequently, it has been identified that a set of possible criteria such as accuracy, adaptability, cognitive adequacy, completeness, conciseness, consistency, expressiveness and grounding can be used for evaluating the design of an ontology. Moreover, the criteria: computational efficiency, congruency, practical usefulness, precision and recall, can be used to evaluate ontologies at the implementation stage. Nonetheless, as shown by the author in [31], the previous works have used only a few characteristics from these to evaluate the ontologies at each stage. Mainly, there is a piece of evidence in using the characteristic: functional completeness (i.e., expressiveness) and practical usefulness frequently in the previous works. To this end, the authors in [15, 31] have pointed out the necessity of introducing a complete methodology or approach for ontology evaluation in order to avoid the quality evaluation getting limited to a certain set of characteristics.

2.6 Ontology Quality Evaluation Tools

Appropriate tools can be utilized to make the ontology quality evaluation process easy, efficient and cost-effective. In the previous studies, many tools have been introduced and a few of them are summarized in Table 2. In analyzing the tools, it can be realized that a number of tools are available for evaluating the syntactic and structural properties. The tools: S-OntoEval, RepOSE, DoORS and OntoKeeper support to assess the characteristics in structural and domain intrinsic aspects. Nevertheless, it can be observed that only a few tools are available to use online such as RDF validator, OWL validator, OOPS!, OntoMetrincs, DoORS and RepOSE.

Table 2. Ontology tools and methods.
Full size table

2.7 Ontology Quality Approaches and Methodologies

It is vital to have approaches and methodologies that systematically describe theories to be used, processes to be followed and the steps to be carried out in doing some work. With respect to the ontology quality evaluation, a few contributions can be found, such as the ROMEO methodology [10], the Two-Fold quality assurance Approach [16] and the Wilson et al. approach [15]. ROMEO methodology provides a set of guidelines to identify the intrinsic ontology quality characteristics from the defined ontology quality requirements of a system. Mainly, it has been constructed by employing the Goal-Question-Metrics (GQM) paradigm in information systems [46]. The Two-Fold quality approach has been introduced to monitor and assess the quality of an evolving knowledge base. It consists of two main phases, namely: coarse grain analysis and fine-grain analysis. During the coarse grain analysis, a quantitative analysis is performed to detect the high-level changes and common quality issues in a knowledge base. On the other hand, fine-grain analysis is performed to detect the detailed changes in a knowledge base and to detect all possible quality issues. The Wilson et al. approach mainly presents the steps to be carried out in determining quality characteristics from the intended needs of an ontology and presents how the identified characteristics are evaluated. Similar to the ROMEO methodology, this approach is also based on the GQM paradigm. However, the Wilson et al. approach supports to separately identify both intrinsic and extrinsic quality characteristics considering the ontology evaluation aspects (i.e., scopes).

3 Ontology Quality Evaluation Methodology

Although quality evaluation has been considered as a phase in most of the ontology methodologies, it is an iterative process that consists of several activities such as; identification of intended needs, elicitation of quality requirements from the identified needs, prioritizing the quality requirements, specifying quality characteristics (i.e., intrinsic and extrinsic aspect) and performing quality assessment across the ontology development [5, 47]. Moreover, these activities should be carried out in parallel with the development of the ontology starting from the stage of requirements analysis. However, there is no proper methodology that provides a set of steps that need to be followed under the quality evaluation of ontologies. To this end, we propose a methodology for the ontology quality evaluation by examining software quality theories described in SQuaRE (Systems and software Quality Requirements and Evaluation) [48] and also based on the experience gained through our previous studies [15, 24]. Accordingly, the proposed methodology consists of four main steps, namely: quality requirement specification, plan & design, execution and user acceptance test (see Fig. 2). These steps should be followed in parallel with the steps in the ontology development as illustrated in Fig. 3. The outer circle of Fig. 3 presents the steps in the ontology development life cycle as defined in [5]. The inner-circle presents the steps to be followed through our methodology which are further described in the below subsections.

Fig. 2.
figure 2

Ontology quality evaluation methodology.

Full size image
Fig. 3.
figure 3

The quality evaluation stages associated with the ontology life cycle.

Full size image

3.1 Quality Requirement Specification

Under this stage, mainly, the quality requirements to be achieved through an ontology should be identified. It is noteworthy that quality requirements have been also defined as quality characteristics [49]. Thus, hereinafter, we use the term quality characteristics to refer to the term quality requirements. To identify the quality characteristics, initially, it is required to recognize,

  • Who are the intended users?

  • What is the intended purpose of using an ontology/ontology-driven system?

  • What is the context of use?

  • What are the intended needsFootnote 1 in the considered context (i.e., use case)? What are the competency questions?

  • What is the scope of the ontology? What set of Competency Questions (CQs)/user needs will be covered through the ontology?

  • What are the resources available for the ontology quality evaluation (i.e., text corpora, documents, domain experts, other related ontologies, users, tools, budget, time)?

Generally, ontology engineers identify the aforementioned factors at the requirement analysis phase in ontology development [5]. Thus, they can be considered when specifying the quality characteristics during this step. To this end, an appropriate approach/methodology can be followed. For instance, the Wilson et al. approach [15] can be used which explains how the characteristics for evaluating each aspect (i.e., intrinsic and extrinsic) are derived from the intended needs. In addition to this approach [15], the ROMEO methodology can also be used. However, it only supports deriving the characteristics associated with the intrinsic aspect of an ontology [10]. Moreover, quality models are required for the mentioned approaches that support determining the characteristics. This is due to the fact that quality modelsFootnote 2 present a possible set of characteristics that are applicable for an artifact evaluation. In software engineering, ISO/IEC 25010 defines a quality model that supports specifying the characteristics required for the considered software product [48]. Moreover, it describes a set of measures that can be used to assess quality characteristics. However, for ontology engineering, there are no such agreed quality models that can be used for the quality requirement specification. Instead, a number of characteristics/metrics proposed in the previous works can be seen such as characteristics/metrics proposed in [3, 8, 23, 26, 43]. Nevertheless, it is difficult for researchers and developers to analyze all these existing works and to identify a proper set of quality characteristics. To this end, we constructed an ontology quality model after performing a comprehensive literature analysis that presents nineteen main characteristics [15, 25]. Meantime, it describes a set of measures that can be used to assess characteristics. Therefore, this model can be utilized when following any of the aforementioned approaches to specify quality characteristics.

The followings should be the outputs of this stage.

  • Specification of ontology quality requirements (i.e., quality characteristics)

  • Specification of quality measures

3.2 Plan and Design

After specifying the quality characteristics, evaluators (i.e., ontology engineers, and curators) can identify the related ontology levels or scopes (see Fig. 1) in which the characteristics are associated. Moreover, they can further identify the measures to be used to assess each characteristic under each level with the support of a quality model. Also, they can define the decision criteriaFootnote 3 for the selected measures. Furthermore, the evaluators can select the tools to be utilized, methods and techniques to be applied in order to assess the measures derived for the characteristics. For instance, assume that functional accuracy has been identified as a quality characteristic for an ontology in a particular context. Then, it can be derived through the approaches [10, 15, 18] and the quality model [15, 25], that external consistency, internal consistency and syntactic accuracy are a set of quality characteristics that are to be achieved through the ontology for functional accuracy. Meantime, the levels and scopes in which the characteristics are associated can be identified with the support of approaches. Table 3 represents the levels and scopes related to the characteristics that we have identified. Moreover, it includes the related measures, tools/methods, and techniques that can be used to assess the characteristics. When selecting techniques and tools/methods, it is required to analyze the resources that are available for the evaluation. For instance, external consistency is “the degree to which an ontology (i.e., ontology definitions) is coherent with the specified domain knowledge” [15, 25]. To this end, a frame of reference is required to check the domain coherency. A frame of reference could be a text corpus, domain experts and a standard ontology. If we have a valid text corpus to check the domain coherency, then the data-driven evaluation techniques can be utilized to assess the external consistency. On the other hand, if the domain expert intervention is readily available to the evaluators, then the human-based techniques can be easily carried out to evaluate the characteristic. Similarly, for all the relevant quality characteristics, the factors: scope, tool, method and techniques should be identified and shall be documented under this stage.

Table 3. The quality aspects related to the selected user requirement: functional accuracy.
Full size table

The following should be the outputs of this stage.

  • Specification of the selected quality measures, tools, methods, and techniques to be used in the quality evaluation.

  • Specification of decision criteria for ontology quality measures.

  • Specification of resources available for the evaluation.

  • Specification of detailed quality evaluation plans including time and budget [48].

3.3 Execution

During the execution stage, the selected quality measures of the characteristics shall be applied to the developed ontology or ontology being developed to check the required quality characteristics are achieved. In other terms, both white-box testing (i.e., domain intrinsic and structural intrinsic) and black-box testing (i.e., domain extrinsic and application extrinsic) shall be performed under this stage. Then, the results of the measures shall be reported and reviewed using the decision criteria. Consequently, any specific deficiencies can be identified with regard to the quality requirements. Thereafter, those deficiencies can be informed to the ontology developers to take the required actions to address them. Moreover, any limitations, constraints and exclusions in an evaluation can be reported including their impact on the use [48].

The following should be the outputs of this stage.

  • Results of the ontology quality measures associated with the characteristics.

  • Report of the limitations, constraints and exclusions in an evaluation.

3.4 User Acceptance

Under this stage, an ontology shall be evaluated in order to ensure whether the ontology meets the intended needs. In the case of an ontology-driven system, the ontology shall be assessed with the intervention of end-users of the system. To this end, the criteria and techniques defined for the acceptance testing in system and software engineering [48] can also be adopted due to the ontology will not appear as an individual component to the end-users at this level. In the case where the ontology is deployed as a standalone reference ontology, the appropriate application-based techniques can be used to observe whether the intended needs in an operational environment are achieved through the ontology [5, 7, 18].

Through this stage, the intended needs that are not covered through the ontology can be detected. These needs may be (i.) a set of new requirements for the ontology or (ii.) the requirements that have been identified during the requirement analysis, but, have not been addressed through the development. In these cases, the missing requirements shall be reported to the development team for further action. Consequently, the quality requirement specification can be refined and the methodology can be repeated.

The following should be the outputs of this stage.

  • Results of the user acceptance test, i.e., new user needs, user feedback, comments and suggestions.

4 Application

Quality Rquirement Specification:

to exemplify the methodology that we have proposed, we consider a use case in the agriculture domain explained in [15, 50]. In Sri Lanka, agriculture is one of the main industries, of which, the farmer is the main stakeholder who struggles to access the right information at the right time to make the right decisions. To address this issue, the requirement of producing an agricultural ontology and the intended needs to be achieved through that ontology have been identified in [50]. With respect to our methodology, we initially identified the answers to the questions which are highlighted in Sect. 3.1 and have been summarized in Table 4. A few of the identified intended needs were selected for the explanation as given below.

  1. a)

    Users need the necessary and sufficient contextual information.

  2. b)

    Users need trustworthy information.

  3. c)

    Users need information in an understandable way.

  4. d)

    Users need up-to-date information.

These needs are a set of inputs for the quality requirement specification in the proposed quality methodology. We illustrate how the quality requirement (i.e., characteristics) can be specified considering only the first requirement (i.e., a) in order to maintain the simplicity of the explanation. However, Table 4 summarized the quality characteristics which are associated with the other needs specified from b to d.

To identify the quality characteristics which are expressed in the user need: “a”, we employed the Wilson et al. approach [15]. Accordingly, we formulated the questions by considering the need mentioned below.

Q1: Does the ontology provide contextual information in a specified context of use?

Q2: Does the ontology provide necessary and sufficient information in a specified context of use?

With the support of the ontology quality model in [15, 25], we identified that the mentioned questions describe the characteristics: relevancy and completeness of ontology information respectively. By further subdividing these questions, the ontology characteristics to be achieved at the domain intrinsic and structural intrinsic levels were identified. For example, the characteristics: conciseness and compliance have been identified as characteristics to be satisfied at the intrinsic level (i.e., domain, structural) to achieve relevancy (Q1). To achieve completeness (Q2), the required internal ontology characteristics have been identified as coverage and compliance. Similarly, the ontology characteristics associated with the other user needs can also be identified. Due to the page limitation, we have not described in this paper the steps in detail which are to be followed in deriving the quality characteristics and further detail is available in [15]. Accordingly, Table 4 presents all the related quality characteristics derived from the mentioned needs.

Table 4. Summarization of the quality requirements.
Full size table

Plan and Design:

During this stage, it is required to plan and design how the specified quality characteristics shall be evaluated. Accordingly, we identified the measures that can be used to assess the specified quality characteristics (i.e., relevancy, completeness, conciseness, coverage and compliance), the tools, methods, and techniques based upon the available resources (see Table 5). In selecting the measures for the characteristic evaluation, we utilized the quality model that we have constructed in [15, 25]. To evaluate the structural intrinsic characteristics, the available tools were explored and selected such as OOPS! [40], OntoMetrics [22], Protégé [51] (see Table 5). Moreover, we defined the data-driven techniques to measure the characteristics: external consistency, conciseness, comprehensibility and coverage as we have the documents provided by the domain experts. To assess the domain extrinsic scope, the human-based techniques were defined as the domain expert and user assistance can be obtained for the considered context. To this end, the unit test discussed in [52, 53] can also be performed by maintaining test cases. Thus, in that case, all the required test cases should be documented and then, they can be used during the test execution. Similarly, the appropriate evaluation measures, tools, methods, techniques and scope are required to decide during this stage in order to use in the quality execution phase.

Table 5. Quality characteristics and the related evaluation aspects. *Note: E denotes extrinsic characteristics, I denotes intrinsic characteristics, O denotes a modeled ontology, and F denotes a frame of reference.
Full size table

Execution:

As planned and designed in the previous stage, the quality evaluation can be performed in this stage. According to the example, during the intrinsic level, the structural intrinsic characteristics were evaluated using the selected tools such as OOPS! [40], OntoMetrics [22], Protégé [51] and reasoners [54]. The characteristics in the domain intrinsic scope were assessed manually using data-driven techniques. Finally, the characteristics in the domain extrinsic scope were evaluated using Protégé by running the SPARQL queries defined for the CQs. In this case, the answers produced to the CQs were validated using the documents provided by the domain experts and obtaining their support. After assessing the identified measures using the selected methods and techniques, the results of this evaluation were reported. The evaluation results related to this example, i.e., the use case in agriculture, can be found in [15].

User Acceptance Test:

In the considered use case, an ontology is used as a component of a decision support system. Therefore, the user acceptance test can be performed by giving the ontology-driven system to the end-users, i.e., farmers and agriculture inspectors. To this end, the quality in use criteria defined in ISO/IEC 25010 can be used to assess the effectiveness, efficiency and user satisfaction of the ontology-driven system [49]. Due to the main system being under development, the acceptance test in a real environment has not yet been performed for the use case. In the future, we expect to present a thorough result analysis including this step.

5 Conclusion

The usefulness of a methodology for ontology quality evaluation has been identified through theoretical and empirical reviews. As an initial step, we constructed a methodology by analyzing the theories in software engineering, i.e., SQuaRE [48, 49] and experience gained through the development and evaluation of ontologies. Consequently, the developed methodology consists of four main steps, namely: quality requirement specification, plan & design, execution and user acceptance test (see Fig. 2). These steps can be performed iteratively and parallelly with the ontology development life cycle. The applicability of this methodology for real applications has been exemplified by discussing a use case in agriculture. Moreover, the methodology has been introduced for undergraduate students who are currently doing research in ontology engineering. Most of the students provided positive feedback by highlighting that the methodology is useful for them to understand the proper set of characteristics to be assessed and how the quality concepts discussed in the literature are associated with it. To this end, we further expect to observe the results of the experiments using the proposed methodology in many use cases and to enhance it with a more comprehensive set of guidelines.