Keywords

1 Introduction

Ontology engineering is a branch of knowledge engineering that focuses on ontologies construction. It contemplates the study of ontology develop process, its life cycle, methods and methodologies to design ontologies, as well as the tools and languages for its construction. Before developing an Information System based on Ontologies, it is advisable to evaluate quality of an ontology design. The OntoPAA ontology [15] is used as an application case. The method is integrated by a set of techniques that group in six categories: minimalist, coherence, flexibility, standardization, redundancy and efficiency. The method is based on the design principles proposed by Gruber [20], Barry Smith [4], and Morbach, Wisner and Marquardt [22].

1.1 State of the Art

Since 1998, Guarino [11] defined term Information System based on Ontologies. Some authors such as Guarino [11], Colomb [17], Soares and Fonseca [3], Yildiz and Micksh [5] agree that the ontologies used in Information Systems contributes an improvement in the applications developed. For this reason, it is important to have a mechanism to evaluate the quality of an ontology design.

Literature on the ontology evaluation is fragmented, there are approaches that address specific evaluation issues, but not in a systematic way. Hartmann [10] introduces itself in the problems by providing a classification of network of ontological evaluation methods that allows to present methods in terms of structure, function, application, users types, and usability, among others. Relevant authors such as Porzel and Malaka [18], Gómez-Pérez [2], and Noy [13], proposed different methods to measure ontologies. Other authors made proposals associated with the evaluation of the quality of an ontology, for example, Tartir [19] presented a proposal based on metrics related to the ontology schema and the knowledge base; while Guarino and Welty [12] identify the problem areas that should be examined for rigidity, identity, unity and dependence in their work of OntoClean.

On the other hand, Gangemi [1] proposed three types of evaluations: 1) functional, it is focused on verifying that ontology meets its objective; 2) usability, it analyzes metadata and annotations; and 3) structural, that validates the structural properties of the ontology as a graph.

Other authors proposed methodologies that use and develop tools to support the ontology evaluation, for example, Corcho [14] proposed a tool called ODEval that automatically detects syntactic problems of ontology, such as cycles in inheritance tree of classes, inconsistency, incompleteness and redundancy. In 2005 Cross and Pal [21] integrated a plug-in in the editor Protegé that allows to evaluate the ontology quality based on the ontology definition and actual occurrences of ontological concepts.

Another approach is based on use of dimensions and metrics. Mostowfi and Fotouhi [7] proposed 8 metrics to evaluate an ontology, unlike other authors, they define a set of transformations to improve ontology quality. In addition, in OnQual, Gangemi [1] propose the ontology evaluation in three dimensions: the syntax and semantics of the ontology; the functional; and usability profiles, including 32 characteristics. Other authors approach the validation of ontology quality from point of view of Information Systems based on Ontologies Barchini [8], Fonseca and Martin [6], Colomb [17] and Colomb and Weber [16]. In general, they proposed dimensions and evaluation indicators to determine ontology quality level. As it is observed, there is a need to validate ontologies that are designed to identify points of improvement.

A proposal that contemplates some of principles of ontology design is the work of Barchini [9], author proposed 4 dimensions to evaluate operationally the ontology: a) descriptive, degree to which ontology provides information about its characteristics, meets a minimum ontological commitment, identifies the recipients, who is it?; b) structural, validated that the ontology expresses concepts explicitly, formal and consensual, associated with syntax and semantics, meets the specified requirements, what knowledge of domain contains? c) functional, valid if ontology does what end user intends; and d) operational, valid use capacity, can be used effectively.

Within structural dimension includes a sub-dimension called ontology in which it establishes indicators to evaluate the ontology design [8]. The main indicators are subdivided into classes and instances, relationships and axioms.

However, as we can noticed, there is a diversity in the criteria to determine characteristics that allow to evaluate ontology quality and way in which evaluation is done, most have an approach based on the components analysis (classes, relationships and properties). They not properly consider the design principles of ontologies. It is a priority that before realizing an Information System Based on Ontologies, to have the certainty that ontology is adequate and has quality.

In this context, this work proposes techniques for the evaluation or verification of the ontology design, based on ontology design principles of Gruber [20], Barry Smith [4] and Morbach, Wiesner and Marquardt [22].

2 Methodology

Methodology used starts with an analysis of design principles of three authors, then the principles are grouped into verification or evaluation collections to establish verification techniques: 1) minimalist verification; 2) consistency verification; 3) flexibility verification; 4) standardization verification; 5) redundancy verification; and 6) efficiency verification. Finally, techniques are applied to an application case, and the OntoPAA ontology [15] quality assessment is performed.

3 Principles of Ontology Design

To guarantee quality of ontology design, authors such as Gruber [20], Barry Smith [4] and Morbach, Wiesner and Marquardt [22] have defined quality criteria that guide the design and construction of ontology, this allows to evaluate quality of the design. The criteria also known as principles of ontology design, aim to design ontologies: reusable, easy-to-use, maintain and update over time. Reuse of an ontology refers to its ability to adapt to arbitrary application contexts, even in those not predicted at the time of its creation. Usability refers to effort required by a user to use ontology, its goal is to minimize the effort required and can be used by humans or machines under a specific application context.

Gruber [20] as part of his work proposes 5 criteria or design principles: clarity, coherence, extensibility, minimal coding tendency and minimum ontological adherence. On the other hand, for Barry Smith [4] is important that an ontology allows its adoption in the future, therefore, it emphasizes support for information exchange of the ontologies. This author proposes 14 principles for the design of an ontology: intelligibility, openness, simple tools, reuse of available resources, terminological moderation, intelligible definitions, terminological coherence, compound terms construction, instances types, non-circularity, singular nouns, consistency in use of operators for the terms construction, non-subjective definitions and non-redundant definitions. Finally Morbach, Wiesner and Marquardt [22] from construction of a enormous ontology of chemical domain called OntoCape, propose a set of recommendations to evaluate the quality of OntoCape, they applied following design principles: consistency, concise terminology, intelligibility, reusability, adaptability, minimum ontological commitment and efficiency.

The principles for the ontologies design proposed by the authors, evidence similarities between them, per contra, there are important contributions that one author considers and another not, due to the nature of the knowledge domain in which they have developed their work and so principles that they conceptualized are related to their domain. Gruber has developed his work in the field of Computing, Smith in the Biomedical area and Morbach in the field of chemistry and chemical engineering.

4 Verification Techniques of the Ontology Design

Based on the ontology design principles of authors described in the previous section, a grouping of principles is performed to determine verification techniques of an ontology design.

The design principles of first group associated with minimalist verification technique, which focuses on validating the compliance with minimum indispensable principles that must be met by ontology design, including: clarity, intelligibility, homogeneity, non-subjective definition, intelligible definitions, non-redundant definitions, compound terms, consistency in the operators use and documentation.

The minimalist verification technique proposes:

  1. 1.

    To axiomatise to greatest extent possible the formal definitions.

  2. 2.

    Use defined classes and bounded constraints.

  3. 3.

    Use a homogeneous style that facilitates understanding of new concepts.

  4. 4.

    Document ontology considering: a) comments within the formal specification of ontology; b) elaborate a reference guide oriented to application developers based on ontology; c) develop a user manual; and d) develop a design document for developers who will maintain and make updates to the ontology.

The technique of consistency verification, focusing on validating compliance with coherence principle that is proposed by authors analyzed. Proposes to use ontology publisher’s tools such as Protégé of Stanford University, to perform validation of syntax and logical consistency. There are reasoners such as Pellet, RacerPro, or FaCT++ with which more sophisticated consistency tests can be performed.

The flexibility verification technique, focused on validating compliance with principles of extensibility, personalization, openness and adaptability. Proposes to modularize an ontology in domains and sub-domains of application or conceptualization and that this facilitates adaptability, extensibility and personalization.

The standardization verification technique focuses on validating compliance with the principles of minimum coding trend, simple tools, and reuse of available resources. Proposes to use an ontology representation language that is standard, and is accepted by community. Where possible reuse and import ontologies that handle generic concepts such as time and measurements, among others.

The redundancy verification technique focused on validating compliance with the principles of concise terminology and terminological moderation. Proposes to use ontology editors such as Protégé that include mechanisms to perform the detection of redundant axioms, for example cardinality constraints that specify a minimum cardinality of zero. However, many of problems of redundancy in an ontology are caused by errors in the model design, to detect them a manual inspection must be done. Therefore, redundancies elimination in the design is achieved gradually, through continuous reviews and ontology re-engineering.

The efficiency verification technique focused on validating compliance with principles of minimum ontological commitment and efficiency. Proposes to comply with the concise terminology principle, which involves fewer axioms and is easier to process. The axioms number is one of many factors that influence efficiency. Also, type of axioms influences, some are more difficult to process than others.

A quantitative model is also proposed that admits evaluating each design principle considered in the verification techniques as shown in Table 1. Compliance with the principle implies the assignment of 1 point, if it is fulfilled to a lesser extent, a specific weight is determined according to with the covered.

Table 1. Quantitative model of minimalist verification techniques.
Full size table

5 Results: OntoPAA Application Case

In [15] an ontological model is designed for customization of learning activities called OntoPAA, this model is taken to evaluate its quality based on 6 minimum verification techniques. The objective of ontological model OntoPAA is to personalize learning activities of a course in such a way that student is assigned learning activities associated with his profile and act as motivators, and others that contribute to development of cognitive skills in accordance with the objectives of the course.

OntoPAA is composed of 4 ontologies: Profiles, Students, Courses and Learning Activities (see Fig. 1). Each ontology is independent of other in order to be able to reuse them.

Fig. 1.
figure 1

OntoPAA ontology [15]

Full size image

The Profiles ontology are constituted by attributes with information from the Cognitive Theory of learning and thinking styles. The Students ontology is integrated by attributes that allow to know student learning style, seeks to characterize the student. While in the Courses ontology the information of available courses is concentrated, dividing content of course into a maximum of 10 sub-themes, each sub-theme having associated learning resources and supporting tools. Finally, Learning Activities ontology, concentrates learning activities of the course for each profile.

It is intended to develop an Information System based on OntoPAA, so it is important to do the evaluation of it to avoid problems with the system development that are generated by the ontologies. Table 2 shows the quality evaluation of the OntoPAA ontology.

Table 2. OntoPAA quality assessment.
Full size table

When evaluating the ontology quality based on quantitative model, missing values are established so that it is clear on which points to work to improve as shown in Table 3.

Table 3. OntoPAA quality assessment.
Full size table

The quantitative model shows that the verification gives a level of 18.4/23. For this reason the areas where should work are shown in Table 4.

Table 4. Areas to work in OntoPAA.
Full size table

According to Table 4, the items with greatest problem have a value of −1, they are the ones that must be addressed first, in this case terms compound terms and efficiency. Furthermore, there is redundancy between Profiles ontology and Learning Activities ontology.

6 Conclusions

According to literature, there is a diversity in criteria to determine characteristics that allow to evaluating ontology quality and way in which evaluation is achieved, most of them have an approach based on the components analysis (classes, relations and properties), however, ontology design principles are not properly considered. The proposed model uses basic ontology design concepts, which in many cases are ignored and its impact is considerable in quality of ontology designed. In addition, it allows to validate ontology with the participation of knowledge engineer, knowledge domain expert and end user, by applying verification techniques: minimalist, consistency, flexibility, standardization, redundancy and efficiency.

The design principles of minimalist verification technique: clarity, intelligibility, homogeneity, non-subjective definition, intelligible definitions, non-redundant definitions, compound terms, consistency in use and documentation of operators, allow to validate compliance with the essential minimum principles in ontology design.

The minimalist verification technique considers in the first place to axiomatize formal definitions as much as possible, to use the defined classes and limited restrictions, as well as a homogeneous style that facilitates the understanding of new concepts, finally it proposes to document ontology considering: a) commenting on formal specification of ontology; b) develop a reference guide for application developers; c) develop a user manual and a design document for those responsible for maintaining and updating the ontology. Thus, it ensures that ontology design principles have been properly applied.

To establish the quality level of an ontology design, the assignment of points for each verification technique is contemplated, considering level 1 in case of complying with only 1 verification technique, while a level 6 will be obtained when complying with all of them.

In the case a checklist is shown that facilitates validation of ontology by applying verification techniques. The results show that OntoPAA has a level of quality level 6, as it complies with all verification techniques.

Although the evaluation of ontologies is not usually integrated in the methodologies for their construction, it allows to identify design errors before development of information system and therefore it should be one of the last steps to be carried out in process of designing an ontology.

There is a need for tools that facilitate the evaluation process, reasoners support consistency validation, however, there are many other variables to verify. As future work, the domains for which the verification technique is most suitable will be analyzed. As well as testing to expand its applicable principles.