1 Introduction

Formal technical reviews of requirements are commonly undertaken to identify missing information, conflicting or unrealistic requirements Pressman [36]. This can be a highly time-consuming and an expensive process. Formalising the requirement models is a way to bring automation to this process. However, this tends to hinder the communication with the client and to reduce the quality of the models [29]. As a way of bridging the communnication gap between systems analysts and users, many authors have advocated the use of agent oriented methodologies to target various types of applications [23]. Those authors set the expectation of reduced communication errors, but they do not actually elaborate mechanisms for the validation of agent based requirement models.

Additional knowledge about the context of the system being developed can be used to cross-validate the correctness and completeness of the requirements models Saguna et al. [37]. To represent context knowledge, ontologies can be deployed [8]. Ontologies can also be used to provide an intermediate knowledge to support the generation of system design knowledge [25]. When used to prescribe system design knowledge, ontologies are often termed metamodels Henderson-Sellers [22]. Those latter observations on harnessing external knowledge sources to support systems development are heeded in this paper. We combine the reuse of two forms of knowledge in supporting the development process: system context knowledge and system design knowledge. The primary focus of this article is the provision of automatic support for controlling the quality of agent oriented requirement models.

Our approach harnesses the use of the system context knowledge through the use of a domain ontology. It also harnesses agent design knowledge through the reuse of a metamodel to ensure decoupling from a specific agent oriented development methodology. The metamodel describes the key constructs across agent oriented methodologies and provides a unified access to their analysis workproducts, as advocated in [33]. This metamodel describes the most common features of extant agent models. It is based on the FAML metamodel which is proved to adequately cover over 20 existing AOSE methodologies [6]. A validation and verification (V&V) process of informal requirement models commonly used in such methodologies is developed and tested. The V&V process described in this paper relies on the reuse of domain ontologies a ‘sounding board’ to check the semantics of the models produced during analysis. Any discrepancies between the ontology and models highlight potential errors that get addressed before they propagate any further. The V&V process intertwines with the development process, validating and verifying models as they spiral toward acceptance and completion. It is worth noting that we do not advocate the use formal languages to express requirement models from the client. We instead advocate an innovative reuse of domain knowledge to provide context knowledge as advocated. In combination with a template of easily accessible models, this enables us to create an automatic validation of agent informal requirement models.

The rest of the article is structured as follows: Sect. 2 elaborates the context of this paper and reviews existing work on uses of ontologies for the V&V of requirements models; Sect. 3 describes the research methodology followed. Section 4 presents the ontology-based V&V process; Sect. 5 discusses the mechanisms to automate the V&V process; Sect. 6 presents the tool which operationalises the mechanisms described in Sect. 5. Section 7 evaluates the tool in two development exemplars; Sect. 8 concludes this article and discusses future work.

2 Related work

Supporting the V&V of requirement models with conceptual models is not new e.g. see [19]. As they are closer to the effort in this paper, we restrict our discussion to works related to employing reuse to support the V&V of requirements. For example, in [27], the authors use OWL ontologies to verify that models are compliant with their metamodel. Unlike our effort, this does not provide a compliance check with the client’s specification. It only focuses on the correctness of the structure of the models. It also insists that the models are expressed in OWL. This condition is also common in other studies [2, 3, 38] which insist on models built in a specified language (e.g. OntoUML). The work here uses OWL to represent the ontology but the models themselves can be expressed in various notations. The proposal is independent of the modelling language, as the defined operators can be adapted to a range of models. This makes the proposal applicable in different development methodologies.

Pires et al. [35] propose an ontology-based requirements engineering process to control the volatility of requirements where users change their mind due to a changing environment. This process is iterative and incremental and uses an ontology to represent the domain and as a consistency assessment tool. In this paper, the focus is on requirements analysis rather than elicitation. This work encourages the involvement of a client in the analysis process. Whilst we do not explicitly restrict the natural language used by the client as advocated in the use of boilers or requirement patterns [1, 15], the agent requirement templates can be used to formulate guiding questions when interacting with the client. Given that agents and agency concepts are part of our natural language, focussing on them can facilitate subsequent communications with the client to review the requirements. This is a common motive for adopting agent oriented analysis of requirements. Given the scope of agent analysis models, to facilitate the lower level design of distributed agent-based systems, their semantic breadth and depth serve more than a collection of elicitation templates.

Goknil et al. [20] highlight the importance of requirements inter-dependencies to manage change propagation. They define a metamodel that describes the types of relations that may exist in requirements documents. Their metamodel is built on top of a layer of first order logic semantics that makes it possible to check requirements consistency and to infer new relations [39]. Also build an ontology defining a requirement metamodel for Goal Oriented Requirements Engineering (GORE). But each particular project would instantiate the domain independent metamodel. This infrastructure makes it possible to check the consistency and completeness of the requirements. Our work in this paper is different from [20] in at least three different ways: we define a metamodel to support the V&V of requirement analysis models, taking into account the client conceptualisation and inter-dependencies of the models. Both proposals define a metamodel of a development process that can be instantiated for particular projects and that underpins the requirements validation. However, the metamodel presented in this article defines the concepts and relationships used in the models of AOSE methodologies, while the metamodel in [20] models a general requirements specification (mostly textual descriptions) for GORE. While the consistency rules are similar in both cases, the completeness rules developed as part of this research are different.

None of the extant AOSE methodologies, to our knowledge, support the validation of the requirement models against the domain problem as conceptualised by the client. Some include domain modelling using ontologies as part of their development process to support internal consistency of models. Examples of such methodologies are O-MaSE [13, 16], MESSAGE [17], PASSI [11], ASPECS [11], MOBMAS [44] and MAS-CommonKADS [26]. These essentially use ontologies only to represent the domain knowledge and the agent local knowledge: MESSAGE uses ontologies to provide context and input information for agent behavioural knowledge and reasoning; O-MaSE uses ontologies in agent communication to formulate exchanged messages; PASSI and ASPECS also use an ontology to specify agent communication to formulate messages. Finally, although MOBMAS hints that ontologies could be used to verify the structure of models, to support interoperability and reuse [5, 42–44], no explicit process towards that end is defined. An initial proposal towards such a process was made in Lopez Lorca et al. [28]. The proposal lacked details and did not take into account the possible need to modify the ontology during the process. This paper is the first to fully elaborate an ontology driven validation process. It also tests it in two exemplars. An agent metamodel is deployed to support the process. This formalises it and enables automatically annotating the domain ontology with corresponding agent concepts. This not only enables automatic support of the process but also supports the interoperability of the process across different agent oriented methodologies. Within the agent oriented software engineering community, our approach is closer to [32] which proposes a parallel testing with requirements analysis to avoid costly rework later. Researchers in [32] propose a process which generates test cases using the goal models. Cases are generated at varying levels of abstractions to accommodate various models for Tropos [30]. This approach relies on trusted goal models. In our approach, the goal models themselves can be validated and rechecked by using the domain ontology as an additional knowledge source. Furthermore, our process is designed to be methodology agnostic by relying on an agent metamodel. In some sense, our process integrates an agent-oriented metamodel with a domain ontology as proposed in TAGOOn+ [31]. However, the integration in TAGOOn+ is restricted to only integrating the organisational models. Our integration is at a higher abstraction layer (the agent oriented metamodel) to ensure all agent models are accommodated in the integration.

3 Research method

Our research objective can be stated as “development of a validation process of agent requirement models which is both domain and methodology agnostic”. This raises the following two research questions:

  1. 1.

    How do we exploit domain ontologies to develop appropriate validation operators for all given agent requirement models in a given development project?

  2. 2.

    How do we exploit methodological knowledge expressed in a generic agent metamodel to ensure method interoperability of the process?

To address the above research questions, the research method followed in this paper is design science. Hevner et al. [24] define two complementary activities in IS design science research: building and evaluation of the artefacts. Once the artefacts are built, they should be evaluated with respect to the utility provided in solving the identified problem. Further iterations of the refinement and evaluation are required until the artefacts can adequately solve the problem.

The aim of the research presented in this paper is to design a reliable MAS models V&V process. The synthesis of the novel process as such requires mitigating significant validity threats to ensure that the synthesis is both domain- and developer-independent. Furthermore, the process is required to be sufficiently general to be applicable in a multitude of AOSE methodologies. This threat of AOSE methodology specificity is an important one to mitigate as many researchers have proposed MAS development methodologies but none of these methodologies have, so far, emerged as an accepted standard. Our goal is to present a MAS models V&V add-on process that can be used with any of the extant MAS development methodologies.

Following design science research methodology, the research is organised into four phases (shown in Fig. 1):

  • Phase 1 focuses on defining the research problem, this was executed in Sects. 1 and 2.

  • In Phase 2, the V&V process that addresses the research problem is defined. This is described in Sect. 3 (next Section).

  • In Phase 3 an automatic mechanism to apply the V&V process is developed. This is described in Sect. 5. The tool that implements the process is described in Sect. 6.

  • Phase 4 evaluates the automated V&V process resulting from Phase 3. The evaluation activities presented will ensure that the three validity threats are mitigated against. Thus the research is designed to test and ensure all independence along three dimensions: developer-, domain- and methodology.

Fig. 1
figure 1

Design science Research phases in this paper

Full size image

4 The ontology-based verification and validation (V&V) process

In our process, a domain ontology provides a collection of connected statements about the context of the application. The ontology will be used to establish a common understanding between the developers and the client, as advocated in [18]. The ontology is automatically annotated with recurring agent concepts. Once annotated, the ontology can then be used as input to cross check validation operators. These operators can be seen as a collection of reusable requirements validation patterns c.f. [15]. An ontology is descriptive and articulating it imposes less restrictions on expressions or the dialog with the client. Indeed, in cases where a ready ontology is not available, an ontology can be obtained from the domain description or requirement documents furnished by the client. For example, in one of our exemplars, the plane turnaround application, much of the ontology content was developed from technical diagrams provided by the client. More fortuitously, ontologies can be acquired from a repository or a synthesized from a collection of relevant ontologies which can be automatically located as shown recently in Beydoun [4].

Our V&V process also acknowledges that the client’s conceptualisation of the system requirements may evolve as the development proceeds. Discussions between the client and the developer regarding the models under development can facilitate an improved understanding of the problem and can lead to uncovering new aspects of the system being sought. This can change the requirements specification and needs to be reflected in the models. The process acknowledges that the added requirements are not necessarily accommodated by the original ontology, and can lead to discrepancies with the original. Thus, the V&V process allows the ontology to evolve, thereby maintaining its consistency with the new version of the requirements specification. Any new requirements will also need to be incorporated into the ontology. Using the ontology directly in modelling would add complexity to the interactions with the stakeholders (the source of the requirements). This can become unpredictable as the stakeholders may be derailed from articulating their concerns. Hence, the automated process would cross check any inconsistencies as the models are developed. The software development lifecycle continues to be initiated and driven by users requirements. When the developer processes a discrepancy report, any discrepancies that have originated from a new requirement are resolved by updating the ontology to reflect the new requirements. With a problem specification as input, the beginning of the whole process is the ontology preparation. The ontology is then augmented with agent concepts and relations. The ontology is validated through consultations with the client readying it to verify and validate the informal models (in Task 2 of Fig. 2). Figure 2 describes the whole process which is detailed in the remainder of this section.

Fig. 2
figure 2

Ontology-based V&V Process for requirement analysis models

Full size image

4.1 Ontology preparation

(Task 1 in Fig. 2) initiates the V&V process and consists of three steps. It begins through elicitation techniques such as interviewing clients and/or acquiring any documents that can describe their business processes. This represents the client’s conceptualisation and it sets the basis for identifying a suitable ontology. This can be either sourced from an existing repository (which can be automatically located as shown recently in Beydoun [4]) or can be developed from scratch. Either way, at the end of the V&V process the domain ontology should be an accurate representation of the client’s specification. Communication with the client has to be initially intensive to ensure that the modelled domain is as detailed as required and matches the conceptualisation of the client. If the sourced ontology lacks details then its effectiveness in the validation or in providing modelling assistance to software developers is reduced. However, there is scope for later evolving the ontology as a result of detected discrepancies with the models.

The second step is to ensure that the domain concepts in the ontology are appropriately linked to concepts from the development/requirement analysis paradigm (Task 1.2 in Fig. 2). This requires identifying relations between the domain concepts using relation templates as defined by the development paradigm used. For instance, concepts that typically characterise the agent oriented paradigm include Goal, Role or Activity. Examples of relations between these concepts: (Goal1) subgoal of (Goal2), (Role) responsible for (Goal), or (Activity) achieves (Goal). Concepts and relations for the agent oriented paradigm will be further elaborated in Sect. 4.

Finally, Step 3 of Task 1 is Ontology Validation (Task 1.3 in Fig. 2). This step ensures that the ontology accommodates the conceptualisation of the client. The ontology is visualised as a simple directed graph with labelled arcs devoid of technicalities. In our experience, this type of visual structure is easily understood by non-technical stakeholders and facilitates immediate feedback. The overhead of this step on the whole process is minimal. To highlight this step, the wine broker study in Sect. 6 illustrates the application of the V&V process using a chosen weak non-validated domain ontology.

4.2 Models V&V

(Task 2 in Fig. 2) consists of two steps. The first compares the informal models received from a developer and produces a discrepancies report resulting from a comparison with the ontology (Task 2.1 in Fig. 2). The informal models are validated against the augmented ontology for consistency and compliance with the client’s specification. Elements of each model are compared against corresponding elements in the ontology. Results of this comparison determine the need for further modelling and validation iterations. A new iteration is necessary if any discrepancy is found between the ontology and models. These will be automatically discovered using comparison operators defined for a standardised set of informal requirements models. At the end of this task a report describing the discrepancies between artefacts is produced.

The second step guides the developer to remove the discrepancies (Task 2.2 in Fig. 2). This can be either by changing the models or modifying the ontology. All discrepancies need to be solved to finalise the V&V process. The discrepancies may occur due to any of the following four issues:

  1. 1.

    Discrepancies due to inconsistency These highlight different interpretations of the domain held by the ontology engineers and modellers. They may occur at any time and tackling them as soon as possible will prevent compound errors. To solve them, either the ontology or the model must be modified.

  2. 2.

    Discrepancies due to incompleteness of the models These highlight the possibility of details being missed from the models and are more frequent in the early stages of validation, when models are still evolving. As the models evolve, such discrepancies will naturally disappear. However, if these discrepancies are detected when the models are mature, they are deemed genuine errors by the developers and should be treated as such.

  3. 3.

    Discrepancies due to incompleteness of the ontology These may occur in later phases of the development. If the client conceptualisation changes, then the domain ontology may have to be revisited. The models may then reflect the clients conceptualisation better than the ontology and lead to its evolution.

  4. 4.

    Discrepancies for requirements elicitation These contribute to the completion of the models. They are not typically caused by developer errors. Rather, they occur because the ontology itself contains details that were not considered by the client. These should be presented to the client for approval.

After improving the quality of the models according to chosen recommendations, the new set of models will be used as input in subsequent iterations. Requirements analysis is repeated as part of each iteration along the sequence of work-products required by the chosen development methodology. Problems in reviewed models are fixed before they are fully developed. Any models yet to be commenced in any iteration will take full advantage of the recommendations, avoiding compounding errors.

A review of extant agent oriented methodologies showed that there is a set of common concepts and relationships across most of them [44]. In fact, the most common elements amongst notable agent oriented methodologies are the following: All considered ten agent oriented methodologies define system activities, agents and the interactions between them; 80 % of the methodologies identify the environment entities that interact with the agents; 70 % of them identify the roles that agents play; 60 % of them define authority relationships amongst agents; and half of them map agents with the goals that they pursue. This set of common concepts and relationships is also included in the generic metamodel for agent systems development that [6] synthetised after analysing over 20 agent oriented methodologies.

5 Automating the ontology-based V&V process

Automating the V&V process relies on three steps: Validation operators that cover common AOSE concepts and relationships in analysis are formulated; common AOSE concepts are formalised as a metamodel to enable automatic augmentation of the ontology. Finally, to classify each concept defined in the domain ontology into its corresponding requirement modelling construct, an OWL 2 DL automatic reasoner is used to classify concepts based on metamodel primitives. Each step is detailed in this section.

5.1 Synthesis of ontology-based V&V operators

As earlier discussed, the most commonly used 7 analysis models in extant agent methodologies are (in order of decreasing popularity): agent models, goal models, interaction models, scenarios, organisation models, role models and environment models. We overview each of them in what follows.

A Goal Model defines a hierarchy of goals that describe the purpose of the system. The root of the hierarchy is the main objective of the system to be developed [9]. Goal models are used by clients and developers to discuss the goals of the system and the roles required to fulfil them. Roles are typically defined and associated with goals in the context of goal models. The Role Model extends their description. In general, roles define who is responsible, or share with other roles the responsibility to fulfil goals [40]. The Organisation Model reflects the authority relations between the roles defined in the role model and can be useful to define the interactions between agents (later in the development process) [41]. An agent based system is always situated in some environment [45] which needs to be modelled to detail the resources needed by the system. The Environment Model describes each environment entity used by an agent in the system. Agent Models define the individual agent classes that will cover the roles defined in the system. Each role has to be mapped to (or played by) at least one agent. Enforcing the rule that each role is played by at least one agent helps with ensuring that the goals of the system are fulfilled DeLoach and Kumar [14]. An Interaction Model typically shows which roles participate in the activities of the system [34, 44]. This model will set the basis for the definition of protocol models later in the design phase of the system, and will detail the exchange of messages between instances of agents. Finally, Scenarios are detailed descriptions of particular example sequences of events associated with a particular goal or sub-goal of the system [34]. Their typical use begins with a high level scenario modelling of the main goal, and this then gets refined into scenarios for the sub-goals in greater detail.

We formulate operators that link the content of the models to the V&V domain ontology. These operators are described in Table 1. For each model, the columns Concept and Relation list the elements that should match in the domain ontology and the corresponding model.

Table 1 Ontology-based validation operators
Full size table

5.2 AOSE metamodel

To enable validation of the interconnections between the various informal analysis models from various methodologies, an AOSE metamodel is used. It will enable further identification of any missing or inconsistent links between and within the models. The metamodel used (shown in Fig. 3) is a subset of the FAML metamodel [5–7] which is a generic metamodel that describes most extant AOSE methodologies at analysis and design level. The use of the complete FAML metamodel is out of the scope of this work, as the use of all its features at analysis and design level would introduce an unnecessary complexity in the automation of the V&V process. The metamodel used uses OWL 2 DL formalisms to describe typical concepts of the AOSE paradigm as classes (boxes in Fig. 3), relationships are modelled as object properties (Fig. 3: the directed arcs between boxes, with the label being the name of the property) and axioms describe the constraints of the domain using classes and object properties (Fig. 3: the triples composed by two boxes, a linking arc and a cardinality).There are six concepts in the metamodel: Goal, Activity, Role, Agent, Event and EnvironmentEntity. The class Goal represents the objectives that will be achieved by the system. The class Activity represents the tasks that must be performed in order to satisfy certain goals. The class Role represents a stakeholder of the system that is able to perform certain functions necessary for the achievement of goals. The class Agent represents an entity in the system capable of performing the functions of one or more roles. The class Event represents a significant occurrence in the environment which activates an agent to pursue its goals, or changes the agent’s course of actions in achieving the goals. Finally, the class EnvironmentEntity represents any resource required to perform the system tasks.

Fig. 3
figure 3

AOSE metamodel

Full size image

The relationships described in the metamodel are modelled as object properties in the ontology. They have been defined by selecting a descriptive base word that describes their purpose, e.g. achieves, concatenated to a prefix (Domain) and a suffix (Range), e.g. ActivityAchievesGoal. The classes that are described by an object property are called Domain. The classes that characterise the individual state of the domain through the object property are called Range. The relationships between classes are described by means of the 27 axioms that they define. These axioms are constraints over each class that specify the governing rules of the problem domain. Examples of these axioms are: Each goal may be the subgoal of other goals; Each activity achieves at least one goal; Each role participates in at least one activity; Each role may control other roles; and each agent plays at least one role.

Some object properties are the inverse of others (represented in Fig. 3 by links between classes in opposite directions). An important aspect of the metamodel is the definition of transitivity for hierarchical structures. However, the limitation of transitivity to only one object property can limit its usability. A more versatile alternative to transitivity is the OWL 2 specification of property chains. This is a composition of several properties, where the object of an assertion becomes the subject of the next one. For example, the metamodel defines the property AgentPursuesGoal as equivalent to the composition of properties AgentPlaysRole, RoleParticipatesInActivity and ActivityAchievesGoal. Over 30 property chains have been defined by analysing the metamodel to discover which properties were always equivalent to the combination of others.

The distribution of concepts across models is slightly different across different methodologies. For a given methodology, elements used in the different models are identified at the beginning of the validation process. For example, in the ROADMAP methodology [40] role and goal models share the use of the concepts Role and Goal and the relation RolePursuesGoal. It can be said that both models are co-dependent in terms of these elements. The modification of any role in the role model must be propagated accordingly to the same role in the goal model.

5.3 Ontology augmentation

The augmentation mechanism is underpinned by the reasoning infrastructure of OWL 2 ontologies. The links established between the metamodel and domain ontologies enable the automatic classification of domain concepts, the consistency and completeness checking of the domain ontology. OWL 2, the ontology language currently recommended by W3C, fully supports automatic reasoning mechanisms. W3C defines two alternative ways of assigning meaning to OWL 2 ontologies. The first uses highly expressive RDF-based semantics. However, using these, it is not always possible to quickly establish the truth or falsehood of a logical statement. The second uses a subset of Description Logics (DL) to define semantics. This limits the expressive power but is much quicker to reason with. The key feature of DL to note here is the difference between Terminological (TBox) and Assertional Box (ABox). The TBox describes the concepts (or classes) and the relationships amongst them. The ABox describes the individuals that belong to the classes and their relationships. This distinction is important because DL reasoners perform different tasks at each of these levels. The elements of the metamodel and domain ontologies are defined at both levels, the TBox and the ABox levels (the automation is based on using Fact++ embedded in Protégé).

The domain ontology is built at the ABox level. All the elements are either individuals or object properties defined without any formalisms. The reasoner cannot detect inconsistencies or infer new knowledge in the domain ontology unless it is linked to the metamodel ontology (defined at the TBox level) via sub properties. Every property defined in the domain ontology can be expressed as a subproperty (i.e. a subtype) of the corresponding property in the metamodel. Whenever two instances are related through the subproperty, the reasoner can infer that they are also related through the superproperty. A consequence of this is that the subproperty also has the characteristics defined for the super-property, in particular the domain and range. The automatic classification of the domain ontology individuals is done through the analysis of the definition of the containing class (intentional definition) and the characteristics of the element to be categorised. The reasoner can highlight subsumption relationships that were previously only implicit. Assume for instance that it is stated in the metamodel ontology that the relation ActivityAchievesGoal always relates elements of the class Activity (domain) with elements of the class Goal (range). If the statement Unload baggage achieves Handle baggage is added to the ontology, as the property achieves is a subproperty of ActivityAchievesGoal, the reasoner will be able to automatically infer that Unload baggage belongs to the class Activity and Handle baggage to the class Goal. Properties in the domain ontology inherit all characteristics defined for their superproperties. This allows further inference of implicit knowledge and consistency checking. Important inherited characteristics are:

  • Transitivity: If A r B and B r C, then A r C. For example, the property GoalIsSubGoalOfGoal.

  • Symmetry: If A r B, then B r A. For example, the property RoleIsPeerRole.

  • Asymmetry: If A r B, then B r A is illegal. For example, the property RoleControlsRole.

  • Inverse properties: If r is the inverse of p and A r B, then B p A. For example, the properties ActivityFollowsActivity and ActivityPrecedesActivity are the inverse of each other.

  • Property chains: If r is equivalent of p composed with q, A p B and B q C, then A r C. For example, the property RolePursuesGoal is equivalent to the composition of activities RoleParticipatesInActivity and ActivityAchievesGoal.

The logic underlying the metamodel ontology supports the domain ontology consistency checking. For example, in the metamodel, the classes Goal, Activity, Role, Agent, Event and EnvironmentEntity have been defined as disjoint classes. This means that an individual belonging to the class Goal cannot belong to any other disjoint class. Assume that the domain ontology contains the assertions Load baggage needs Bulk cargo loader and Load baggage subgoalOf Aircraft turnaround. When the reasoner tries to classify the instances an inconsistency is triggered, as Load baggage has to be classified as an individual belonging to both classes Activity and Goal. This happens because the domain of the property subgoalOf is Goal and the domain of needs is Activity, therefore an individual participating as subject in both properties has to simultaneously belong to classes Goal and Activity, which is explicitly forbidden by the disjointness axioms.

Fig. 4
figure 4

Example of domain ontology

Full size image
Table 2 Domain and range of metamodel properties
Full size table

Using the metamodel, the domain ontology is automatically augmented to align with the requirement modelling constructs identified. The number of object properties defined in the metamodel is the upper limit for the number of properties relevant for the V&V process in the domain ontology. The number of concepts in the ontology will grow according to the complexity of the domain. But the number of object properties linking the concepts will remain constant. In the AOSE metamodel the characteristics of object properties—domain and range, amongst others—are already defined. Figure 4 shows an excerpt of a domain ontology for the aircraft turnaround problem (the individuals Bulk Cargo Loader and Train Cargo Loader have been duplicated to improve readability). It is relatively lightweight, and hence quick to develop. No characteristics have been defined for the properties and all concepts are undistinguished individuals (instances of the generic class Thing). The object properties defined in the ontology are expressed as sub-properties of those defined in the metamodel (Fig. 3) according to Table 2. Due to this hierarchical relation between object properties, the object properties defined in the domain ontology share their domain and range with their superproperties (Table 2). The reasoner can then classify the domain ontology individuals as instances of metamodel classes corresponding to the domain and range of the property to which they relate (see Table 3).

The automatic extraction of implicit knowledge is also possible by the reasoner to uncover inverse properties and property chains defined in the AOSE metamodel. For each individual defined in the ontology that participates in relations involving only 13 types of domain ontology properties, 21 additional properties will be automatically inferred. This enriches the domain ontology considerably, producing a more complete artefact for V&V and to support modelling.

Table 3 Domain ontology individuals as members of their corresponding classes in the metamodel
Full size table

The reasoning process guarantees that all knowledge included in the augmented ontology is consistent. Having a rich and consistent domain ontology to compare with the models provides and indirect mechanism to eliminate inconsistencies and to add detail to the models. The semantic relationships included both in the ontology and the models, such as GoalIsSubGoalOfGoal or RoleParticipatesInActivity allow for a richer validation, as the mechanism is not limited to subsumption or part-of relationship. The approach supports the modelling of a wide range of semantic relations suitable for a variety of models. Any discrepancy between models and ontology will lead to the identification of potential inconsistencies or incompleteness in the models as per Sect. 3. For example: identifying a triple in the ontology relating a role with an activity through the participation relationship that is not included in the models, may lead, in closer inspection, to adding the relationship to the model.

6 The tool automating the ontology-based V&V process

We develop a CASE tool to harness reasoning mechanisms as discussed in Sect. 5. The tool is composed of two modules: The domain ontology augmentation module and the operators comparison module. Each module is detailed in what follows.

6.1 Domain ontology augmentation module

This module automates the augmentation of the domain ontology with AOSE concepts and relationships. An interface provides access to the augmentation mechanism. The functionality of the module is described in the following six steps:

6.1.1 Merging of metamodel and domain ontology

This initiates the augmentation by process, merging the lightweight domain ontology and the AOSE metamodel. To later enable automated reasoning, the object properties of the domain ontology are defined as corresponding object subproperties in the metamodel (as earlier described). The output of this step is a new ontology that includes the AOSE metamodel (with its classes, object properties and axioms) and the domain ontology (with its individuals and object properties assertions).

6.1.2 Initial classification and relation inference

The automatic reasoner uses the output ontology from Step 1 to perform four our essential tasks: consistency checking, individual classification, extraction of implicit assertions and materialisation of the resulting ontology. The reasoner identifies inconsistencies where individuals as simultaneously belong to disjoint classes or when asserted object properties violate the rules defined by the metamodel. Detection of inconsistencies halts the augmentation process until they are eliminated. If the ontology is free of inconsistencies, the reasoner automatically classifies the individuals as instances of the AOSE metamodel classes. Some implicit knowledge is also made explicit. The final task of the reasoner is to materialise the inferred knowledge into an output ontology for further processing.

6.1.3 Augmentation with relations between roles and environment entities

This seeks to extend the ontology. Assertions that link individuals belonging to the class Role with instances of the class EnvironmentEntity are made explicit through the use of the object property RoleUsesEnvironmentEntity. This enables adding a property chain to infer all relations unconditionally. In particular, the module examines the ontology to identify the activities that have only one participating role. The environment entities used in those activities can be associated with the role participating in the activity, through the relationship RoleUsesEnvironmentEntity, to produce the output ontology.

6.1.4 Augmentation with agents

This creates agents covering the functionalities defined for the roles in the ontology. The original domain ontology defines the stakeholders of the problem as roles. Agents are the realisations of the roles at design time and therefore they are not part of the client conceptualisation. However, the concept is essential and is involved in a significant number of relationships in the metamodel. The module adds an agent (as an individual) for each existing role in the ontology.

6.1.5 Second reasoning to infer agent-related relations

This uses the ontology produced in from Step 4 to make explicit assertions in relation to the agents. New individuals are classified as members of the class Agent. Assertions involving roles are then extended to include agents, i.e. if Agent1 plays Role1. New assertions are created to relate Agent1 with the goals pursued by Role1 (and the any involved activities and used environment entities). The reasoner materialises the inferred knowledge to create the output ontology in this step.

6.1.6 Cardinality violation detection

This checks that the domain knowledge fulfils the cardinalities of the relationships as defined in the AOSE metamodel. In an axiom, the metamodel defines that every goal is achieved by at least one activity. If the individual Goal1 belonging to the class Goal is not achieved by any activity in the domain ontology, the reasoner does not detect this as a cardinality violation, but simply as missing knowledge: there might be an unknown activity that achieves Goal1. To solve this problem, this step looks at the object assertions defined in the ontology to ensure that the cardinalities of all object properties are satisfied. For example, the individuals Activity1 and EnvironmentEntity1 belong to the classes Activity and EnvironmentEntity respectively. If Activity1 is not related to any environment entity through the relation ActivityNeedsEnvironmentEntity, no violation will be detected because the cardinality of the object property is zero to many. However, if EnvironmentEntity1 is not related to any activity through the relation EnvironmentEntityNeededInActivity, a violation will be detected as the cardinality of the object property is one to many. The output of the step is a list of violations, indicating which individuals are violating which axioms. The ontology engineer is required to solve any violations to restart the automatic augmentation process.

The ontology augmentation process concludes with an inconsistency free ontology that satisfies the axioms defined for the AOSE paradigm. This ontology can then be discussed with the client to ensure that it accurately represents their problem conceptualisation. The augmented domain ontology will be the input of the ontology-models comparison module, used to detect discrepancies between artefacts.

6.2 Ontology-models comparison module

This module compares each of the requirement models with the available domain ontology to detect discrepancies. The inputs of the module are the augmented ontology and the models to be validated. The functionality of the module can be described in the following three steps:

6.2.1 Model-ontology comparison

This performs the comparison between model and ontology. For each model, difference lists are generated for each concept and relationship . Equivalent lists are generated from the augmented domain ontology. Common elements in corresponding lists (extracted from the ontology and model) are first removed. When no further elements can be removed, any remaining elements of the ontology list denote elements defined in the ontology but not in the model. Conversely, remaining elements of the model lists correspond to elements defined in the models but not in the ontology.

6.2.2 Dependency analysis

This takes the lists generated in Step 1 as input to discover how changes provoked by solving these discrepancies would affect other models. For each element in each list for each model (a concept or a relationship), the AOSE metamodel is examined to identify which other models may use it. These dependent models are compiled to produce the final report in next step.

6.2.3 Report compilation

This summarises the outcome of the comparison process specifying which elements have been defined in the ontology but not in the models, and vice versa. For each discrepancy, the report states which models share the use of the discrepant element. A simple graphic user interface (GUI) has been developed to present the analysis of the report.

Two development exemplars are used to evaluate the tool and ontology-based V&V process in the next section. The exemplars will use different domains and different methodologies.

7 Validation of the ontology-based V&V process and tool

Our goal is to present a MAS models V&V add-on process that can be used with any of the extant MAS development methodologies. Therefore, the evaluation activities presented will ensure that the three validity threats are mitigated against. Thus the research is designed to test and ensure all independence along three dimensions: developer-, domain- and methodology: We select two different domains and two modelling methodologies. We also illustrate the resilience of the process when applied with low quality artefacts from different developers. This will also seek to simulate a lack of client involvement in the requirement elicitation process to create the domain ontology and the models. Finally, we study how the various types of discrepancies (described in Sect. 3) may occur and assess the time cost of applying the process.

7.1 Setup of the evaluation studies

Two domains have been used in the evaluation of the approach described in this paper. One system will implement a web-based Wine Broker and the other an agent-based simulator to estimate the delay of an Aircraft Turnaround. These domains are described here next. In addition to confirming the domain independence of the process, each of the exemplars will use a different AOSE methodology. The first will be developed with ROADMAP and the second will be developed with MOBMAS. The specifications of these two exemplars come from industry. The specification for the plane turnaround comes from a simulation software company based in Melbourne and the specification for the wine broker interface come from a wine brokering firm based in Hunter Valley in New South Wales (Australia).Footnote 1

Each of these two domains represent a class of applications that are typically associated with the use of multi agent systems analysis techniques. Together chosen, they provide a representative coverage of multi agent systems applications. The first domain represents an example of a class of problems often labelled as Electronic Institutions or online auctions [12]. These are characterized by having mediating trusted agents and indeterminate number of external agents. These systems typically operate in heterogeneous environments. The second domain represents a typical simulation application and exemplifies a class of simulations applications that are typically tackled using agent-based modelling [21]. As such, these two problems, an online auction example and a simulation example, provide complementary representative examples of the set of applications that typically motivate the use of agent based modelling.

Domain 1: The Wine Broker Problem A Wine Broker company specialises in establishing links between wineries and their potential clients, to facilitate wine distribution. The business model of the wine broker is described below and is depicted graphically in Fig. 4.

Purchase wine Clients can browse an on-line catalogue of wines and purchase those that best fit their needs. An automatic system is available to recommend wines to clients given their preferences. Additionally, club members can request a tailored wine consultancy with one of the wine experts associated with the broker.

Organise cellar door visit Any client can arrange a visit to any of the associated wineries through the wine broker company, depending on the availability of time slots at the winery.

Join wine club Regular clients can apply for membership of the wine club. Wine club members commit to buying a certain amount of wine regularly and, in exchange, they obtain benefits such as special prices on some products, preferential invitations to special events and waiving of delivery costs.

Organise home tasting Wine club members are entitled to apply for a home tasting delivered by a wine consultant associated with the wine broker.

Participation in special events The wine broker periodically organises special events, such as wine tasting in scenic environments or a combined dinner and wine tasting.

In the first exemplar, we use the ROADMAP methodology [40] to model an agent oriented system for the wine broker domain. We simulate a low involvement of the client by initiating the process with deliberate low quality artefacts. The ontology is developed by a group of students as part of an introductory course on knowledge modelling. The agent models are developed by a student as part of his Honours thesis. The first author of this paper played the role of the client.

Fig. 5
figure 5

Aircraft turnaround problem description

Full size image

Domain 2: The Aircraft Turnaround Problem: Aircraft turnaround refers to the process of preparing an arriving aircraft for departure after it lands. Figure 5 depicts the activities comprised by the aircraft turnaround as described by the client specification. Whilst the domain knowledge for this exemplar is largely based on practices at Melbourne Airport in Australia, the practices described are largely common across different airports and countries. The aircraft turnaround process commences once the aircraft has landed and taxied to its ramp area and it concludes once the aircraft is prepared to take-off. The manoeuvres of aircraft landing, parking and take-off are out of the scope of the problem domain. The modelling exemplar focuses on the flow of the turnaround activities. Modelling or simulating the activities themselves is out of the current scope. Once the turnaround process commences, the airport ground staff responsible spring into action. The first task is to set the wheel chocks to ensure that the aircraft will remain stopped and then to position the airbridge to access the cockpit of the aircraft. Five parallel workflows then follow: three workflows refer to technical aircraft maintenance activities, one refers to the handling of passengers and the last refers to the handling of baggage. The aircraft maintenance engineers and technicians carry out the routine and non-routine maintenance activities.

The fuelling activities are carried out by the airport ground staff. The activities related to the baggage handling include unloading and loading the baggage of the incoming and departing passengers, respectively. The last parallel workflow involves managing the passengers and preparing the cockpit of the aircraft. The first activity is the disembarkation of passengers and then of the crew. Cleaning and catering staff work then in parallel to clean the cabin and cockpit and cater the aircraft. Only when cleaning and catering are completed can the new crew then embark and perform a routine cockpit check. The last activity of the workflow, embarking passengers, is a coordinated effort of the crew (on-board) and the airline ground staff (at the boarding gate). When all the parallel workflows of the aircraft turnaround are concluded, the final sequence of activities prepares the aircraft for departure. The departure preparation is conducted by the airport ground staff and involves the removal of the airbridge, the wheel chocks and attaching a tug to position the aircraft in its assigned departure position.

In the second exemplar, we used the MOBMAS methodology [44] to model an agent oriented system for the aircraft turnaround domain. In this occasion, we initiated the process with higher quality artefacts. We reused an ontology for the aircraft turnaround domain from a previous project [28]. The first author of this paper acted again as a client. The agent models were developed by the same developer from the previous exemplar. The experience in modelling agent systems acquired in the first exemplar, was key to produce an initial model of higher quality.

7.2 Execution of the development exemplars

7.2.1 The wine brokering exemplar

The wine broker domain ontology was developed by the students as a set of undistinguished instances of the generic class Thing (as per Sect. 4). The relations between individuals are modelled as object properties. Figure 6 shows an excerpt of the domain ontology (individuals in the right-hand side of the figure have been duplicated to improve readability).

Fig. 6
figure 6

Excerpt from the domain ontology for the wine broker problem

Full size image

During the domain ontology augmentation process, an inconsistency is detected. The ontology engineer had modelled Special events manager needs Wine club member form instead of the correct assertion Special events manager uses Wine club member form. The object property needs is associated with activities while the property uses is associated with roles. Once the ontology is amended, the augmentation concluded without further human intervention. The domain concepts were automatically classified into their corresponding AOSE category (for example, Club member as a role and Publish special events as an activity). The number of assertions in the ontology rose from 222 to 995, adding relations such as the ones depicted in Fig. 7.

The V&V process required four iterations. The ontology was originally augmented in the first iteration (described above) and, because of changes in the client conceptualisation, three more times during the second, third and fourth iteration. In all cases the augmentation was done automatically requiring human intervention only to solve cardinality axioms introduced by the ontology engineer. For example, during the second iteration, the augmentation process detected that the definition of the environment entity Bank violated the axioms Environment entity needed in at least 1 Activity and Environment entity needed for at least 1 Goal.

Fig. 7
figure 7

Augmented wine broker relations involving goals, activities, roles and agents

Full size image

During the first iteration of Task 2.1 the comparison operators were automatically applied to the available applications, goal, role, organisation and environment models. The discrepancy report was instantaneously generated without human intervention. The report collected discrepancies affecting the definition of goals, roles and environment entities. For example, the structure of the goal hierarchy was simpler in the ontology than in the goal model. In terms of relationships, the organisation model differed greatly from the ontology because no authority relations (RoleControlsRole and RoleIsPeerRole) had been modelled by the ontology engineer. The discrepancy report was studied to resolve all discrepancies in Task 2.2. It was determined that it was necessary to simplify the definition of goals in the models. Concepts such as Login or Process payment should not be considered as goals but as necessary activities to achieve the goal of Sell wine online and, therefore, should be removed from the goal and role models. The discrepancies involving the authority relations between roles originated a discussion. Although none of them had been defined in the ontology, they were not blindly added. It was decided that there was no real authority between roles in the wine broker, therefore the relation RoleControlsRole should not be used in the organisation model and thus not added to the ontology. The discrepancy between environment model and ontology with regards to the environment entity Courier being defined only in the model was solved by adding it to the ontology. Furthermore, this discrepancy inspired the addition of a new environment entity to both the ontology and environment model, Bank, to represent the financial services needed for the online shopping service.

Fig. 8
figure 8

Augmented aircraft turnaround relations involving roles, agents, events and environment entities

Full size image

The second iteration of Task 2.1 processed a refined version of the existing models and the scenario model for the first time. The models that had undergone one iteration of the V&V process did not present many discrepancies with the ontology. However, as activities were first introduced during this second iteration, an important number of discrepancies in the definition of activities between scenario model and ontology were detected. Although some activities were only defined in the scenario model (such as Make payment), many more were only defined in the ontology (for example Club member books for special event). It was decided that many activities defined only in the ontology (such as Add selected wine to purchase cart) and some in the scenario model (for example Receive query) were too low level and should be removed.

The third iteration of the V&V process introduced for the first time the agent and interaction model. These models share many features with the scenario model, in particular the definition of activities and the participation of roles in them. As the scenario had been improved as result of the previous iteration, the agent model benefited avoiding many compound errors. The automatic application of the operators detected few discrepancies between the ontology and scenario, interaction and agent model. Organisation, environment, goal and role model presented no discrepancies and were considered validated. The report was studied to remove the existing discrepancies. The fourth iteration of Task 2.1 showed no discrepancies with any model and the V&V concluded.

7.2.2 The aircraft turnaround exemplar

The concepts describing the problem domain are again represented in the ontology as individuals belonging to a generic class, Thing. The augmentation module automatically processed the domain ontology to structure its concepts according to the AOSE paradigm (e.g. Baggage train was classified as an environment entity and Baggage handling as a goal) and extract implicit knowledge (as illustrated in Fig. 8). The augmentation module concluded the preparation of the domain ontology by raising the number of object property assertions (i.e. relationships between domain concepts) from 122 to 824. The second and third iteration of the process required minor modifications of the domain ontology to comply with the decisions made in Task 2.2. In both iterations the support tool automatically augmented the ontology without detecting inconsistencies or cardinality violations.

In the first iteration of Task 2.1, the comparison module received the augmented domain ontology and the available models—task, organisational and role model—as input, to automatically apply the ontology-based operators and produce the discrepancy report. The discrepancy report highlighted discrepancies involving the definition of activities and roles. For example, the organisational model role Engineer was defined in the ontology as two separate roles Engineer and Fueller. This discrepancy was resolved by adding the role Fueller to the model and it also originated a discussion that resulted on adding a new role, the Technician, to ontology and models. The role Passenger was defined in the ontology and in the role model, but not in the organisational model. This uncovered an inter-model inconsistency that was resolved by adding the role to the model.

During the second iteration, the developer provided a refined version of the task, organisational and role models, and new versions of the resource, agent and goal models. The discrepancy report showed no new discrepancies between the task and organisational models and the ontology. The task and organisational models were considered validated. Most discrepancies concerned the definition of environment entities or goals. The modeller had included in the goal model a goal corresponding to every task to be performed by the system, such as Remove wheel chocks or Crew check. In Task 2.2, it was decided that all goals defined in the model but not in the ontology that corresponded to system tasks were unnecessary and therefore were removed from the models. During the third, and final, iteration of Task 2.1, the application of the comparison operators showed that all models had converged with the ontology, and the V&V process concluded.

7.3 Discussion of the V&V process evaluation

In both exemplars, the operators were successfully applied to identify discrepancies between the ontology and the models. These discrepancies proved useful for detecting modelling flaws in the models, improving the overall quality of models. Within four iterations, the models converged with the client conceptualisation. The final iteration in each exemplar simply confirmed the convergence. Changes in the ontology were also accommodated enabling the client to better articulate the system requirements. The quick convergence however suggests that the changes in the ontology were not due to major modifications of the client requirements. This focus on improving the requirements specification further highlights the importance of the iterative design of the process. In particular, models are not compared directly with each other for consistency, but indirectly through the ontology. Models often represent aspects of the domain from different points of views. This multi-faceted knowledge must be consistent across models. The ontology provides a unique representation of the domain that can ensure that, when all models comply with it, no inconsistency will be created. The third iteration of the first exemplar illustrated this mechanism. In Task 2.1, discrepancies between the ontology and the agent and environment models were detected because of the modification of the ontology in the second iteration.

Table 4 Breakdown of estimated times for the exemplars
Full size table

The process execution time was shorter for the second exemplar. This can be understood when comparing the individual task times for both exemplars. Table 4 provides estimations of the times required to perform the manual tasks associated with the process, i.e. the overhead that we introduce into the development process. These overhead includes the ontology acquisition, the decision making process consisting of examining the report to decide how to solve the discrepancies (either changing the ontology or the models) and the modification of the ontology to comply with these decisions after the first iteration. Additionally, Table 4 presents the refinement of the models as the last activity of each iteration (except for the last iteration). The second exemplar concluded in three iterations whereas the first exemplar required four iterations. The faster execution of the exemplar can be attributed, amongst other factors, to the degree of participant expertise. Both the ontology engineer and MAS modeller were experienced in the second exemplar. This led to a higher initial quality of ontology, and models that required fewer modifications and converged faster. As shown in Table 5, the domain ontology required fewer modifications in the second exemplar (1.5 h of work) than in the first exemplar (4 h of work). Actually, the modification of the ontology in the second exemplar required 1.5 h while the models required 4 h worth of work. C.f with the first exemplar where the initial ontology was of a lower quality, the models remained more stable than the ontology, requiring only 3 h worth of work versus the 4 h taken for the modifications to the ontology. This indicates that although the initial quality of the ontology has an impact on the V&V process by introducing some delay (one additional iteration in this case), it does not severely affect the efficiency of the V&V process, as the models and the ontology eventually converged to comply with the client’s conceptualisation. For both exemplars, the models were modified in each iteration to resolve errors that, potentially, could have critically affected the functionality of the final system.

Part of the reduction in time between the first and the second exemplar cannot, however, be directly attributed to the experience of the ontology engineer. The acquisition of the ontology required half of the time for the second exemplar compared with the first one (Table 4). Whilst in the first exemplar there was no initial domain ontology available (and hence a new one had to be developed from scratch), in the second exemplar it was possible to re-use a domain ontology that only required minor adaptations.

The exemplars also highlighted quickly that early validation of models prevents the propagation of errors to other models that have not yet been developed. That is, with the process rework is avoided and the chance of discovering bugs in systems in production is minimised. An example of this can be found in the second iteration of Task 2.1 in the second exemplar, when the extended version of the scenario was developed. This meant that it was possible to avoid the error detected in the first iteration with regard to the interaction model: the overcentralisation of the aircraft turnaround activity through the role of the Manager.

Table 5 Comparison of ontology and models improvement times
Full size table

Discrepancies that lead to the identification of new requirements introduce an innovative use of ontologies, to assist in requirements elicitation. Developers valued these discrepancies as they pointed out facts that had not been taken into account during the initial modelling process. All discrepancies of this type were considered correct after they were assessed. Some of them were incorporated in the models, but many others created additional complexity that was considered unnecessary. Arguably, his assessment would be different in development of critical applications where they could be relevant for added reliability and tolerance of the system. Discrepancies leading to complete the models and to elicit new requirements uncovered another innovative benefit of the ontology, speeding the modelling process. These potentially reduce the development time and possibly cost.

The effects of automatically augmenting the ontology are very noticeable. For instance, in the third iteration of the first exemplar, the automatic augmentation mechanism raised the number of relations from 150 to 864. As the base domain ontology had to be modified to introduce the new requirements, the ontology is again automatically augmented from scratch, to propagate the new changes. By automatically regenerating the full augmented ontology in each iteration, the manual modifications are kept to a minimum. Table 6 shows the number of relations of the ontologies used in the exemplars throughout the iterations. The ratio of relations in the base ontology to the augmented ontology varies from 1:4 to 1:6. This variation is caused by the interrelations between the elements in the ontology (e.g. Roles, Activities or Goals). The AOSE metamodel that underpins the augmentation mechanism is capable of extracting from the use of 13 types of relations in the base ontology—explicit knowledge—21 additional types of relations—implicit knowledge—(see Sect. 5.1). Table 7 compares the number of modifications in the base ontologies with the propagated modifications in the augmented ontologies. By manually modifying the base ontology, rather than the augmented ontology, and letting the augmentation mechanism automatically propagate the changes, the ontology engineer significantly reduces their intervention in a factor of at least 3. The drastic augmentation in the number of asserted relationships (on average five times the initial figure) provides a rich benchmark to suggest improvements to the models. It is expected that in the V&V of models for critical applications, many of these additional relationships would be incorporated to the models to remove all traces of ambiguity and achieve a higher quality final system. In our exemplars however, we have not found indication of potential scalability obstacles. The reasoner completed the inference process in a tolerable time (a small fraction of the whole analysis phase). Further experiments will be conducted to study the augmentation times for larger ontologies—i.e. thousands of relationships. Computational complexity is problematic if the number of relations is exponential to the size of the ontology and the ontologies are large. If faced with a large ontology, fragmenting the ontology and recombining validation results may well be a way to bypass such complexity issues. However, resolving the technical steps involved in fragmenting and recombining is not easily answered and is beyond the scope of this paper. If indeed complexity issues are great and are due to exponential explosions, new reasoning mechanisms in the future may well be required. Additionally, as independent research on reasoners advances and new techniques are incorporated to existing software, the computational cost of reasoning will also reduce, contributing to solving the problem.

Table 6 Number of relations in exemplars ontologies by iteration
Full size table
Table 7 Number of changes in exemplars ontologies by iteration
Full size table

The analysis of the discrepancy report also highlighted another interesting benefit of the approach. The discussion of certain discrepancies helped in uncovering relatively unrelated requirements. An example of this can be found in the second iteration of the first exemplar. The ontology and model differed in that only the model included the concept of the Courier to represent an environment entity in charge of delivering a certain product. Not only was it decided to add it to the ontology, but by analogy it was decided to add to a similar environment entity, the Bank, to both the ontology and model, to represent the provider of the financial services necessary for the system. Similarly, in the first iteration, the V&V process drew the attention of the developer to the authority relations between roles as they were discrepant with the ontology (they had not been modelled by the ontology engineer). The discussion that followed determined that there should be no control relationships between roles, so the organisation model and ontology could be adjusted accordingly.

A goal of the second exemplar was to test the methodology independence of the ontology-based V&V process and its associated support tool. This exemplar uses the V&V process in another AOSE methodology, MOBMAS. The transition to MOBMAS, as an AOSE methodology, was largely seamless. The domain ontology was acquired in a similar way as earlier, without any assumptions about any AOSE methodology. The augmentation of the domain ontology was conducted automatically and knowledge was generated as described by the AOSE metamodel. The application of the operators was also done without human intervention. The support tool automatically generated reports for the correct set of models and detected discrepancies for the adequate concepts and relationships.

8 Conclusion and future work

This article presented a new approach to enhance the quality of the models acquired through interactions between the developers and the future users of a software system. The interactions between the clients and the analysts are complemented with the use of domain knowledge. We set out to develop a validation process of agent requirement models which is both domain and methodology agnostic. Thus, a new ontology-based V&V process is created to support the development of requirement models. The process is then evaluated in two different methodologies (MOBMAS and ROAMAP) and in two different domains (a plane turnaround simulation and in wine brokering application).

The process is underpinned by the automatic reasoning capabilities of ontologies. Automatically augmenting the ontology with agent oriented concepts drastically increments the number of explicit knowledge assertions available in the V&V ontology. This helps uncover hidden requirements or inconsistencies. The V&V process is defined to introduce tolerable overhead in creating the improved requirement models. An automatic support tool of the process is developed to facilitate its application. The process is conducted iteratively accommodating a typical AOSE methodology lifecycle. It incrementally and concurrently validates the models produced. The ontology augmentation and operators application tasks are computationally demanding. This adds a time constraint that may need to be accounted for in planning of some projects. This automation is nevertheless critical to reduce the human intervention to simply deciding on best way to remove modelling errors.

The process and its automatic support tool were iteratively refined and validated through the planning and execution of the two development exemplars. Our evaluation exemplars were configured to mitigate against several validity threats: to test the domain independency, two different domain problems were used, the aircraft turnaround problem and the wine broker problem. The evaluation across two different domains ensured that the first research question How do we exploit domain ontologies to develop appropriate validation operators for all given agent requirement models in a given development project? is affirmatively answered through the set of the developed operators. The evaluation across two methodologies, MOBMAS and ROADMAP ensured that the second research question How do we exploit methodological knowledge expressed in a generic agent metamodel to ensure method interoperability of the process? is affirmatively answered through incorporating the FAML subset to guide the reasoning processes to identify inconsistencies. The methodology independence was also evaluated by using two different methodologies, ROADMAP and MOBMAS. The evaluation thus far indicates that the ontology-based V&V process is, indeed, independent of the participants, problem domain and the development methodology. However, to further mitigate against these and other validity threats, further evaluation is required.

Our evaluation showed that the quality of the ontology and the initial quality of the models had limited impact on the V&V times. Even when the stakeholders were unexperienced, the models converged relatively quickly, requiring only one additional iteration when compared with the more experienced developers. The final outcome was invariably a set of models fully compliant with the client’s specification. This makes the automatic V&V process particularly suitable in projects where the domain is complex or unknown. The V&V process introduces an overhead to the development process. The additional cost to achieve higher quality requirements models will be justified for critical applications, where errors can be costly in terms of economic impact or even human lives. The ontology V&V process has been designed for developing agent oriented models which are less commonly understood, complex and prone to modelling errors. This easily justifies the overhead of applying the V&V process. However, it is theoretically possible to adapt the process to other development paradigms (as long as they support iterative modelling), for example object oriented methodologies. This is one goal of future research. We are currently assessing the computational complexity associated with using ontologies in the development process. Recently, in [10], we showed that retrieving an ontology based on requirement models is computationally minimal. With respect to propagating changes, and exploring all reasoning outcomes, this work showed that it is a manageable overhead. However, before making any definite statements, further work is required to ensure that in even larger domains the computational complexity remains manageable.