1 Introduction

Multi-sensor data fusion has recently attracted a great deal of attention in military and non-military applications. Data fusion has been defined differently over the years. In 1991, data fusion was defined as “a multilevel, multifaceted process dealing with the automatic detection, association, correlation, estimation, and combination of data and information from multiple sources” by the data fusion working group of the Joint Directors of Laboratories (JDL) [1]. In 2013, a principle definition of Information fusion was proposed as “the study of efficient methods for automatically or semi-automatically transforming information from different sources and different points in time into a representation that provides effective support for human or automated decision making” [2].

Different architectures and models of data fusion exist. These are mainly divided into information-based, activity-based and role-based [3] models. The focus of information-based models such as JDL [4] is on the abstraction of the data generated during fusion. Activity-based models are recognized according to the activities which need to be done via data fusion system. These activities and the exact sequence of execution are clearly defined, as in Boyd Control Loop [5] and Omnibus model [6]. Role-based models focus on the fusion roles and the relationships between them. Object Oriented Model [7] and the Frankel–Bedworth architecture [8] are the two members of this generation.

Compared to other available models, JDL model is more common in the fusion research literature. In this model, data fusion problem is divided into four different functional levels, in which situation awareness (SA) is level 2, and attempts to present a contextual description of the relationship between objects and experimental events by applying a priori knowledge and environmental data. SA is defined as “the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning and the projection of their status in the near future” [9].

With the growth of Semantic Web (SW), various research areas have adopted SW technologies. Semantics is used for extracting important information from raw data and thus facilitates the development of smart applications. That simplifies the presentation of user-defined domain knowledge in the form of formal ontologies and rule sets. Therfore, SA could greatly benefit from applying SW technologies.

On the other hand,it is necessary to mention that earlier approaches to SA are not appropriate for obtaining knowledge in real-time and cannot process the data stream. Therefore, SA methodologies improve with the use of data stream management techniques and Complex Event Processing (CEP). CEP is a kind of event processing which combines data from several sources to gather patterns or complicated events [10]. So the present paper offers an approach to situational awareness by combining the technologies of data stream management and SW.

The rest of the paper is organized as follows. Section 2 presents a brief background of the topic and relevant studies in three groups: JDL model with JDL-based works, SW technologies and semantic fusion works, and management of data stream and CEP. Section 3 introduces the proposed approach for enhancing SA. Section 4 gives details about the implementation of the proposed approach and its evaluation. Finally, Sect. 5 presents the conclusion of the paper.

2 Background and Related Works

In this part of the paper, the concepts and relevant studies are categorized in three main groups: JDL-based research, SW technologies and semantic fusion research, and management of data stream and CEP.

2.1 JDL Based Approaches

JDL data fusion model is the most extensively used and adopted functional model for data fusion. This model consists of five processing levels, a related database, and a data bus linking all the components. Figure 1 shows the structure of this model. Definition of the components of this model is summarized below [4]:

  • Sources There many types of source data. They can provide information from various levels. The range of data includes sensor data, as well a priori information from databases or even human inputs.

  • Level 0 Source Preprocessing At the very first level, data prescreening and allocating data to appropriate processes are performed. This allows fusion process to concentrate on the data that is most pertinent to the current situation. It also reduces the load of data fusion processing.

  • Level 1 Object Refinement To achieve representatives of individual objects, this level combines locational, parametric, and identity information. There are four key functions areas: (1) transforming data into a consistent reference frame and units; (2) estimating or predicting the position, kinematics, or attributes of objects; (3) assigning data to objects for statistical estimation; and (4) refining estimates for object identification or classification.

  • Level 2 Situation Refinement At this, the development of a contextual description of the relationship between objects and observed events is considered. The processing executed at this level determines the meaning of a collection of entities and then tries to incorporate the environmental information, a priori knowledge, and observations.

  • Level 3 Threat Refinement The projection of the current situation into the future is done at Level 3 to draw inferences about enemy threats, friendly and enemy vulnerabilities, and opportunities for operations. As this assessment deals, not only with computation of possible engagement outcomes, but also with the assessment of an enemy’s intent based on many knowledge, is especially difficult.

  • Level 4 Process Refinement The processing at this level is a meta-process. Key functions are: (1) monitoring the real-time and long-term data fusion performance; (2) identifying the information required to improve the multi-level data fusion product; and (3) allocating and directing sensor and sources to achieve the goals.

  • Database Management System Due to the variety and amount of data, and the requirements for data retrieval, storage, archiving, compression, etc., this system is very vital for data fusion.

  • Human Computer Interaction (HCI) This component provides both a mechanism for perception of human inputs, and the communication of data fusion results to the operators and users.

Researchers have investigated the JDL model extensively over the years. Some of the earlier studies have changed the levels of JDL model. As an example, the human role is specified in a fusion model and level 5 is added to JDL model as “user refinement” [11]. Lambert offers a revisions of definitions of level 1, level 2 and level 3 in the JDL model, while including level 0 within level 1 and absorbing level 4 within each of the other levels [12, 13]. So these three levels, defined as: level 1 is about the identification of objects from their properties, level 2 is about the identification of relations between these objects; and level 3 is about the identification of the effects of these relationships between these objects. In another study, authors revisit the JDL based on technology developments in recent years and highlight three information exploitation technology developments affecting the JDL model that include: (1) an enterprise architecture that supports information management to store and access data through cloud computing, (2) machine analytics for data mining, and (3) man-machine interfaces to support users as active observers of contextual reasoning [14].

Fig. 1
figure 1

The JDL model [4]

Full size image

The JDL model has been applied to other fields as well. As an example, the main goal of [15] is to use JDL model in the area of bioinformatics. They have prepared a mapping for the JDL data fusion model to bioinformatics and examined its applicability in this field by applying the model in a stem cell differentiation study. Or authors in [16] adapted the JDL model to suit national level cyber sensor data fusion with the aim to formally define and reduce data ambiguity and enhance fusion capability in a real world system. Also authors in [17] discovered the underlying processes in the JDL data fusion model and explained them further in a context of cyber security.

2.2 Semantic Web Technologies and Semantic Fusion Works

SW technologies aim to describe and represent the semantics related to data or information. Automatic processing, integration, sharing and reuse of the data are facilitated through SW technologies, including ontology and semantic reasoning.

An ontology is an explicit, shared specification of the various conceptualizations in a domain. It usually describes agreed data/knowledge structures, i.e., field concepts, their features, and their correlation. Ontologies can be understood by both humans and machines because they identify the semantic terms at the conceptual level, according to the explicit conceptualization of a domain.

In [18], the role of ontologies in integrating the data has been discussed and five applications have been presented for ontologies in data integration. These include: metadata representation, global conceptualization, high-level queries support, declarative mediation, and mapping support.

According to [19] ontologies have been utilized in one of the three following ways in data fusion systems:

  1. 1.

    Single ontology approach: in this approach, all the source schemas are connected to a shared global ontology, which offers a uniform interface to the applicant. A normal example for a system applying this method is SIMS [20].

  2. 2.

    Multiple ontology approach: here, there is a separate local ontology for all data sources and local ontologies are mapped to each other, instead of applying a global ontology. OBSERVER system is an example of this approach [21].

  3. 3.

    Hybrid ontology approach: a hybrid of the two previous methods is applied. In this approach similar to multiple ontology approaches, the semantics of each source is described by its own ontology. But in order to make the local ontologies comparable to each other, they are built from a global shared vocabulary.

A core ontology for situation awareness has been presented by Matheus and et al. [22]. The Situation Awareness Assistant (SAWA) has been introduced in [23]. SAWA simplifies the development of user-defined domain knowledge in the form of ontologies and rule sets and then monitors the evolution of higher-order relationships within a situation. Many of the research on SA has benefited from the ontology.

Data fusion models like JDL model, suffers from semantics issues. In order to achieve semantic interoperability and extract the implicit knowledge in a heterogeneous information system, semantic technologies can be used. A semantic fusion framework (SemFus) was proposed based on JDL in [24].

2.3 Data Stream Management and CEP

In recent years, streaming data are considered in different areas. Streaming data is produced continuously by many of data sources. Data stream was defined as: “a data set that is produced incrementally over time, rather than being available in full before it’s processing begins” [25]. Therefore, effective management techniques are needed for data stream management and processing.

Complex Event Processing (CEP) is technology for detecting situations in real-time among events. The purpose of CEP is to find meaningful events by combining data from several sources according to patterns. Luckham [26] defined CEP as a method which identifies complicated events by examining and associating other events.

EsperFootnote 1 is an open source software product for event processing that facilitates the identification of rich situations in event sequences and triggers actions when event conditions happen between event sequences, by providing aggregate function, pattern matching, event windowing and joining [27].

In this study, data stream processing techniques for enhancing SA was applied and further explanation of this approach is given in the next section with examples.

3 Proposed Approach for SA Using SW Technologies and CEP

Generally, situation awareness provides a contextual explanation of the correlation between objects and observed events. It utilizes a priori knowledge and environmental data for identifying a situation, assessing the current situation, and projecting it into the future for identifying probable threats, vulnerabilities, and opportunities for operations [9]. According to this definition, Fig. 2 presents the block diagram of the proposed approach for situation awareness using SW technologies and CEP. Note that in Fig. 2, the task and objective of each part of the chart are written inside the rectangle, and the tools applied to achieve the aims are written inside the oval. The required files and database can be seen at the top of the chart. In addition, as the lines specify, all the stages are according to the ontology. In other words, each stage needs the information included in the ontology for performing its task.

Fig. 2
figure 2

Block diagram of proposed situation awareness approach

Full size image

At first, data are received from several sources. Esper engine assists the management of streaming data and recognition of simple and complex events. For implementation, it is needed to express events as Java classes, write queries and rules in EPL (Event Processing Language) syntax, and store them in the specified file. It should be noted that EPL permits to express rich event conditions and correlation. Therefore, it minimizes the development effort needed for setting up a system which is able to react to complicated situations.

As an example, one of the used queries in the current study has been shown in Fig. 3. It means that if the mean speed of the considered machine is more than 0.8 of the highest speed of this type of machine, the fast movement of this machine is confirmed. In this query, length-window is applied. In the current study, different kinds of windows were used to define queries like time window, length batch window, time batch window, and externally-time window.

Some of the queries require information involved in the ontology, such as the highest speed for all the types of machine. Consequently, as shown in Fig. 2, Esper engine is related to the ontology and this relationship was considered in the implementation in the present study.

Fig. 3
figure 3

A sample EPL statement

Full size image

Then, data are converted to RDF format through RDFizer and are stored in the RDFStore. A timestamp is available for each Resource Description Framework (RDF).Footnote 2 Primarily, RDF permits specifying the data semantics in a standardized and interoperable manner. On the other hand, RDFSFootnote 3 (RDF Schema) is a language that describes vocabularies of RDF data and is applied for the definition of the semantic relations between properties and resources. In the implementation of the current study, we applied RDFS to represent ontology and all the data are provided in RDF format.

Semantic programming is needed for implementing the proposed approach. To this end, the jenaFootnote 4 which is a Java framework used to build SW applications, was used in the present study. It offers a programming environment for RDF, RDFS and SPARQL and contains a rule-based inference engine.

A semantic reasoner derives additional RDF, which is entailed from several base RDF along with any optional ontology information, and the axioms and rules associated with the reasoner. In order to optimize the application of this reasoner, specific rules were defined by an expert in military field and a comprehensive ontology was offered for our system. Therefore, our reasoner engine can perform the correct reasoning by exploiting these rules and ontology. Figure 4 indicates an example rule called “in weapon range relation”. According to the rule, object B is in the weapon range of object A, if object A has a weapon whose its effective distance is larger than the distance between the two objects.

Fig. 4
figure 4

An example rule for reasoner

Full size image

As Fig. 2 shows, as reasoner requires the knowledge comprised in the ontology, it links to the ontology. In the rule example, ontology presented information about the weapon characteristics and effective distance.

We have prepared built-ins implemented as procedural attachments in the type of Java method calls. As an example, distance is a user-defined built in.

Finally, a new inferred RDF was stored in RDFStore through timestamp. If needed, data can be retrieved from RDFStore using SPARQLFootnote 5 query.

4 Implementation and Evaluation

The following stages are defined and summarized for implementing the proposed approach for situation awareness in each area:

  • Defining associated ontologies

  • Defining the reasoner rules

  • Defining pattern event in EPL language for Esper engine

  • Implement RDFizer unit

  • Performing configuration of Esper engine and semantic reasoner

These stages can be applied to all fields such as wireless sensor networks, environmental monitoring, smart home, medical applications, and military environments. The present study is a test of the functionality of the use of the proposed approach in a military scenario.

As mentioned earlier, the proposed approach can overcome semantic problems through SW technologies. A brief comparison of the characteristics of the proposed approach with those of non-semantic approaches is summarized in Table 1.

Table 1 Comparison of proposed approach and other approaches
Full size table

5 Conclusion

Situation awareness is an interesting subject of theoretical and practical study particularly in dynamic systems and its improvement has been the subject of some research studies. In the present paper, a semantic method is offered to overcome the semantic problems in heterogeneous systems. SW technologies like ontology, RDF format, and semantic reasoning were used in the study.

On the other hand, situation awareness can attain the related situational and complicated events in real-time by applying methods of CEP and streaming data processing. Therefore, we offered a situation awareness approach which applied both SW technologies and data stream management techniques.

Finally, in the implementation, Esper engine was applied for the management of data stream and recognition of the complicated events and jena framework was utilized for semantic programming to describe ontology and have a semantic reasoner.