Keywords

1 Introduction

Digital images are of great importance nowadays in every domain such as medical, education, astronomy, fashion and security [1, 2]. Everyday huge amount of images are generated by either military or civilian equipment that need to be organized for efficient and accurate retrieval [2]. Image retrieval is the science of finding images that fulfill a specified user need [3].

In text based image retrieval system [4], images are annotated with keywords based on human perception. On the other hand, keywords are included in a user query based on his/her requirements. Query keywords are matched with the annotated keywords for image retrieval [5]. The relationship between queries and images and the relevance of the retrieved images is considered as a foundation for image retrieval system. This process has been extended with ontology to resolve the problem of semantic heterogeneity. Ontology is a type of background knowledge. A complete description of ontologies and their usefulness in image retrieval system have been mentioned in [6]. Ontological query language system: immense, has been proposed by Town et al. [7], for efficient image retrieval. Similarly ontology based systems such as OLYBIA and OntoPic have been proposed in [8, 9]. In the existing systems, the annotation of images with keywords is binary that is a keyword is either present or not in an image. However, annotation and the retrieval processes involve human perception that is mostly approximate or uncertain [10, 11]. We believe that the images cannot be precisely represented with keywords using binary model of annotation. Therefore existing systems could not produce the desired results. To overcome this problem fuzzy based retrieval systems are required as the relative importance of a particular keyword in annotation and retrieval process is different for different users and this information is essential for accurate satisfaction of a query.

In this paper we have proposed a fuzzy ontology-based model that makes use of ontology to improve the retrieval performance. Images are represented with concepts and a category. An image illustrates multiple concepts. A concept describes objects that the image contains. In order to annotate the image with all the possible concepts, it is divided into regions and regions are classified into concepts by adopting the technique proposed in [12]. The frequency of occurrence of the concepts inside an image determines the category. A category depicts a scene. This categorization allows for semantic comparisons of scenes and also helps in search space reduction while querying for specific concepts inside a category. Concepts, categories and images are linked among themselves with fuzzy values in the ontology. By adding a value for degree of membership to each concept and category, the retrieved images from ontology based search reflect the likely information need. For mapping the query terms and ontology concepts, fuzzy search mechanism is applied that search and rank the retrieved results based on the relevancy degree between the keywords of a query and images.

The remaining paper is organized as follows: Sect. 2 describes the related work. Section 3 describes the proposed methodology and image retrieval process. Section 4 contains the experimental results. The paper has been concluded in Sect. 5.

2 Related Work

Image retrieval systems are either content based or text based. In content based image retrieval systems (CBIR) low level features are extracted automatically and images are indexed by their visual content like color, shape, texture [1] but there is a gap between what image features, a system can recognize and what human perceives from the image. The focus of this research is on text based image retrieval systems so we will not discuss CBIR systems. Literature survey is categorized as: text based, ontology based and fuzzy ontology based retrieval systems.

2.1 Text Based Retrieval Systems

In text based image retrieval, images are annotated with text descriptors [4]. On the other hand, keywords are included in a user query based on his/her requirements. Query keywords are matched with the annotated keywords for image retrieval [5]. Keyword and content based image retrieval has been proposed in [13]. The model is built for qualitative spatial relationships like before, after or more, less. In [14], text based image retrieval system has been combined with content based model for efficient search. First text based search was applied and then content based filtering was applied on the resulting set.

2.2 Ontology Based Retrieval Systems

Ontology is an explicit specification of the terms in a domain and the relations among them [5]. It defines a common vocabulary that can be shared among people and domain knowledge that can be reused [15]. In [16], author built and proved the ontologies in reducing the gap between low level features and high level semantics. Keyword and ontology based image retrieval has been compared by Wang et al. [17]. Result showed that ontology based system performed better as compared to keyword based system in terms of precision. A complete description of ontology’s and their usefulness in image retrieval systems has been mentioned in [6]. In [12] a supervised learning system named OntoPic has been proposed that allows semantic searches. It has used DAML + OIL for domain knowledge but system performance was not mentioned. In [18], semantic based image retrieval system has been proposed. A domain specific, flower family, low level feature based ontology has been created. These low level features were set as a data property in OWL. Users can enter a text/image query. Features are extracted from a query image and matched with the ontology and matched images are retrieved and shown to the users. Semantic image representation model containing local and global categorization of scenes has been proposed in [19]. Ontology based image annotation (OLYBIA) system has been proposed in [8]. Low level features were extracted and were mapped to high level concepts through object ontology and inference rules. The experimental results have not been compared with any other model. In [20], content based image retrieval system has been proposed that has used ontology for object recognition. Retrieval algorithm has not been discussed and results were not compared with any other system. In [21], image annotation and retrieval through ontology has been discussed. Ontology was constructed for animal domain. Although the proposed work shows the benefits of using ontologies but the burden of manual annotation was still there.

2.3 Fuzzy Ontology Based Retrieval Systems

Document search using fuzzy set theory has been described in [22]. The proposed model considered the importance of text descriptors in search and the relevancy score between the query and the documents. Highly relevant documents were retrieved based on fuzzy set operations and shown to user. Information retrieval model based on ontology encoded with fuzzy relations has been proposed in [23]. When a user enters a query, composed of concepts, the system performs query expansion and may add new concepts based on the ontology knowledge. After expansion the similarity between the query and the documents is calculated by fuzzy operations. Author has compared his proposed model with Ogawa [24] and Horng [25] model. In Ogawa model, a keyword connection matrix has been proposed for computing the relevance of the document with the user keywords. Apart from it, users can enter compound queries containing operators: and, or, not. In Horng model, a multi-relationship fuzzy concepts network has been proposed that shows the fuzzy relations between the concepts and their relevance degree with the documents. Results show that the model proposed in [23] gives better retrieval accuracy as compared to Ogawa and Horng model. The above mentioned fuzzy based systems were tested for text documents retrieval.

3 Proposed Methodology

An overview of the proposed image retrieval system is shown in Fig. 1. Images are annotated with concepts and category by adopting the technique followed in [12]. Fuzzy knowledgebase and Fuzzy search are two main modules of the proposed system. An image with associated concepts and category are the inputs to the Fuzzy knowledgebase. To conceptually represent an image fuzzy ontology is constructed that utilizes the concepts and categories associated with the image. The fuzzy values in the ontology are then calculated by applying data mining approaches on the input images. For image retrieval, users are provided with an interface where they can input multiple keywords based on their requirements. Fuzzy search mechanism is applied and the retrieved images are ranked and shown to the user based on the relevancy degree between the image and keywords.

Fig. 1.
figure 1

The proposed image retrieval system

Full size image

3.1 Fuzzy Ontology Construction

The fuzzy ontology is constructed based on the idea of [23] that was proposed for documents retrieval. The fuzzy ontology shows the relationship between the images and concepts, and concepts and categories and categories and images as a value between 0 and 1 (both 0 and 1 inclusive).

The steps followed for computing the fuzzy values in the ontology is as follows: Let I {I1, I2, I3, . . ., IM}, A = {A1, A2, A3, . . ., AN} and B = {B1, B2, B3, . . ., BO} be a set of images, concepts and categories consisting of M, N and O number of elements respectively.

Let WCB be the matrix representing binary weights of category to image and is written as:

$$ {\mathbf{W}}_{\text{CB}} = \left[ {\begin{array}{*{20}c} {\varvec{w}_{11} } & {\varvec{w}_{ 1 2} } & \cdots & {\varvec{w}_{{ 1 {\text{M}}}} } \\ {\varvec{w}_{21} } & {\varvec{w}_{ 2 2} } & \cdots & {\varvec{w}_{{ 2 {\text{M}}}} } \\ \vdots & \vdots & {} & \vdots \\ {\varvec{w}_{\text{O1}} } & {\varvec{w}_{\text{O2}} } & \cdots & {\varvec{w}_{\text{OM}} } \\ \end{array} } \right], $$
(1)

where wkj = 0 or wkj = 1, 1 ≤ k ≤ O and 1 ≤ j ≤ M. Let WCI be the matrix representing the frequency of concepts in images and is written as:

$$ {\mathbf{W}}_{\text{CI}} = \left[ {\begin{array}{*{20}c} {\varvec{f}_{ 1 1} } & {\varvec{f}_{ 1 2} } & \cdots & {\varvec{f}_{{ 1 {\text{M}}}} } \\ {\varvec{f}_{ 2 1} } & {\varvec{f}_{ 2 2} } & \cdots & {\varvec{f}_{{ 2 {\text{M}}}} } \\ \vdots & \vdots & {} & \vdots \\ {\varvec{f}_{\text{N1}} } & {\varvec{f}_{\text{N2}} } & \cdots & {\varvec{f}_{\text{NM}} } \\ \end{array} } \right], $$
(2)

where fij is the frequency of concept Ai in image Ij, and 1 ≤ i ≤ N and 1 ≤ j ≤ M.

The image content is originally a crisp set as defined by WCB and WCI and is made fuzzy by proposed methodology. In our system, image content is represented by three matrices namely weight of the concept to image WA, weight of the category to image WB, and weight of the concept to category WCF and are defined as:

$$ {\mathbf{W}}_{\text{A}} = \left[ {\begin{array}{*{20}c} {\varvec{a}_{ 1 1} } & {\varvec{a}_{ 1 2} } & \cdots & {\varvec{a}_{{ 1 {\text{M}}}} } \\ {\varvec{a}_{ 2 1} } & {\varvec{a}_{ 2 2} } & \cdots & {\varvec{a}_{{ 2 {\text{M}}}} } \\ \vdots & \vdots & {} & \vdots \\ {\varvec{a}_{\text{N1}} } & {\varvec{a}_{\text{N2}} } & \cdots & {\varvec{a}_{\text{NM}} } \\ \end{array} } \right], $$
(3)

where aij is the relevancy between the concept Ai and image Ij, and 0 ≤ aij ≤ 1, 1 ≤ i ≤ N and 1 ≤ j ≤ M. Element of weight of concept to image matrix aij is calculated as:

$$ \varvec{a}_{\text{IJ}} = \frac{{\varvec{f}_{\text{IJ}} }}{{\varvec{T}_{\text{J}} }}, $$
(4)

where fij is the frequency of concept Ai in image Ij and Tj is the total number of concepts in image Ij. The weight of the concept to category is a matrix as shown below:

$$ {\mathbf{W}}_{\text{B}} = \left[ {\begin{array}{*{20}c} {\varvec{b}_{ 1 1} } & {\varvec{b}_{ 1 2} } & \cdots & {\varvec{b}_{{ 1 {\text{O}}}} } \\ {\varvec{b}_{ 2 1} } & {\varvec{b}_{ 2 2} } & \cdots & {\varvec{b}_{{ 2 {\text{O}}}} } \\ \vdots & \vdots & {} & \vdots \\ {\varvec{b}_{\text{N1}} } & {\varvec{b}_{\text{N2}} } & \cdots & {\varvec{b}_{\text{NO}} } \\ \end{array} } \right], $$
(5)

where bik is the relevancy between concept Ai and category Bk, and 0 ≤ bik ≤ 1, 1 ≤ i ≤ N and 1 ≤ k ≤ O. The proposed formula for calculating weight of the concept to category bik is as follows:

$$ \varvec{b}_{\text{ik}} { = }\frac{{\mathop \sum \nolimits_{{\varvec{j} = 1}}^{\varvec{M}} \varvec{a}_{\text{ij}} \,\varvec{w}_{\text{kj}} }}{{\mathop \sum \nolimits_{{\varvec{j} = 1}}^{\varvec{M}} \varvec{w}_{\text{kj}} }}, $$
(6)

The weight of the category to image is a matrix as shown below:

$$ {\mathbf{W}}_{\text{CF}} = \left[ {\begin{array}{*{20}c} {\varvec{c}_{ 1 1} } & {\varvec{c}_{ 1 2} } & \cdots & {\varvec{c}_{{ 1 {\text{M}}}} } \\ {\varvec{c}_{ 2 1} } & {\varvec{c}_{ 2 2} } & \cdots & {\varvec{c}_{{ 2 {\text{M}}}} } \\ \vdots & \vdots & {} & \vdots \\ {\varvec{c}_{\text{O1}} } & {\varvec{c}_{\text{O2}} } & \cdots & {\varvec{c}_{\text{OM}} } \\ \end{array} } \right], $$
(7)

where ckj is the relevancy between category Bk and image Ij, and 0 ≤ ckj ≤ 1, 1 ≤ k ≤ O and 1 ≤ j ≤ M. Element of weight of concept to image matrix ai,j is calculated as:

$$ \varvec{c}_{\text{kj}} { = }\frac{{\mathop \sum \nolimits_{{\varvec{j} = 1}}^{\varvec{M}} \varvec{a}_{\text{ij}} \varvec{b}_{\text{ik}} }}{{{\mathbf{F}}_{\text{ik}} }}, $$
(8)

where Fik is the number of concepts in a category.

3.2 Image Retrieval

A user query consists of keywords that can be concepts, categories or combination of concept and category. Table 1 shows the retrieval algorithm.

Table 1. Retrieval algorithm
Full size table

The detail of algorithm is illustrated below through example.

3.3 Walk-Through Example

Let I {I1, I2, I3, I4}, A = {Sky, Foliage, Grass, Water} and B = {Sky_Cloud, Field} be a set of images, concepts and categories. The matrix WCB representing binary weights of a category to an image is defined as:

$$ {\mathbf{W}}_{\text{CB}} = \left[ {\begin{array}{*{20}c} {\mathbf{1}} & {\mathbf{0}} & {\mathbf{1}} & {\mathbf{0}} \\ {\mathbf{0}} & {\mathbf{1}} & {\mathbf{0}} & {\mathbf{1}} \\ \end{array} } \right] $$

The matrix WCI representing the frequency of concepts in images is defined as:

$$ {\mathbf{W}}_{\text{CI}} = \left[ {\begin{array}{*{20}c} {{\mathbf{80}}} & {{\mathbf{20}}} & {{\mathbf{70}}} & {{\mathbf{40}}} \\ {\mathbf{0}} & {{\mathbf{20}}} & {{\mathbf{30}}} & {\mathbf{0}} \\ {{\mathbf{20}}} & {{\mathbf{60}}} & {\mathbf{0}} & {{\mathbf{60}}} \\ {\mathbf{0}} & {\mathbf{0}} & {\mathbf{0}} & {\mathbf{0}} \\ \end{array} } \right] $$

The fuzzy weights in matrices WA, WB, and WCF are computed according to the proposed formulas (4), (6) and (8) and are as follows:

$$ {\mathbf{W}}_{\text{A}} = \left[ {\begin{array}{*{20}c} {{\mathbf{0}}{\mathbf{.8}}} & {{\mathbf{0}}{\mathbf{.2}}} & {{\mathbf{0}}{\mathbf{.7}}} & {{\mathbf{0}}{\mathbf{.4}}} \\ {\mathbf{0}} & {{\mathbf{0}}{\mathbf{.2}}} & {{\mathbf{0}}{\mathbf{.3}}} & {\mathbf{0}} \\ {{\mathbf{0}}{\mathbf{.2}}} & {{\mathbf{0}}{\mathbf{.6}}} & {\mathbf{0}} & {{\mathbf{0}}{\mathbf{.6}}} \\ {\mathbf{0}} & {\mathbf{0}} & {\mathbf{0}} & {\mathbf{0}} \\ \end{array} } \right]\quad \quad {\mathbf{W}}_{\text{B}} = \left[ {\begin{array}{*{20}c} {{\mathbf{0}}{\mathbf{.75}}} & {{\mathbf{0}}{\mathbf{.3}}} \\ {{\mathbf{0}}{\mathbf{.15}}} & {{\mathbf{0}}{\mathbf{.1}}} \\ {{\mathbf{0}}{\mathbf{.1}}} & {{\mathbf{0}}{\mathbf{.6}}} \\ {\mathbf{0}} & {\mathbf{0}} \\ \end{array} } \right] $$
$$ {\mathbf{W}}_{\text{CF}} = \left[ {\begin{array}{*{20}c} {{\mathbf{0}}{\mathbf{.31}}} & {{\mathbf{0}}{\mathbf{.08}}} & {{\mathbf{0}}{\mathbf{.285}}} & {{\mathbf{0}}{\mathbf{.18}}} \\ {{\mathbf{0}}{\mathbf{.18}}} & {{\mathbf{0}}{\mathbf{.15}}} & {{\mathbf{0}}{\mathbf{.12}}} & {{\mathbf{0}}{\mathbf{.24}}} \\ \end{array} } \right] $$

Fuzzy ontology constructed according to the above computed weights is shown in Fig. 2.

Fig. 2.
figure 2

Fuzzy ontology

Full size image

The next step is to take user requirements and retrieval size and apply retrieval algorithm to get the list of retrieved images. If query contains concepts \( Q = \left\{ {A1} \right\} \). From WA following vector is extracted against the given query:

$$ R = \left[ {0.8\;0.2\;0.7\;0.4} \right], $$

The above vector is sorted in descending order \( \left[ {0.8\;0.7\;0.4\,0.2} \right] \) and with the retrieval size of 3, images I1, I3, and I4 will be shown to user.

When a query contains categories \( Q = \left\{ {B1} \right\} \). From WCF following vector is extracted against the given query:

$$ R = \left[ {0.31\;0.08\;0.285\;0.18} \right], $$

The above vector is sorted in descending order \( \left[ {0.31\;0.08\;0.285\;0.18} \right] \) and with the retrieval size of 3, images I1, I3, and I4 will be shown to user.

When query contains both the concepts and categories \( Q = \left\{ {A1,B1} \right\} \). The query is first split into two queries \( Q1 = \left\{ {A1} \right\} \) containing the concepts and \( Q2 = \left\{ {B1} \right\} \) containing the categories. \( Q1 \) will return following vector from WA:

$$ R = \left[ {0.8\;0.2\;0.7\;0.4} \right], $$

and \( Q2 \) will return following vector from WCF:

$$ R = \left[ {0.31\;0.08\;0.285\;0.18} \right] $$

Sort both the vectors in descending order, take the intersection and store the result of intersection in R. Keeping the retrieval size of 3, images I1, I3, and I4 will be selected for user illustration.

4 Experimental Results

A dataset of seven hundred annotated images (i.e. N = 700) about natural scenes has been used to validate the proposed retrieval model. The dataset consists of six categories (i.e. O = 6) namely sky_clouds, forest, field, coast, waterscapes and landscape with mountains and ten concepts (i.e. M = 10) namely sky, foliage, grass, rocks, mountains, trunks, flower, water, sand and fields. The dataset have been used for building the proposed fuzzy ontology. In order to compare our system, we have selected the fuzzy relational ontological model proposed in [23]. The performance of the proposed methodology and reference model have been evaluated in terms of precision, recall, normalized modified retrieval rank (NMRR), average normalized retrieval rank (ANMRR), and normalized discounted cumulative gain (NDCG) [26, 27].

In order to evaluate the proposed methodology total of 167 queries have been used in which 122 queries are based on concepts, 11 queries are based on categories and 34 queries are based on combination of concepts and categories. Figure 3 shows the result of different queries against different evaluation measures. Higher precision values indicate that more relevant results are retrieved than irrelevant results; whereas higher values of recall indicate that most of the relevant results are retrieved. On the other hand, NMRR and ANMRR scores indicate the performance of algorithms based on the ranking of results; lower the value of NMRR and ANMRR better the algorithm ranked the results. NDCG measures the usefulness of images based on its position in a retrieved list. Higher NDCG indicates that the highly relevant images are retrieved at the top of the list.

Fig. 3.
figure 3

Evaluation measure results against different queries consisting of concepts, categories and combination of concept and category. (a) Evaluation results against 122 different queries by concept. (b) Evaluation results against 11 different queries by category. (c) Evaluation results against 34 different queries by combination of concept and category.

Full size image

Figure 4 shows the retrieved results for the proposed methodology and reference model against 3 different queries with retrieval size of 15. The first row shows the results of query-by-concept that is flower. Second row shows the results of query-by-category that is field. Third row shows the results of query-by-concept-and-category that is flower and field.

Fig. 4.
figure 4

Retrieval output of the proposed methodology and reference model for retrieval size 15. Left column: Proposed methodology, Right column: Reference model [23]

Full size image

It is evident from the results that the proposed methodology outperforms the reference model. The proposed methodology shows better results in case of a query-by-concept and query-by-concept-and-category. However, the reference model shows slightly better result in the case of a query by category that is tolerable.

5 Conclusion

In this paper, we have proposed a fuzzy ontology based model for improving the performance of image search results. First a fuzzy ontology is constructed by utilizing the concepts and categories associated with images. Then users are provided with an interface to input keywords. Search results are ranked based on the relevancy between the keywords of a query and images. The advantages of the proposed model are that relationship between an image and concepts and image and categories are fuzzy values that resolve the problem of binary annotation and retrieval. Secondly, it allows an image to belong to different categories with different degrees of membership. Experimental results showed that the proposed model achieves better results as compared to the reference model. Currently we are improving the ranking of the retrieved results where user requirements include multiple concepts and categories.