1 Introduction

The representation of data has been growing in every day. The users represent the data in form of images and store many information in image database. As the image based representation increases, the size of image data base has been growing to different extend. The number of images stored in the database has no limit but the process of populating the required image is highly complex one. For example, the search engines maintains different images in their database and populate them when they requested by the web users. In modern web search, the web user submits a query to the search engine and the search engine produces certain results to the user. Mining the image from huge database can be named as image mining. How the image mining is performed till now is the question here. Now a day, the image mining is performed based on the key word and the spatial information like color values. The popular google search engine indexes the images based on the keywords. When a user submits a query, it produces the image results based on matching of the keyword. This produces higher irrelevant result due to the feature being used.

The problem with the existing approach is, they does not include the semantic meanings and features of natural language. For example, there is no such algorithm which can mine the images from huge image database for the query “car on road” or “Residence Buildings”. The term “residence building”, has many meanings like “hotel”, “house”,”guest house”,”form house”. The semantic meanings of the term residence has not been handled by the previous approaches to produce image results to the user. In earlier approaches the image retrieval has been performed only based on the pixel values. To motivate the image retrieval towards the semantic base, this paper is presented.

Towards the semantic image retrieval, the multi level object similarity measure based approach is presented in this paper. The semantic ontology is the concept of maintaining related terms and features under a common name. For the image retrieval, the object of a class and their features can be represented under a common name of semantic meaning. For example, the meanings of the residence building can be represented under a common name. The features of house, hotel and more can be represented under the common name. By maintaining the spatial features and objects under a common name for the related terms, the problem of image mining can be improved efficiently.

The image would contain number of features like objects, color and so on. Among them, this paper consider the object features which maintains them in multiple levels. For example, for the same class of the semantic, the method groups the features under different class. If the query has been given as “residence with trees”, then it has been considered as different sub class under the residence class, for each sub class under the residence class, the presence of the tree in the object has been considered. Similarly the features has been grouped under different levels. This would improve the performance of image mining and image retrieval.

This paper introduce a Multi level Object Relational Similarity (MORS) measure, which has been computed between different objects of the same class in multiple levels. Based on the similarity of objects in multiple levels the object class has been identified. The detailed approach is discussed in the next section.

2 Related works

There are number of approaches available for the problem of image mining from large image database. This section discuss about some of the methods related to the problem.

Image Mining: A New Approach for Data Mining Based on Texture [1], discuss that Image data mining can be done manually by slicing and dicing the data until a pattern becomes obvious. Or, it can be done with programs that analyse the data automatically. Colour, texture and shape of an image have been primitive image descriptors in Content Based Image Retrieval (CBIR) system. Primitive features of an image used to identify and retrieve closely matched images from an image database. It is very difficult to extract images manually from image database because they are very large. This paper presents a novel framework for texture information of an image and achieves higher retrieval efficiency than the shape features of an image. There is a trade-off between accuracy and computational cost. The trade-off decreases as more efficient algorithm is used to solve the problem and increases the computational power and will decreases the cost of the whole system as well.

An improved and efficient image mining technique for classification of textual images using low-level image features [2], proposes a system that classifies textual images (images that encounter text within) using low-level image features. Image classification and content based image retrieval is a growing field in the area of image classification. In this paper, the approach is based on various low-level image features including GLCM features like mean, skewness, energy, contrast, homogeneity. Using these various features, the differences between images are measured, and then these are used to classify the textual images by performing classification and clustering techniques on datasets. The proposed method experimented on 60 different textual images to obtain an improved result that was not obtained in earlier systems along with classification of images in three main categories: document, scene and caption.

Multi-class Enhanced Image Mining of Heterogeneous Textual Images Using Multiple Image Features [3], proposes an enhanced image classifier to extract patterns from images containing text using a combination of features. Image containing text can be divided into the following types: scene text image, caption text image and document image. A total of eight features including intensity histogram features and GLCM texture features are used to classify the images. In the first level of classification, the histogram features are extracted from grayscale images to separate document image from the others. In the second stage, the GLCM features are extracted from binary images to classify scene text and caption text images. In both stages, the decision tree classifier (DTC) is used for the classification. Experimental results have been obtained for a dataset of about 60 images of different types. This technique of classification has not been attempted before and its applications include preprocessing for indexing of images, for simplifying and speeding up Content Based Image Retrieval (CBIR) techniques and in areas of Machine Vision.

Unsupervised Learning for Image Classification based on Distribution of Hierarchical Feature Tree [4], present a novel unsupervised model for the image classification based on feature’s distribution of particular patches of images. Our method firstly divides an image into grids and then constructs a hierarchical tree in order to mine the feature information of the image details. According to our definition, the root of the tree contains the global information of the image, and the child nodes contain detail information of image. We observe the distribution of features on the tree to find out which patches are important in term of a particular class.

In [5], the author attempts to provide a comprehensive survey of the recent technical achievements in high-level semantic-based image retrieval. Major recent publications are included in this survey covering different aspects of the research in this area, including low-level image feature extraction, similarity measurement, and deriving high-level semantic features. We identify five major categories of the state-of-the-art techniques in narrowing down the ‘semantic gap’: (1) using object ontology to define high-level concepts; (2) using machine learning methods to associate low-level features with query concepts; (3) using relevance feedback to learn users’ intention; (4) generating semantic template to support high-level image retrieval; (5) fusing the evidences from HTML text and the visual content of images for WWW image retrieval.

Towards the improvement of textual anatomy image classification using image local features [6], tackles the anatomy image classification problem using a hybrid approach. First, a mutual information (MI) based filter is applied to select a set of terms with top MI scores for each anatomical class and help reduce the noise existing in the raw text. Second, local features extracted from the images are transformed as visual descriptors. Last, a hybrid scheme on the results from the textual and visual methods is applied to achieved further improvement of the classification results. Experiments show that this hybrid scheme improves the results over the sole textual or visual method on different anatomical class settings.

Improving the classification of an industrial document image database by combining visual and textual features [7], present method for classifying document images by combining textual features extracted with the Bag of Words (BoW) technique and visual features extracted with the Bag of Visual Words (BoVW) technique. The BoVW is widely used within the computer vision community for scene classification or object recognition but few applications for the classification of entire document images have been submitted. While previous attempts have been showing disappointing results by combining visual and textual features with the Borda-count technique, we’re proposing here a combination through learning approach.

Image-Text Cross-Modal Retrieval via Modality-Specific Feature Learning [8], a novel model based on modality-specific feature learning is proposed. Considering the characteristics of different modalities, the model uses two types of convolutional neural networks to map the raw data to the latent space representations for images and texts, respectively. Particularly, the convolution based network used for texts involves word embedding learning, which has been proved effective to extract meaningful textual features for text classification. In the latent space, the mapped features of images and texts form relevant and irrelevant image-text pairs, which are used by the one-vs-more learning scheme. This learning scheme can achieve ranking functionality by allowing for one relevant and more irrelevant pairs. The standard back-propagation technique is employed to update the parameters of two convolutional networks.

Efficient Image Classification using Data Mining [9], propose skin detection method and some realized works in this area. Thus, we will detail the methods proposed for the face detection and the related studies found in literature. We will finish, by showing and analyzing the results obtained from tests made on a base of colored images with a complex background and make comparison to similar works which demonstrate that our algorithm achieve good accuracy.

Research on image classification based on a combination of text and visual features [10], extracted three kinds of features, including global visual features, local visual features, and text features using both the image content and context. Then, we tried various feature combination methods and train classifiers for each kind of feature vector. Finally, we used a classifier fusion strategy based on weight learning, combining classifier outputs together, and we obtained the category of unlabeled images.

Machine Learning and Content-Based Multimedia Retrieval [11], proposed an enhanced relevance-feedback method to support the user query based on the representative image selection and weight ranking of the images retrieved. The support vector machine (SVM) has been used to support the learning process to reduce the semantic gap between the user and the CBIR system. From these experiments, the proposed learning method has enabled users to improve their search results based on the performance of CBIR system.

Tree Representation and Feature Fusion Based Method for Multi-Object Binary Image Retrieval [12], proposes an effective solution for multi-object binary images retrieval by fusing several features. We first employ a Tree Representation Model (TRM) to describe the topology structure of multi-object binary images. Secondly, we propose two new descriptors to describe the density and the spatial location feature of the objects, respectively. In addition, we combine two descriptors and shape feature to distinguish the difference of the image objects. Finally, the similar matching algorithm based on TRM is given and applied to trademark database retrieval.

The personalized web search has been a highlighted topic at any time. In [13], a dual mode search framework is presented. It works with the user profile which has been build from user details. Then the results from the search engines are reranked according to their own measures. The results are produced to the user according to their own strategy. The method searches based on the input inform of images and text. Also the users are grouped according to the interest score.

In [14,15], the performance of different descriptors of image has been analyzed towards disease image retrieval. The usage of combined image descriptors would improve the performance of similar crop disease image retrieval system. The method estimates the image similarity based on the combined descriptors which have been generated from the images. This helps to identify the similar images of disease crop from the large image database.

All the methods suffers to produce relevant results on image mining and suffers with higher irrelevant results.

3 Semantic image mining using multi level object similarity

The proposed multi level object similarity based image mining algorithm extracts the object features from the input image and uses the semantic meanings of the image from the domain ontology. At the training stage, the method computes the object similarity measure at different level of the same class to identify the subclass of the image. Similarly in the testing phase, the method computes the semantic similarity for different semantic classes available on the ontology. Based on the similarity measure, a subset of image has been returned as result by the proposed algorithm. The detailed approach is discussed in this section.

The Fig. 1, shows the architecture of MORS based image mining algorithm and shows various functional stages involved.

Fig. 1
figure 1

Architecture of MORS image mining technique

Full size image

To start with the image mining, it is necessary to consider the features of the image being used to perform image mining. In this approach, we considered the object features for the classification and the semantic ontology about various concepts. It is necessary to view the model semantic ontology being used for classification.

The above segment shows the sample ontology being considered for the development of image mining algorithm. The ontology contains different terms of residence where each of them intend for boarding and lodging. Against each property, the features from the image can be extracted and stored in the database. The proposed algorithm has the following stages.

4 Feature extraction

The feature extraction is the process of identifying the object features present in the image. In this stage, the method identifies various features present in the image by mapping them with object mapping. The object map has been generated from the domain ontology which contains different features of the objects of various classes. The method extracts the features of the image and maps with different objects available in the object map. Based on the object similarity, the method identifies the object being present in the image.

The above discussed algorithm extracts the objects being present in the image and extracts their names and features from the image.

5 MORS estimation

The multi level object relational similarity measure represent the similarity of the object towards the objects and features of a domain ontology. The domain ontology would contain number of classes and sub classes. At each sub class, there would be number of object features or images. For example, for the image to be named as car, it must contain a car object first. Similarly there will be number of features to be present for an image to be indexed under a class. Based on this, the MORS measure is estimated based on the similarity of the objects of the class with the input object set. At each level of the semantic, the method computes the object similarity and finally a cumulative MORS measure has been estimated.

The above discussed algorithm computes the multi level object relational similarity measure for the input feature being extracted from the input image. Based on the MORS measure estimated, the method identifies the class of the input image.

6 Image mining

The image mining is the process of extracting the relevant images from the image database considered. When the user enters the input query, the method first identifies the semantic class of the query. Then method computes the semantic relevancy for different class of domain ontology. Based on the semantic relevancy, a specific class has been identified. Based on the identified class, the method identifies the list of image objects from the database which are more relevant to the input query.

The above discussed algorithm computes the multi level semantic similarity for different class. Based on the estimated measure, a subset of image has been selected as the result to the user.

7 Results and discussion

The proposed algorithm has been implemented and evaluated for its efficiency in various parameters of image mining. The proposed MORS image mining algorithm has been implemented using Matlab. The method has been validated with different image databases which are available online. The method has produced higher results on image mining and increases the relevancy also.

The Table 1, shows the data set being used to evaluate the performance of proposed algorithm. For evaluation purpose, we have used both online dataset as well as cooked data set. The cooked data set has been framed by collecting different semantics and images available. Using the cooked data set, the proposed algorithm has been evaluated for its efficiency.

Table 1 Details of data set
Full size table

The Fig. 2, shows the comparison on image retrieval accuracy produced by different methods. The result shows that the proposed algorithm has produced higher retrieval accuracy than other methods.

Fig. 2
figure 2

Comparison on retrieval accuracy

Full size image

The Fig. 3, shows the comparison on time complexity produced by different methods. The result shows clearly that the proposed method has produced less time complexity than other methods.

Fig. 3
figure 3

Comparison on time complexity

Full size image

The Fig. 4, shows the comparison result on irrelevancy produced by different methods. The result shows clearly that the proposed method has produced fewer irrelevancies than other methods.

Fig. 4
figure 4

Comparison on irrelevant ratio

Full size image

8 Conclusion

In this paper, an efficient multi level object relational similarity based image mining algorithm has been presented. The method maintains number of semantic classes where each has different properties and there will be number of entries according to the number of properties considered. The images has been indexed into number of sub classes according to the object features available and how they deviates. For indexing, the method extracts the features and computes the multi level object similarity on each sub class of domain ontology. Based on the MORS value the image has been indexed. Similarly for retrieval, the input query has been used to compute the multi level semantic similarity measure. Based on the semantic similarity measure a single class has been identified and the images of the class have been returned as the result. The proposed algorithm produces efficient results in image retrieval and reduces the false ratio or irrelevancy.