RETRACTED ARTICLE: Customer behavior analysis using Naive Bayes with bagging homogeneous feature selection approach

Abstract

The significant success of an organization greatly depends upon the consumers and their relationship with the organization. The knowledge of consumer behavioral and a excellent understanding of consumer expectations is important for the development of strategic management decisions in support of improving the business value. CRM is intensively applied in the analysis of consumer behavior patterns with the use of Machine Learning (ML) Techniques. Naive Bayes (NB) one of the ML supervised classification models is used to analyze customer behavior predictions. In some domain, the NB performance degrades which involves the existence of redundant, noisy and irrelevant attributes in the dataset, which is a violation of underlying assumption made by naive Bayes. Different enhancements have been suggested to enhance the primary assumption of the NB classifier-independence assumption between the attributes of given class label. In this research, we suggest a simple, straight forward and efficient approach called BHFS (Bagging Homogeneous Feature Selection) which is based upon Ensemble data perturbation feature selection methods. The BHFS method is applied to eliminate the correlated, irrelevant attributes in the dataset and selecting a stable feature subset for improving performance prediction of the NB model. The advantage of the BHFS method requires less running time and selects the best relevant attributes for the evaluation of naive Bayes. The Experimental outcomes demonstrate that the BHFS-naive Bayes model makes better predictions compared to the standard NB. The running time complexity is also less with BHFS-NB since the naive Bayes is constructed using selected features obtained from BHFS.

1 Introduction

In the advanced competing business environment, the success of an enterprise greatly depends upon its service and the product offered to the customers. Analyzes of customer data helps to gain insights about the potential customer within the enterprises and based upon analyzes, helps to develop new business strategies to boost the business and create new customer acquisitions and retaining customers (Christry et al. 2018). Developing a business strategy or practices by analyzing data can be achieved with the use of CRM. Customer Relationship Management is a business technique that handles and analyzes customer data within an enterprise using advanced technology and automates the business process (Payne and Flow 2005) and also helps in better turnover, information can be accessed easily and understanding customer patterns (Mithas et al. 2006). The collected data and interactions from the customer are used to analyze and transform into valuable information, in turn, to make managerial decisions. The decision, in turn, provides opportunities for new customer, increase profitability and sales growth. The key facets of CRM are customer satisfaction that includes service quality, handling customers and service access. The CRM or customer analytics process is performed to various reason that includes customer segmentation, profitability analysis, predictive modeling, compute customer service and event monitoring (Christry et al. 2018). The predictive modeling in CRM analytics trends to evaluate the current and historical customer data to find insights about the current and future forecasts (Soltani et al. 2018). This predictive analytics constantly makes to set new business objectives and actions with future outcomes. To proceed with the predictive approaches the use of ML techniques has a significant role.

ML is a study of algorithms that comes under the field of Artificial Intelligence gives the ability to the system to learn automatically from the experience. Naive Bayes (Idiot Bayes and Simple Bayes) straight forward probabilistic induction classifier that is simple, efficient, works in linear running time and performs effectively in diverse classification problems (Abellan and Castellano 2017; Frank et al. 2002). The classifier is robust to noise and missing data, with a limited number of data that could be used for learning (Bakar et al. 2013).

Consider the Learning set $T$ with $n$ instances and with input variables $X = \left\{ {x_{1} ,x_{2} , \ldots ,x_{d} } \right\}$ and the associated class label $Y = \left\{ {y_{1} ,y_{2} , \ldots ,y_{j} } \right\}$. NB aims to predict $y$ class label by using the new sample $x_{i}$,

$$y = argmax_{y} \left( {P\left( {y |x_{i} } \right)} \right)$$

(1)

Based on the central assumption of NB-conditional independence

$$y = \arg \max_{y} \left( {P\left( y \right)\mathop \prod \limits_{i = 1}^{n} P\left( {\left. {x_{i} } \right|y} \right)} \right.$$

(2)

Naive Bayes makes two imperative assumptions with the datasets. One is Independence between the features (that is input features should not be correlated) and the second one is all input attributes in the datasets are equal. The assumption made by NB is grossly violated in some domain datasets due to the existence of correlated attributes (Ratanamahatana and Gunopulos 2003) and also, the existence of missing and Noisy features in dataset causes the NB to perform poorly in prediction (Domingos and Pazzani 1997). A different technique has been adopted to enhance better performance of NB and to reduce unpractical assumptions. Many researchers have made more attention to improving the goodness of the model and tired of using various methodologies with naive Bayes.

In this research, we suggest a simple, straightforward and more efficient strategy for improving the performance prediction of the NB classifier. The Bagging Homogeneous Feature Selection (BHFS) is based upon ensemble data perturbation feature selection procedure, which uses the merits of the bagging and filters FS approach. The BHFS uses bagging to generated $t = \left\{ {t_{1} ,t_{2} ,t_{3} , \ldots .,t_{n} } \right\}$ learning subsets from the original learning set $T$ and applies a filter FS method to rank the attributes accordingly to relevance with the class label. Then, the BHFS method uses different aggregation techniques to combine the attribute ranking list obtained from $\{ FL_{1} ,FL_{2} ,FL_{3} , \ldots .,FL_{n} \}$ into single attribute ranking list and uses different threshold values to select the attributes from the final ranking list $FS_{enl}$ for constructing naive Bayes. The use of the BHFS method enhances the stability in FS method and improves the performance prediction of the NB model. Stability analyzes are performed to check whether feature selection applied to different learning subsets yield similar results. Experimental is constructed using client datasets from the UCI and the results of BHFS-NB and standard NB are compared using the validity scores.

2 Related work

To improve the primary assumption of Naive Bayes different methodologies are proposed and experimented. From a different approach, the methods applied can be spitted into two types. One is based upon relaxing the independence assumption made by NB and another one is involving the use of the feature or attribute selection techniques a preprocessing method to select the features which are dependent with the class label and independent with the other input features (Kononenko 1991). Proposed SNB (“Semi-Naive Bayes”) model- the methodology checks the attributes with the dependencies. Then attributes that have dependencies are joined using Chebyshev Theorem. The procedure has experimented with four medical datasets (Primary tumor, Thyroid, Rheumatology and Breast cancer). The experimental analyses indicate primary tumor and Breast cancer dataset got the same results and where Rheumatology and Thyroid datasets got improved results. Combining the attributes number of parameters increases and computational time also affects. Pazzani (1996): Applied FSSJ (“Forward Sequential Selection and Joining”) and BSEJ (Backward Sequential Elimination and joining)—the methods to join the attributes which have dependencies, by searching the pair features with the dependencies. Given three attributes $A_{1} ,A_{2}$ and $A_{3}$

$$P\left( {A_{1} = \left. {V_{{1_{j} }} } \right| C_{i} } \right)P\left( {A_{2} = \left. {V_{{2_{j} }} } \right| C_{i} } \right)P\left( {A_{3} = \left. {V_{{3_{j} }} } \right| C_{i} } \right)P\left( {C_{i} } \right)$$

(3)

If there are dependencies between the $A_{1} \& A_{3}$ and $A_{2}$ is not relevant, then attributes $A_{1} \& A_{3}$ are joined as

$$P\left( {A_{1} = V_{{1_{j} }} \& A_{3} = \left. {V_{{3_{j} }} } \right|C_{i} } \right)P\left( {C_{i} } \right)$$

(4)

The Experiment is tested using datasets acquired from UCI and results show accuracy increases and from the two methods, BSEJ performs better than FSSJ. Friedman et al. (1997): Proposed TAN (“Tree Augmented Naive Bayes”) method, which uses the tree structure model imposed in the NB structure. To build a tree structure, the features of the parent must be selected and the correlation between variables should be measured. Add the edges (which are correlated) between the variables. To use continuous variables then the features should be prediscretized. The results are compared with C4.5, wrapper feature methods, and NB models. Friedman (1998): applied the enhanced version of TAN to overcome the problem with the continuous variables. By using parametric (with Gaussians method) and semi-parametric (with Gaussians mixture methods). The procedure is tested using UCI datasets. Keogh and Pazzani (1999): Proposed the SP-TAN(Super Parent TAN) an revamped version of TAN. Follow the same method of TAN but differs in choosing the direction links and criteria to build the parent function. Space and time complexity are the same in both TAN and SP-TAN. Zheng and Geoffrey (2000): propose the LBR (Lazy Bayes Rule) method which is comparably similar to LazyDT. Webb (2005): Proposed—“Aggregating One-Dependence Estimators” (AODE)—To minimize the computational complexity of LBR and SP-TAN and to overcome the conditional independence of NB AODE is proposed. The average of all dependencies estimation is carried to overcome independence assumption and computational complexity is improved with compare to LBR and SP-TAN. Langley and Sage (1994): applied forward selection procedure which employs greedy search methods to find the feature subset. By excluding the redundant features and electing the important features trends to improve the prediction accuracy. The procedure is tested using UCI datasets and results are compared with the Naive Bayes and C4.5 model. The results pattern shows the classifier prediction can be improved using the selected features. Ratanamahatana and Gunopulos (2003): applied the Selective Bayesian Classifier to select the features using C4.5 DT and in turn applied the select feature set to construct the NB model. The test is conducted on 10 UCI datasets and NB gets better accuracy with using the SBC procedure. Fan and Poh (2007): used the preprocessing procedures to improve the NB classifier. Three procedures have been employed PCA, ICA and Class-conditional ICA to make independence assumptions true. The experimental results conducted using UCI data. Bressan and Vitria (2002): Class-conditional ICA(CC-ICA) method proposed to carry out the preprocessing strategies for NB and results shows better prediction is obtained. Karabulut et al. (2012): The authors makes a study on use of variable selection to minimizes the dimensions of dataset and to see the effect of improving performance accuracy in classifier. Six different attribute selection are considered and four different classification model are applied. The experiment is conducted using 15 different datasets obtained from UCI and results shows there is improvement in the accuracy. Rahman et al. (2017): The authors applies feature selection methods to enhance the prediction of the model in students academic performance. In this research information gain and wrapper attribute method and NB, DT, ANN classifier are applied. Omran and El Houby (2019): The author predicate the problem of electrical disturbances by applying ML Model. The method uses ant colony attribute selection method and five different ML model are considered. The experimental procedure is conducted using electrical disturbances open source dataset and depending upon the classifier model the prediction accuracy is improved till 86.11. Moslehi and Haeri (2020): The performance of classifier can be enhanced by removing unnecessary attributes in datasets and which can be carried by using feature selection. The author applies a new hybrid variable selection method in which wrapper and filter methods are applied. The experiment is carried out using 5 datasets and results reveals there is better classification accuracy.

3 Feature selection

Consider a Learning set $T$ consists of $\left\{ {\left( {y_{n} ,x_{n} } \right)} \right\}$ where ($n = 1, \ldots , N)$ $y$ denotes the output label or Output variable and $x$ points out the input attributes. Now by using the learning set to form a NB classifier $\varphi \left( {x,T} \right)$, were $x$ is the input variables which predicate $y$ using $\varphi \left( {x,T} \right)$. The intention is to obtain maximum accuracy prediction and to get detail insight of learning set $T$. In the learning set $T$ due to existence of noisy, irrelevant and correlated attributes which induce high computational cost and prediction performance degrades (Kononenko 1991; Pandey et al. 2020). In such cases involving the feature selection, a preprocessing step is encouraged. FS is a crucial process in machine learning classifiers which trends to identify the important attributes in the datasets. By using evaluation criterion or searching strategy helps to identity very important feature subset which is hugely correlated the with class label and maximize the prediction of the NB classifier. FS lineup with multiple benefits such as enhancing classifier performance, reducing over fitting, minimizing the learning cost, getting better insights of processes by the data and using only selected features (Saeys et al. 2008; Pes 2019). FS trends to improve the classification prediction accuracy and eliminating such attributes will lead to reducing the learning algorithm running time (Huan and Yu 2005). FS can be categorized into wrapper, filter and embedded method. The filter method works by using some statistical method to rank the input variables accordingly to class label and works fast. But the wrapper uses some ML classifier to select best attribute set, but the method works slow (need high computational resources) compare to filter method. In this research, filter methods are considered, since it requires less time complexity and works fast.

Feature selection can be summarized from various perspectives into one as: given the dataset $D = \left\{ {x_{1} , \ldots ,\left. {x_{n} } \right|y_{n} } \right\}$ with $x$ input variables and $y$ class label. The feature selection should be idealized (identity minimum attribute subset that is enough to class target concept), Classical, Improving prediction(improving the classifier prediction using only selected subset features) and approximating same distribution(the selected features are close to original same class distribution) (Dash and Liu 1997). A new ensemble learning paradigm based FS is studied. This mechanism is based on integrating the ensemble methods and feature selection $FS_{enl}$. The motivation to focus ensemble methods is inspired by better performance gained in supervised learning and also trends to enhance the stability of $fs$ (Donghai et al. 2014; Yu and Lin 2003). Ensemble learning is based upon combining the results of sequence algorithm $FS_{enl} = \{ FS_{1} ,FS_{2} ,FS_{3} , \ldots , FS_{n}$} into single algorithm output such that reducing in bias, variance and improving prediction accuracy. The aggregated results $FS_{enl}$ obtained from the ensemble method are more reliable, stable and accurate when compared to the single model. This process leads to better enhanced performance prediction when compared with single models. The ensemble is more decisive than the single model and overcomes the local optima with the individual feature selection. Simple averaging, bagging, stacking, and boosting belongs to ensemble method. In this research bagging (ensemble method) is applied.

4 Bagging homogeneous feature selection (bhfs)

In ensemble, there are different methods in which our study uses bagging (bootstrapping and Aggregation) based ensemble methods. Integrating the ensemble method to feature selection $FS_{enl}$ is based upon heterogeneous and homogeneous approach. If the feature selectors are same type then it is referred to homogeneous, otherwise with different feature selectors refer to heterogeneous. In our study, a homogeneous methodology is studied. Homogeneous is also referred to data (instance) perturbation. The same feature selectors are applied to various subsets samples derived from the learning set $T$ (Seijo-Pardo et al. 2016).

In BHFS approach consists of following procedure (1) Bootstrap process (Generating $\left\{ {t_{1} ,t_{2} ,t_{3} , \ldots .,t_{n} } \right\}$ different subset from the learning set $T$), (2) Apply feature selectors and aggregation of results(Apply same feature selectors to different generated $\left\{ {t_{1} ,t_{2} ,t_{3} , \ldots .,t_{n} } \right\}$ subset samples and aggregate the multiple outputs into single one $FS_{enl}$), (3) Setting Threshold value(Based upon the threshold value feature subset are selected from the $FS_{enl}$).

Bagging (Bootstrap aggregation) simple meta-algorithm ensemble learning method which helps in reducing the variance and to enhance the prediction and stability of the feature selection. Bagging avoids over fitting for the unstable procedure. Bagging trends to get insights about various variance and biases and achieves better performance by combining the multiple independent weak learners into a single strong learner using the aggregation process. Bagging has two steps one is creating $t = \left\{ {t_{1} ,t_{2} ,t_{3} , \ldots .,t_{n} } \right\}$ bootstrap samples from the original set $T and$ then applying a diverse set of feature selectors to $t = \left\{ {t_{1} ,t_{2} ,t_{3} , \ldots .,t_{n} } \right\}$ and aggregation them into single feature selector $FS_{enl}$ = aggregation $\left( {FL_{i} } \right), where FL_{i} = \{ FL_{1} ,FL_{2} ,FL_{3} , \ldots .,FL_{n}$}

4.1 Bootstrap procedure

• 1. Consider learning set $T$ with $n$ instances $= \left\{ {x_{1} , \ldots ,\left. {x_{n} } \right|y_{n} } \right\}$

• 2. Initialize $t = \left\{ {t_{1} ,t_{2} ,t_{3} , \ldots .,t_{n} } \right\}$ as empty learning subset.

• 3. Repeat $n$ times

• 4. Randomly with replacement select $n$ instances from $T$

• 5. Add $n$ to $t_{1}$ (Repeat the process up to $t_{n}$ times)

• 6. Output: Generated Learning subsets $t = \left\{ {t_{1} ,t_{2} ,t_{3} , \ldots .,t_{n} } \right\}$

In the bootstrap procedure, consider the learning set $T$ consists of $n$ instances $= \left\{ {x_{1} , \ldots ,\left. {x_{n} } \right|y_{n} } \right\}$ where $x$ are set of input predictors and $y$ target class. Then create the empty learning subsets $= \left\{ {t_{1} ,t_{2} ,t_{3} , \ldots .,t_{n} } \right\}$, with the random sampling with the replacement select $n$ instances from $T$ and add to $t_{1}$ and repeat the procedure until $t_{n}$ learning subset is generated.

4.2 Applying feature selectors and aggregation procedure

Input $T$—Learning set with $t = \left\{ {t_{1} ,t_{2} ,t_{3} , \ldots ,t_{n} } \right\}$ learning subset generated by applying bootstrap procedure. $fs$ feature selection

$th$—Threshold values (no of features to be selected)

• 1. From the generated learning subset $t = \left\{ {t_{1} ,t_{2} ,t_{3} , \ldots ,t_{n} } \right\}$ (4.1 Bootstrap procedure step 6)

• 2. for $\left( {i = 1,2, \ldots ,n} \right)$ do

• 3. 3. $FL_{i} = fs\left( {t_{i} } \right)$ [Feature selectors using ranking]

• 3.1 Initialize Feature List $FL_{i} = \left\{ \right\}$

• 3.2 For each attribute $x_{i} where i = 1, \ldots ,n$ from $t_{i}$ do

• 3.3 $m_{i}$ = Compute $\left( {x_{i} , fs} \right)$ where $fs = feature selection method$ using ranking

  • 3.4 Position $x_{i}$ into $FL_{i}$ according to $m_{i}$

  • 3.5 End for

  • 3.6 Return $FL_{i}$ in decreasing or Ascending order of relevant features

• 4. End For

• 5. $FS_{enl}$ = aggregation $\left( {FL_{i} } \right), where FL_{i} = \{ FL_{1} ,FL_{2} ,FL_{3} , \ldots .,FL_{n}\}$

• 6. $FS_{{enl\left( {th} \right)}}$ = Select top set features from $FS_{enl}$

• 7. Build classifier NB with the $FS_{{enl\left( {th} \right)}}$ (using selected features)

• 8. Obtain classification prediction accuracy and error rate

With the standard learning sets $T$, using bootstrap process sequence of learning subsets are generated $t = \left\{ {t_{1} ,t_{2} ,t_{3} , \ldots ,t_{n} } \right\}$. Assume one feature selector $\left( {fs} \right)$ method, the $\left( {fs} \right)$ used are based on ranking the attributes accordingly to their relevance. From bootstrap learning subsets $t_{n} ,$ the feature selector $fs$ is applied to each generated learning subset and end up with ranking the features. For each bootstrap sample from $\left\{ {t_{1} ,t_{2} ,t_{3} , \ldots ,t_{n} } \right\}$ one feature selector with rank list is generated. Therefore for one feature selector, there will be $n$ ranked lists $\{ FL_{1} ,FL_{2} ,FL_{3} , \ldots .,FL_{n} \}$. Then by using aggregation methodology the $n$ ranked lists is aggregated into $FS_{enl}$ list. The procedure is applied for single feature selector and same can be carried to multiple feature selectors.

4.3 Aggregation function

Aggregation function combines the output from multiple feature selectors based on learning subsets into a single output. Based on the outcome the feature selectors it can be further categorized to three types. Feature Weighting, Ranking, and subset. Feature selectors used in this study are based on the ranking method and so our focus is aggregation based on feature ranking methodology. For one feature selector, there will be $n$ ranked lists $\{ FL_{1} ,FL_{2} ,FL_{3} , \ldots .,FL_{n} \}$, then by using aggregation methodology the $n$ ranked lists are aggregated into $FS_{enl}$ list.

There are various combination techniques are available and this study uses Mean, Median, Geomean and Minimum methods (Seijo-Pardo et al. 2016; Bolon-Canedo and Alonso-Betanzos 2018).

Mean: $FS_{enl} =$ $\frac{1}{n}\mathop \sum \limits_{i = 1}^{n} FL_{i}$ $\{ F_{1} ,F_{2} ,F_{3} , \ldots .,F_{n} \}$ by total $n$.

Median: $FS_{enl} =$ Median $\left\{ {FL_{1} ,FL_{2} ,FL_{3} , \ldots ,FL_{n} } \right\}$

GeoMean: $FS_{enl} =$ $(\mathop \prod \limits_{i = 1}^{n} (FL_{i} ))^{1/n}$ $\sqrt[n]{{FL_{1} FL_{2} FL_{3} \ldots .FL_{n} }}$.

Min: $FS_{enl} =$ Min $\left\{ {FL_{1} ,FL_{2} ,FL_{3} , \ldots ,FL_{n} } \right\}$

4.4 Threshold values

The feature selection techniques applied will rank the features accordingly to relevance. The need of cutoff value is necessary to select the optimal feature set from the final $FS_{enl}$. In this research, we have applied different threshold value to select the features subset (Seijo-Pardo et al. 2016; Bolon-Canedo and Alonso-Betanzos 2018).

$\log_{2} \left( \varvec{n} \right)$: Using $\log_{2} \left( n \right)$ criteria choose the relevant feature subset. $n$ denotes no of features in the ordered final ranking.

10 percentage The top 10 percentage features are selected are considered for model construction from the ordered final ranking $FS_{{enl\left( {th} \right)}}$

25 percentage The top 25 percentage features are selected are considered for model construction from the ordered final ranking $FS_{{enl\left( {th} \right)}}$

50 percentage The top 50 percentage features are selected are considered for model construction from the ordered final ranking $FS_{{enl\left( {th} \right)}}$

4.5 Feature selectors

There are an array of feature selectors are available in practice, but for this study, we have chosen four filter-based feature selectors. The filter FS techniques are faster, scalable, algorithm independent and great computational compare to the wrapper techniques. Filter Method elects the $m$ subset features from the original $n$ features which maintain the relevant information as in the whole feature set. In the filter method, the evaluation of relevance variables score fully dependent upon the data and its properties and independent of any induction algorithm. In the case of large dimensional datasets, the use filter method is encouraged with low computation time and no data over fitting issues. The features with the low score are eliminated and features trends to have high features are considered as input for model construction. The selection of high features score is carried through the use of threshold values (Huan and Yu 2005).

4.5.1 Chi square

Chi square is based upon statistical test to compute the dependency between two variables. The method compute scores between each variables with the output label and rank the attributes accordingly to the relevance. If the class label and attribute variables is independent, then less score is assigned otherwise high is assigned. The features with top relevant rank are considered for the algorithm by assigning some threshold values.

Consider the two variables of data, the Chi square compute the expected frequency and observed frequency using

$$x^{2} = \frac{{\left( {observed frequency - Expected frequency } \right)^{2} }}{Expected frequency}$$

(5)

$x^{2}$ with high rank is taken as better features

4.5.2 ReliefF

ReliefF (“Kononenko et al. 94”) is heuristic and instance based method which deals with noisy, multi class problem and incomplete data and it is revamped version of Relief. ReliefF belongs to filter type FS method. Consider ${\text{D}}$ as the dataset the instance $x_{1} ,x_{2} , \ldots ,x_{n}$ with the attribute of vector ${\text{Y}}_{\text{i}} , {\text{i}} = 1, \ldots , {\text{a}}$, and ${\text{a}}$ number of features and with class label ${\text{A}}_{\text{i}}$. Compute quality estimation of ${\text{W}}\left[ {\text{A}} \right]$ using ${\text{H}}_{\text{j}} - {\text{hits}}$, ${\text{M}}_{\text{j}} - {\text{miss}}$ and ${\text{R}}_{\text{i}}$

$$W\left[ A \right]\text{ := }W\left[ A \right] - \mathop \sum \limits_{j = 1}^{k} \frac{{diff\left( {A,R_{i} H_{j} } \right)}}{m.k} + \mathop \sum \limits_{{c = class\left( {R_{i} } \right)}} \frac{P\left( C \right)}{{1 - P(class(R_{i)} }}\mathop \sum \limits_{j = 1}^{k} diff\left( {A,R_{i} ,M_{j} \left( C \right)} \right)]/\left( {m.k} \right)$$

(6)

Select the features having higher values (Robnik-Sikonja and Kononenko 2003)

4.5.3 Symmetrical uncertainty (SU)

SU is filter based FS approach which compute the fitness of the attributes with the class label. SU compute the uncertainty in the variable using information theory of entropy (Huan and Yu 2005). The entropy of feature $X$ is computed as

$$H\left( X \right) = - \sum P\left( {x_{i} } \right)\log_{2} (P\left( {x_{i} } \right)$$

(7)

The entropy of $X$ after checking another feature $Y$ is computed as

$$H\left( {X\left| Y \right.} \right) = - \sum\nolimits_{j} {P\left( {x_{i} } \right)} \sum\nolimits_{i} {\left( {x_{i} \left| {y_{j} } \right.} \right)\log 2} \left( {x_{i} \left| {y_{j} } \right.} \right)$$

(8)

$P\left( {x_{i} } \right)$ denotes prior probabilities of $X$ and $P(x_{i} /y_{j}$) denotes posterior probabilities $X$ given value $Y$.

The IG is computed as

$$IG\left( {\left. X \right|Y} \right) = H\left( X \right) - H\left( {\left. X \right|Y} \right)$$

(9)

IG is symmetrical for $X \& Y$ random variables. Symmetry computes the correlation between variables is desired property, but IG is biased towards the attributes with large values. So, SU for information gain for the features with large values are normalizes the value range between [0,1]

$$SU\left( {X,Y} \right) = 2 \frac{{IG\left( {X/Y} \right)}}{H\left( X \right) + H\left( Y \right)}$$

(10)

SU values lies between [0,1]. The feature with high values 1 indicate the correlated with target class, otherwise 0 uncorrelated with target class.

4.5.4 Gain ratio

GR is a filter based attribute selection technique and it is enhanced version of IG which minimize its bias and consider the size and number of branches, while selecting a attribute.GR is measured by

$${\text{Gain Ratio}} = \frac{{{\text{Gain}}\left( {\text{attribute}} \right)}}{{{\text{split info}}\left( {\text{attribute}} \right)}}$$

(11)

The attribute with max gain ratio is taken as splitting feature. Split information of an attribute is computed using

$$split info\left( D \right) = - \mathop \sum \limits_{j = 1}^{v} \left( {\frac{{\left| {D_{j} } \right|}}{\left| D \right|}} \right)\log_{2} \left( {\frac{{\left| {D_{j} } \right|}}{\left| D \right|}} \right)$$

(12)

Gain for an attribute is computed using

$$Gain\left( A \right) = I\left( D \right) - E\left( A \right)$$

(13)

$$E\left( A \right) \, = \mathop \sum \limits_{i = 1}^{n} I\left( D \right)\frac{{d_{1i} + d_{2i} , + \cdots + d_{mi} }}{d}\;I\left( D \right) \, = \mathop \sum \limits_{i = 1}^{n} p_{i} \log_{2} p_{i}$$

(14)

$p_{i} -$ probability of sample (belongs to class)

4.6 Stability in feature selectors

Stability is considered as important concern connected with while using the Ensemble FS and analysis the variation in results obtained due to varying different learning subsets. Since the feature selectors are applied to different sub samples learning set, the variation in the output should be analyzed, to measure whether each subsample produce similar output. Then stability is computed based the output obtained from same feature selectors applied to different varying sub learning sets. From the $t = \left\{ {t_{1} ,t_{2} ,t_{3} , \ldots .,t_{n} } \right\}$ generated subsample learning sets with size of $n$ instances (from Sect. 4.1), each feature selectors(in Sect. 4.5) are applied to $t$ subsample sets and the stability is computed based upon the output from each feature selectors. The stable FS applied to different learning subset samples should yield similar feature output. Based upon the output of FS, similarity measurement can be considered. Since the output produced by feature selectors are based upon ranking the attributes according to their relevance, here Spearman correlation $\rho \left( {rho} \right)$ is applied (Sanchez et al. 2018).

$\rho \left( {rho} \right)$ coefficient is defined as

$$S\left( {FL_{i} ,\;FL_{j} } \right) = 1 - 6\mathop \sum \limits_{l} \frac{{\left( {FL_{i}^{l} - FL_{j}^{l} } \right))^{2} }}{{N\left( {N^{2} - 1} \right)}}$$

(15)

where $S(FL_{i}$,$FL_{j}$) defines the likeness between $FL_{i} \& FL_{j}$.The $\rho$ values lies between − 1 and +1

The similar output from $\{ FL_{1} ,FL_{2} ,FL_{3} , \ldots .,FL_{n}$ }implies stable results are obtained.

5 Experimental design

The experimental procedure are conducted for the two different methodology separately. One is BHFS- NB selecting optimal feature subset to construct NB model and second one Standard NB model without applying any preprocessing procedure.

5.1 Dataset and validity scores

The dataset considered for experimental purpose is obtained from UCI and dataset consists of 45,211 instances with 17 attributes and with two classes. The experimental output are compared using different metrics like Accuracy, Sensitivity or Recall (TPR) computes the actual positives identified correctly, Specificity (TNR) computes the actual negatives identified correctly, Precision (PPV), False Negative (FNR), False positive (FPR). The formula to measure the metrics are given below (Dhandayudam and Krishnamuthi 2013):

$${\text{Accuracy }} = \frac{TP + TN}{TP + TN + FP + FN }$$

(16)

$${\text{Sensitivity or Recall }} = \frac{TP}{TP + FN}$$

(17)

$${\text{Specificity }} = \frac{TN}{TN + FP}$$

(18)

$${\text{Precision }} = \frac{TP}{TP + FP}$$

(19)

$${\text{False Negative Rate}} = \frac{FN}{FN + TP}$$

(20)

$${\text{False Positive Rate }} = \frac{FP}{FP + TN}$$

(21)

5.2 Experimental procedure (BHFS)

• 1. The dataset used consists of 45,211 instances with 17 attributes and two classes.

• 2. In the bootstrap procedure totally $t = 25$ bootstrap subset is generated from the original dataset with $n = 90$ percentage of instances in each bootstrap subset with randomly replacement (Sect. 4.1)

• 3. Four diverse filter based feature selectors (Sect. 4.5) are applied to each $t = 25$ learning subsets. Each feature selector applied will rank the features accordingly to feature relevance (Sect. 4.2)

• 4. Aggregation procedure is applied using different combination strategies to get aggregated feature ranking for each filter based FS methods (Sects. 4.2 and 4.3)

• 5. Finally, applying various threshold percentage to each final aggregated ranking feature selector to select top features (Threshold chosen are 10, 25, 50 , $\log_{2} (n)$ percentage) (Sect. 4.4)

• 6. From the selected top 10, 25, 50 and $\log_{2} (n)$ percentage of features are considered for the construction of naive Bayes classifier using 10 fold cross validation.

• 7. Comparison is made between NB constructed using feature subset obtained from BHFS and Standard NB without using BHFS.

5.2.1 Results of BHFS-NB and standard naive Bayes

The experimental method is conducted in two different approach. One is using naive Bayes with BHFS approach (Sect. 5.2) and other one is standard naive Bayes without applying any preprocessing approaches.

5.3 Stability in BHFS

To compute the stability in feature selection (BHFS), similarity measurement is taken for each feature selectors applied to $t = \left\{ {t_{1} ,t_{2} ,t_{3} , \ldots .,t_{n} } \right\}$ here $t = 25$ subsample learning subsets with 90% instance in each sub learning sets. The feature selectors applied to sub learning sets will end up with 25 ranking feature lists ($FL = \{ FL_{1} ,FL_{2} ,FL_{3} , \ldots .,FL_{n}$ }. Then similarity approach is taken using the spearman rank method. The averaged similarity results for each feature selectors are noted in Table 5

The results indicate the output ranking produced by each feature selectors have very strong similar output ranking from the subsample learning sets.

5.4 Result analysis

The experiment results for the BHFS procedure (Sect. 5.2.1) are tabulated from the Tables 1, 2, 3 and 4. The results are compared using validity scores (Sect. 5.1). From the results it clearly shows NB constructed using BHFS feature subset improve the prediction compare to standard NB with applying any preprocessing strategies. The naive Bayes constructed using top 10 percentage feature subset gets maximum accuracy of 89.28 (in BHFS using gain ratio) and using top 25 percentage feature subset gets maximum accuracy of 89.27 (in BHFS using Chi square) and using 50 percentage feature subset gets maximum accuracy of 89.82 (in BHFS using Chi square) and using $\log_{2} (n)$ percentage feature subset gets maximum accuracy of 89.27 (in BHFS using Chi square). But the standard NB obtains the maximum of 88.0073 accuracy. The validity measure of Specificity, Precision, FNR, FPR using from different aggregation strategies using prescribed threshold value gets better prediction with BHFS-NB. But the validity measure of Sensitivity gets less prediction compare to Standard Naive Byes. Results shows setting different threshold value selects best relevant feature subset for NB. This shows NB executed using feature subset obtained from BHFS procedure improves the performance prediction. The stability analyses results are in the Table 5. The results shows each filter based FS used in BHFS approach yields similar outputs, when applied to different subset samples. The BHFS (Chi square) stability analyses gets 0.9705 and BHFS (ReliefF) stability analyses gets 0.9896 and BHFS (Symmetrical Uncertainty) stability analyses gets 0.9572 and BHFS (Gain Ratio) stability analyses gets 0.9554. Among the four feature selectors, BHFS (ReliefF) gets more similar stable results of 0.9896. The ensemble method makes to reduce the variances and to improve the prediction and stability of the feature selection. The BHFS (ReliefF) gets more similar outputs come to other FS methods. The stability measure indicate the FS applied to different subsets produces stable output. This shows BHFS selects more stable feature subset for NB evaluation.

The Fig. 1 illustrate the accuracy comparision of experimental results shown in Table 1.

Table 1 Top 10% features are selected from different aggregation strategies are considered for naive Bayes model construction and Standard Naive Bayes with summary of accuracy, sensitivity, Specificity, Precision, FNR and FPR

Fig. 1

Accuracy comparison of top 10% features selected from different aggregation strategies for naive Bayes model and with standard naive Bayes

The Fig. 2 illustrate the accuracy comparision of experimental results shown in Table 2.

Table 2 Top 25% features are selected from different aggregation strategies are considered for naive Bayes model construction and Standard Naive Bayes summary of accuracy, sensitivity, Specificity, Precision, FNR and FPR

Fig. 2

Accuracy comparison of top 25% features selected from different aggregation strategies for naive Bayes model and with Standard Naive Bayes

The Fig. 3 illustrate the accuracy comparision of experimental results shown in Table 3.

Table 3 Top 50% features are selected from different aggregation strategies are considered for naive Bayes model construction and Standard Naive Bayes summary of accuracy, sensitivity, Specificity, Precision, FNR and FPR

Fig. 3

Accuracy comparison of top 50% features selected from different aggregation strategies for naive Bayes model and with Standard Naive Bayes

The Fig. 4 illustrate the accuracy comparision of experimental results shown in Table 4.

Table 4 Top $\log_{2} (n)$ features are selected from different aggregation strategies are considered for naive Bayes model construction and Standard Naive Bayes summary of accuracy, Sensitivity, Specificity, Precision, FNR and FPR

Fig. 4

Accuracy comparison of top $\log_{2} (n)$ features selected from different aggregation strategies for naive Bayes model and with Standard Naive Bayes

Table 5 Stability analysis for feature selectors used in BHFS

6 Conclusion

The analysis of customer behavior is carried out the ML techniques and dataset applied of analysis may possibly holds correlated, irrelevant and noisy data. These data makes poor performance prediction using NB model. To enhance the NB prediction using BHFS approach is suggested. The BHFS procedure using ensemble data perturbation feature selection approach. Filter based FS technique is studied, since the method uses the statistical techniques to rank the attributes accordingly to their relevance and are computationally fast and independent of ML models. The use of ensemble methods helps to minimize variance and makes to select robust more feature subsets by combing multiple models to single models. The use of stability analyzes measure whether the output produced by FS applied to different subsets yields similar results. The selection of different feature subset is archived by setting threshold value. The BHFS procedure is used to choose the best relevant feature subset for improving the Naive Bayes is studied and experimented. The results analysis shows feature selection using BHFS procedure improves the Naive Bayes performance prediction compare to standard Naive Bayes without using any preprocessing methods. The NB build using BHFS procedure results in reduced running time compare to standard naive Bayes. Because the elimination of correlated/irrelevant variables in the dataset makes the reduced learning and testing data. Further the research can be proceed with other feature selection techniques and also experimenting using heterogeneous ensemble with stability analyses also encouraged. Also the experiment can be applied on different dataset with more high dimensional.