1 Introduction

There are many ways to conduct an image analysis or run pattern recognition. The use of image segmentation is growing, particularly because digital cameras are becoming more accessible for implementations. The process of image segmentation divides an image into homogenous regions and classifies the pixels. Some of the practical applications of image segmentation are satellite images [1], agriculture automation [2], medical images [3, 4], traffic control systems [5] and optical character recognition (OCR) [6].

An RGB image consists of three primary color channels [7]: red (R), green (G), and blue (B). In most cases RGB images are converted to grayscale because intensity can still be measured, and it is considered sufficient to showcase the possible performance gain in image segmentation.

There is a wide variety of image segmentation techniques, and thresholding methods based on image histograms are some of the most widely used segmentation methods. These histogram-based methods are very fast and effective compared to other existing segmentation methods because they only need to pass through the pixel once [8, 9]. Moreover, bi-level thresholding divides image pixels into two different homogenous regions, whereas multilevel thresholding divides image pixels into many homogenous regions. The between-class variance method (Otsu’s method) [10] and the entropy criterion method (Kapur’s method) [11] are two of the most well-known thresholding algorithms. Although these algorithms have been developed for bi-level thresholding, these algorithms can be extended for multilevel image thresholding. Because of iterative processes of these methods, which need to increase due to the thresholding values, the computational complexity of the algorithm increases exponentially; there are also problems regarding the length of time it takes these models to find the optimal solution. To overcome this obstacle we have turned to evolutionary algorithm models [12,13,14,15,16,17,18,19]. Many of these models have specifically tackled the multilevel thresholding problem.

Research on multilevel image thresholding based on optimization methods has expanded over the past few years, with researchers building off of Otsu’s and Kapur’s works and integrating new techniques. Sathya and Kayalvizhi [20,21,22] developed a multilevel image thresholding method based on the minimum variance and maximum entropy criterions using an original bacterial foraging algorithm, an adaptive bacterial foraging algorithm, an amended bacterial foraging algorithm, and a modified bacterial foraging algorithm. Yin [23] created a particle swarm optimization (PSO) algorithm for the multilevel minimum cross entropy criterion, while Horng and Jiang [24] applied the ABC optimization algorithm to optimize the maximum cross entropy criterion. The multilevel thresholding method of Oliva et al. [25], based on Otsu’s and Kapur’s works, uses the harmony search optimization algorithm; Ayala and et al. [26] also employed an improved beta differential evolution algorithm to optimize Otsu’s minimum variance criteria. Both Muppidi [27] Chao and et al. [15] suggested multilevel image thresholding techniques based on fuzzy entropy: Muppidi’s utilizes a genetic algorithm (GA), while Chao et al.’s uses a modified gravitational search algorithm. Pal and et al. [28] worked with the spider monkey optimization algorithm to generate a multilevel thresholding technique based on Otsu’s and Kapur’s works, and Khairuzzaman and Chaudhury [29] brought together Kapur's entropy and Otsu's between-class variance for image segmentation using the grey wolf optimizer (GWO). More recently, Rahkar-Farshi [19] employed an animal migration optimization (AMO) algorithm for a multilevel thresholding method that maximizes Otsu’s and Kapur’s objective functions. These are all novel solutions, but there are still issues that need to be solved and optimized, such as computational complexity and the amount of time it takes to run the segmentation.

No optimization scheme can satisfy all aspect of all optimization and search problems. The no-free-lunch (NFL) theorem [30] states that the advantages gained by a optimization algorithm are offset by other problems and that no one algorithm works the best for all situations. Therefore, we need to develop new algorithms and try to enhance the performance of existing algorithms to tackle new class of problems. In this paper, a hybrid firefly and PSO (HFPSO) [31] algorithm is utilized to address the multilevel image thresholding problem.

The rest of the paper is organized as follows: In Sect. 2, Otsu’s and Kapur’s methods are described; in Sect. 3, we give a detailed overview of the particle swarm optimization (PSO) and the firefly optimization algorithms and introduce our hybrid PSO and firefly optimization algorithm. In Sect. 4, we describe our hybrid method in detail, in Sect. 5, we provide the comprehensive experimental results and discussion, and in Sect. 6 we give a summary and some concluding remarks.

2 Otsu’s and Kapur’s methods

The image segmentation process uses thresholding methods that are specifically created to maximize an objective function(s); in this case, the objective functions set forth by Otsu and by Kapur were used to determine the optimal thresholding values. The algorithms work to maximize these objective functions, and the thresholding results in a histogram that displays segmented groups that reflect the desired objectives of the thresholding process. \(\{\mathrm{0,1},\cdots (L-1)\}\) is a range given to show the probability of ith gray level that is mathematically calculated per \({p}_{i}={h}_{i}/(m\times n)\) where \({h}_{i}\) indicates the number of pixels that corresponds gray level i,\(0\le i\le (L-1)\) and where M and N correspond to the height and width of the image, respectively. According to Otsu’s method, the objective function for bi-level thresholding is shown as:

$$J\left({t}_{1}\right)={\sigma }_{0}+{\sigma }_{1},$$
(1)

where,

$$\,\sigma_{0} = \omega_{0} (\mu_{0} - \mu_{T} )^{2} \;{\text{and}}\;\,\sigma_{1} = \omega_{1} (\mu_{1} - \mu_{T} )^{2} ,$$

with,

$$\,\omega_{0} = \sum\limits_{i = 0}^{{t_{1} - 1}} {p_{i} } ,\,\,\mu_{0} = \sum\limits_{i = 0}^{{t_{1} - 1}} {\frac{{i\,p_{i} }}{{\omega_{0} }}} \;{\text{and}}\;\,\omega_{1} = \sum\limits_{{i = t_{1} }}^{L - 1} {p_{i} } ,\,\,\mu_{1} = \sum\limits_{{i = t_{1} }}^{L - 1} {\frac{{i\,p_{i} }}{{\omega_{1} }}} ,$$

and,

\(\,\mu_{T} = \sum\limits_{i = 0}^{L - 1} {i\,p_{i} } ,\)

Therefore, it is easy to show that:

\(\,\omega_{0} \mu_{0} + \omega_{1} \mu_{1} = \mu_{T}\) and \(\,\omega_{0} + \omega_{1} = 1\).

Otsu’s objective function can be extended to multilevel model as follows:

$$J\left({t}_{1},{t}_{2},\cdots ,{t}_{m}\right)={\sigma }_{0}+{\sigma }_{2}+\cdots {\sigma }_{m},$$
(2)

where

$$\,\sigma_{0} = \omega_{0} (\mu_{0} - \mu_{T} )^{2} ,$$
$$\,\sigma_{1} = \omega_{1} (\mu_{1} - \mu_{T} )^{2} ,$$
$$\,\sigma_{2} = \omega_{2} (\mu_{2} - \mu_{T} )^{2} ,$$

⋮

$$\,\sigma_{m} = \omega_{m} (\mu_{m} - \mu_{T} )^{2} ,$$

with

$$\,\omega_{0} = \sum\limits_{i = 0}^{{t_{1} - 1}} {p_{i} } ,\,\,\mu_{0} = \sum\limits_{i = 0}^{{t_{1} - 1}} {\frac{{i\,p_{i} }}{{\omega_{0} }}} ,$$
$$\,\omega_{1} = \sum\limits_{{i = t_{1} }}^{{t_{2} - 1}} {p_{i} } ,\,\,\mu_{1} = \sum\limits_{{i = t_{1} }}^{{t_{2} - 1}} {\frac{{i\,p_{i} }}{{\omega_{1} }}} ,$$

⋮

$$\,\omega_{m} = \sum\limits_{{i = t_{m} }}^{L - 1} {p_{i} } ,\,\,\mu_{m} = \sum\limits_{{i = t_{m} }}^{L - 1} {\frac{{i\,p_{i} }}{{\omega_{m} }}} ,$$

[\({t}_{1},{t}_{2},\cdots ,{t}_{m}\)] states the m optimal thresholds for a gray scale image. There are some constraints, which are defined as:\({t}_{1}<{t}_{2}<\cdots <{t}_{m}\).

The objective function according to Kapur’s method for bi-level thresholding is shown as:

$$J\left({t}_{1}\right)={H}_{0}+{H}_{2},$$
(3)

where

$$H_{0} = - \sum\limits_{i = 0}^{{t_{1} - 1}} {\tfrac{{p_{i} }}{{\omega_{0} }}} \ln \tfrac{{p_{i} }}{{\omega_{0} }},\,\,\omega_{0} = \sum\limits_{i = 0}^{{t_{1} - 1}} {p_{i} } ,$$
$$H_{1} = - \sum\limits_{{i = t_{1} }}^{L - 1} {\tfrac{{p_{i} }}{{\omega_{1} }}} \ln \tfrac{{p_{i} }}{{\omega_{1} }},\,\,\omega_{1} = \sum\limits_{{i = t_{1} }}^{L - 1} {p_{i} } ,$$

where H0 and H1 are partial entropies of histograms. The threshold value t1 is the gray level that maximizes the objective function given in Eq. (3). Moreover, Kapur’s entropy criterion can be extended for multilevel thresholding through the following equation:

$$J\left({t}_{1},{t}_{2},\cdots ,{t}_{m}\right)={H}_{0}+{H}_{2}+\cdots {H}_{m},$$
(4)

where,

$$H_{0} = - \sum\limits_{i = 0}^{{t_{1} - 1}} {\tfrac{{p_{i} }}{{\omega_{0} }}} \ln \tfrac{{p_{i} }}{{\omega_{0} }},\,\,\omega_{0} = \sum\limits_{i = 0}^{{t_{1} - 1}} {p_{i} } ,$$
$$H_{1} = - \sum\limits_{{i = t_{1} }}^{{t_{2} - 1}} {\tfrac{{p_{i} }}{{\omega_{1} }}} \ln \tfrac{{p_{i} }}{{\omega_{1} }},\,\,\omega_{1} = \sum\limits_{{i = t_{1} }}^{{t_{2} - 1}} {p_{i} } ,$$
$$H_{2} = - \sum\limits_{{i = t_{2} }}^{{t_{3} - 1}} {\tfrac{{p_{i} }}{{\omega_{2} }}} \ln \tfrac{{p_{i} }}{{\omega_{2} }},\,\,\omega_{2} = \sum\limits_{{i = t_{2} }}^{{t_{3} - 1}} {p_{i} } ,$$

⋮

$$H_{m} = - \sum\limits_{{i = t_{m} }}^{L - 1} {\tfrac{{p_{i} }}{{\omega_{m} }}} \ln \tfrac{{p_{i} }}{{\omega_{m} }},\,\,\omega_{m} = \sum\limits_{{i = t_{m} }}^{L - 1} {p_{i} } ,$$

3 Hybrid firefly and particle swarm optimization

Since each optimization algorithm has its own advantages and disadvantages, a hybrid that combines these algorithms would allow for an increased robustness and more flexibility to tackle complex problems [32]. This section is devoted to a review of a hybrid algorithm (HFPSO) that combines the firefly and the particle swarm optimization algorithms, originally proposed by Aydilek [31]. In the following, we first review some basic principles of the PSO and firefly algorithms, then discuss the combination these two algorithms in a hybrid model.

3.1 Particle swarm optimization

Particle swarm optimization (PSO) is a population-based optimization algorithm that was introduced by Russell Eberhart and James Kenned in [6]. The model was originally inspired by the collective movements exhibited by bird and fish swarms while searching for food or escaping from perceived threats.

The PSO algorithm has shown better performance in comparison to many other search algorithms because it is quick to find the results, uses less parameters, and has a lower chance of getting stuck at the local optima. The PSO algorithm starts with a bunch of random solutions, which are called particles, and a community of particles is called a crowd or flock. Initialization of ith particle is mathematically calculated per:\({x}_{i}=lb+r \mathrm{and}\times \left(ub-lb\right).\) Here \(ub\) and \(lb\) are lower and upper bund of the problem space. The search process starts by giving each particle a random value in the solution space. At every iteration each individual particle position \({x}_{i}\) will get updated based on its best position \({p}_{1}\) and the best position of the other particles \(g\mathrm{best}\) in its topological neighborhood. The new position will be found by adding velocity vector \({v}_{i}^{t}\). PSO is an iterative algorithm; the position components are updated at each iteration by calculating the particle’s velocity components using Eq. (5a).

$${v}_{i}^{t}=w{v}_{i}^{t-1}+{R}_{1}{C}_{1}\left({p\mathrm{best}}_{i}-{x}_{i}\right)+{R}_{2}{C}_{2}\left(g\mathrm{best}-{x}_{i}\right).$$
(5a)
$${x}_{i}={x}_{i}+{v}_{i}^{t}.$$
(5b)

Here \({v}_{i}^{t}\) and \({x}_{i}\) represent the velocity and position of ith particle at iteration t respectively, w is the inertia weight, which plays an important role in the exploration/exploitation aspects of the PSO algorithm, and \({pbest}_{i}\) and \({g}_{\mathrm{best}}\) represent the position of the best solution found so far by ith particle and the best overall solution, respectively.\({R}_{1}\) and \({R}_{2}\) are two random numbers generated from a uniformly distributed range [0,1]. \({C}_{1}\) and \({C}_{2}\) are the cognition and social learning factors, respectively, since \({C}_{1}\) is multiplied by the distance to the best position of an individual particle and \({C}_{2}\) is multiplied by the distance to the best position among all the particles. The main steps of the PSO algorithm can be expressed in the form of the pseudo-code covered in Algorithm 1.

figure a

3.2 The firefly optimization algorithm

The firefly algorithm (FA) is a well-known optimization algorithm based on swarm intelligence and was first introduced in 2008 by Yang [33], who was inspired by the natural behavior of fireflies. Fireflies use their flashing lights for several reasons: the light can help them find a mating partner based on the pattern of flashing, it can be a means to attract prey, or it can be a way to deter predators. The bioluminescence light is produced when oxygen combines with calcium, the luciferin enzyme, and adenosine triphosphate [34]. The communication of fireflies through flashing lights and their subsequent behaviors inspired Xin-She Yang to propose a metaheuristic optimization algorithm. The firefly algorithm is not a detailed simulation of the behavior of fireflies but rather an idealized version of it. For this purpose, one need to assume that (1) all the fireflies are unisex (so there is no difference between females and males in terms of attraction); (2) mutual attraction between two fireflies is proportional to the intensity of the light emitted and observed by each of them; and (3) the brightness depends on the objective function, which is the subject of optimization. Therefore, the intensity of the light produced by each firefly depends on how optimal its position is and is proportional to its fitness value. In each itaration the intensity of the light of each firefly will be compared to the intensity of the light of the other fireflies, and those which are less bright will move towards brighter ones. The amount of light perceived by other fireflies depends on the distance; fireflies receive lights with varying intensities. The firefly with best fitness will search the space in a random manner to find the optimum solution. The mathematical formula that describes the motion of a less bright firefly moving toward the brighter one is given as follows:

$${x}_{i}={x}_{i}+{\beta }_{0}{e}^{-\gamma {r}_{i,j}^{2}}\left({x}_{j}-{x}_{i}\right)+\alpha \left(r \mathrm{and}-0.5\right).$$
(6)

Here \({\beta }_{0}\) is the coefficient of the maximum attraction between fireflies at positions \({x}_{i}\) and \({x}_{j}\), with Euclidean distance \({r}_{ij}\). γ is the light absorption coefficient, and α is the coefficient of random displacement vector. The pseudo-code of the firefly algorithm is given in Algorithm 2.

figure b

3.3 Hybrid firefly and PSO

It has been demonstrated that for some problems the PSO algorithm has faster convergence in comparison to other algorithms. However, it has the potential to slow down as it approaches the global optimal point. Since the algorithm adjusts itself at each iteration with the global best position, this increases the risk of getting stuck in the region of local optima. The other drawback to the PSO algorithm is its dependency on search parameters. Different parameter settings can lead to different convergence speeds [35]. Many variants of the PSO algorithm have been proposed to enhance its performance, with the main goal being to balance the exploratory and exploitatory aspects. These proposals include changes in the strategy of the evolution, tuning the parameters, changing the updating rules, and incorporating better evolving strategies. For the method proposed in this paper, the velocity, calculated using Eq. (5a), will be substituted in the formula of Eq. (5b) to update the position of the particles in the next stage. If this velocity is too slow or too fast, it can lead to difficulties such as oscillation around the solution, which can cause the speed of convergence to decrease. To tackle this problem, we need to adjust the inertia weight \(w\) and acceleration coefficients (\({c}_{1},{c}_{2}\)) and then calculate velocity according to Eq. (5a). These adjustments are challenging problem: the FA does not include velocity in its algorithm, but it has also been shown that the FA is more efficient and has higher success rate in multimodal problems in comparison to PSO [33]. The intent behind the hybrid firefly and particle swarm optimization algorithm (HFPSO) [31] algorithm proposed by Aydilek is to benefit from the strengths of each of these algorithms. The PSO algorithm will help with exploration, while the FA will take care of local search. Meanwhile the inertia weight will be updated dynamically.

The first step in the HFSPO algorithm is to initialize all of the parameters. Then the positions and velocities of particles will be initialized to random values (in the predetermined ranges). The fitness, global best (gbest), and personal best (\({p\mathrm{best}}_{i}\)) will be calculated as the next step. Then the particle’s fitness in the current stage and its last iteration according to Eq. (8) will be compared. If the fitness value of particle is the same or improved, the FA will take over and local search will start; otherwise, the PSO algorithm continues according to Eq. 5. IF the FA algorithm takes over, then the position and velocity will be calculated according to Eqs. ((9 and ((10. In the next step the position and velocity ranges are checked for all the fireflies and particles. If the fitness value reaches the desired value, the algorithm will be terminated, and the final result will be given as output.

$$w= {w}_{i}-\left(\frac{{w}_{i}-{w}_{i}}{{\mathrm{iteration}}_{\mathrm{max}}}\right)\times \mathrm{iteration}.$$
(7)
$$f\left(i,t\right)= \left\{\begin{array}{c}true, if fitness ({\mathrm{particle}}_{i}^{t})\le {g\mathrm{best}}^{t-1}\\ false, if fitness \left({\mathrm{particle}}_{i}^{t} \right)>{g{\mathrm{best}}}^{t-1}.\end{array}.\right.$$
(8)
$${x_i}\left( {t + 1} \right) = {x_i}\left( t \right) + {B_0}{e^{ - \gamma r_{ij}^2}}\left( {{x_i}\left( t \right) - g{\text{best}}{^{t - 1}}} \right) + \alpha \left( {{\text{r}}\;{\text{and}} - \frac{1}{2}} \right).$$
(9)
$${V}_{i}\left(t+1\right)= {x}_{i}\left(t+1\right)-{x}_{i-\mathrm{temp}}$$
(10)

Here, indices i and t refer to the number of the step and the number of the particle, respectively. Choosing the right strategy to control the inertial weight \(w\) parameter is helpful in balancing the exploratory and exploitary aspects of the PSO algorithm. Here a linearly decreasing inertia weight is implemented, which will help the PSO to converge better [36]. The pseudo-code of HFPSO is given below:

figure c

4 Multilevel thresholding using HFPSO

In this section we will use the HFPSO algorithm to search in the threshold’s space. The aim of the optimization is to find the thresholds that will give the desired segmentation of the histogram. This can be done by maximizing the Otsu objective function or Kapur’s entropy criterion given in Eqs. (2) and (4), respectively. The dimension (D) of the optimization algorithm is equal to the number of thresholds (x = [\(t_{1} ,t_{2} ,...,t_{D}\)]). The gray level \(q\) in an image is bounded from below by 0, and its upper bound is 255 (0 \(\le q<\) L − 1 = 255). Thresholds are then subjected to the constraint \(1 < t_{1} < t_{2} < ... < t_{D} < L\).

In the HFPSO algorithm the first step is to initialize the thresholds randomly as follows:

$${x}_{j,i,0}={x}_{j,\mathrm{min}}+{r \mathrm{and}}_{j,i}[\mathrm{0,1}]\times ({x}_{j,\mathrm{max}}+{x}_{j,\mathrm{min})}.$$
(11)

The randomizer \(r\;{\text{and}}_{i,j}\) gives a number that is randomly generated from a uniform distribution in the interval [0,1]. Here \(i\) is the number of the particles ( i = 1, …, N). The number of dimensions is shown by\(j=\mathrm{1,2},\dots ,D\).

5 Experimental results and discussion

To compare different algorithms with the proposed algorithm, we carried out computations using MATLAB R2018 on PC with Intel Core (TM) i7- 3632QM 2.2 GH processor and 8 gigabytes of RAM. We evaluated the efficiency of these algorithms by applying them to the benchmark images: “Cameraman,” “Lenna,” “Livingroom,” “Hunter,” “Pepper,” and “Mandrill.” The images are shown in Fig. 1. These images are difficult to segment using the more traditional, bi-level thresholding methods because their gray scale image histograms are multimodal. The multimodal histograms of “Lenna” and “Pepper” exhibit many valleys and peaks, while “Livingroom” and “Hunter” are characterized by extreme changes in the number of pixels. “Mandrill’s” histogram has a smooth distribution in the gray level in comparison to “Cameraman.”

Fig. 1
figure 1

Images used for performance evaluation

Full size image

The performance of the algorithm that we implemented here depends on the parameters. To perform a fair comparison among the proposed algorithm and the other algorithms—AMO, PSO, GA, and FA—we used the following parameters: the number of agents (animals, particles, chromosomes, and fireflies in AMO, PSO, GA, and FA, respectively) and number of maximum generations for all algorithms is taken to be 200. Table 1 contains the parameters of the different optimization algorithms we used in the experiments.

Table 1 Parameter settings for the optimization algorithms
Full size table

To compare the outcome of different algorithms, we evaluated them both visually and numerically. Figure 2 depicts the segmented version for different levels of thresholding (m = 2, 3, 4, and 5).

Fig. 2
figure 2

Segmented Cameraman with located thresholds

Full size image
Fig. 3
figure 3

Segmented Lena with located thresholds

Full size image
Fig. 4
figure 4

Segmented Livingroom with located thresholds

Full size image
Fig. 5
figure 5

Segmented Hunter with located thresholds

Full size image
Fig. 6
figure 6

Segmented pepper with located thresholds

Full size image
Fig. 7
figure 7

Segmented Mandrill with located thresholds

Full size image

Tables 2, 3, 4, 5, 6, 7 give the numerical evaluation of algorithms. The performance of the algorithms was evaluated using well-known indicators: peak signal to noise ratio (PSNR), structural similarity index measure (SSIM), Feature SIMilarity (FSIM), standard deviation (SD), and fitness value. The mathematical representation of PSNR is as follows:

Table 2 Comparison of best obtained objective and optimal threshold values based on Kapur
Full size table
Table 3 Comparison of standard deviation for based on Kapur
Full size table
Table 4 Comparison of PSNR for based on Kapur
Full size table
Table 5 Comparison of best obtained objective and optimal threshold values based on Otsu
Full size table
Table 6 Comparison of standard deviation based on Otsu
Full size table
Table 7 Comparison of PSNR based on Otsu
Full size table

\(\mathrm{PSNR}=10{\mathrm{log}}_{10}(\frac{255}{\mathrm{MSE}})\), with mean squared error

$$\mathrm{MSE}=\frac{1}{M\times N}\sum_{x=0}^{M-1}\sum_{y=0}^{N-1}{[I\left(x,y\right)-\stackrel{\sim }{I}(x,y)]]}^{2}.$$
(12)

where \(I(x, y)\) is the value of a pixel of original image located at \((x, y)\), and \(\stackrel{\sim }{I}(x, y)\) is the segmented image value at \((x, y)\). The size of the images is M \(\times\) N. Also, SSIM is calculated using the following equation:

$$\mathrm{SSIM}\left(I,\mathrm{Seg}\right)=\frac{(2{\mu }_{I}{\mu }_{seg}+{c}_{1})(2{\sigma }_{I,seg}+{c}_{2})}{({\mu }_{I}^{2}+{\mu }_{seg}^{2}+{c}_{1})({\sigma }_{I}^{1}+{\sigma }_{seg}^{2}+{c}_{2})}$$
(13)

where \({\mu }_{\mathrm{seg}}\) and \({\mu }_{I}\) indicate the mean gray value of the segmented and original image, respectively.\({\sigma }_{\mathrm{seg}}\) and \({\sigma }_{I}\) are the mean standard deviation of the segmented and original image, respectively. \({\sigma }_{I,\mathrm{seg}}\) is the covariance of I and Seg. Here, \({c}_{1}\) and \({c}_{2}\) are replaced by 6.5025 and 58.52252, respectively. Additionally, FSIM is computed as follows:

$$\mathrm{FSIM}=\frac{\sum_{w\in\Omega }{S}_{L}(w){PC}_{m}(w)}{\sum_{w\in\Omega }{PC}_{m}(w)},$$
(14)

where \(\Omega\) indicates the entire domain of the image and their value is calculated as foloows:

$${{S}_{L}\left(w\right)=S}_{PC}\left(w\right){S}_{G}\left(w\right){ S}_{PC}\left(w\right)=\frac{{2PC}_{1}\left(w\right){PC}_{2}\left(w\right)+{T}_{1}}{P{C}_{1}^{2}\left(w\right)+{PC}_{2}^{2}\left(w\right)+{T}_{1}} {S}_{G}\left(w\right)=\frac{{2G}_{1}\left(w\right){G}_{2}\left(w\right)+{T}_{2}}{{G}_{1}^{2}\left(w\right)+{G}_{2}^{2}\left(w\right)+{T}_{2}}.$$
(15)

The gradient magnitude (GM) of a digital image and is defined by \(G\), and the value of \(PC\) that is the phase congruence is computed per: \(G= \sqrt{{G}_{x}^{2}+{G}_{y}^{2}}\) and \(PC\left(w\right)=E(w)/(\varepsilon +\sum_{n}{A}_{n}(w))\), here \({A}_{n}(w)\) indicates the local amplitude on scale \(n\) and \(E(w)\) is the magnitude of the response vector in \(w\) on \(n,\) \(\varepsilon\) is a small positive number and finally, \({PC}_{m}\left(w\right)=\mathrm{max}({PC}_{1}\left(w\right),{PC}_{1}(w))\). After all to calculate the standard deviation (SD), we use the following formula:

$$\sigma = \frac{1}{N}\mathop \sum \limits_{i = 1}^{N} \left( {x_{i} - \mu } \right)^{2} ,$$
(16)

where N is the number of runs, \(x_{i}\) is the fitness value obtained in ith run, and \(\mu\) is mean value obtained over all runs of the algorithm.

We used the AMO, PSO, GA, and FA algorithms to optimize the corresponding objective functions. These results were obtained by applying the algorithms to the 6 test images; we ran each algorithm 30 times over the test images. In Table 2, we report the thresholds and the corresponding fitness values, which were obtained and averaged from the 30 runs for each algorithm on each test image, for Kapur’s objective function. Additionally, Table 5 shows the corresponding thresholds and average fitness values for Otsu’s objective function.

Figures (2, 3, 4, 5, 6, 7) show the segmented “Cameraman”, “Lenna”, “Livingroom”, “Hunter”, “Pepper” and “Mandrill” images using Otsu’s method with various “m” levels. These images highlight the segments identified using a different color palette than the original image. The figures to the right of each image depict the histograms, which are superimposed (shown by dashed vertical lines) with multilevel thresholds. These images show that increasing the number of the levels (m) helps identify more details in the picture. However, this also increases the risks of splitting a segment in the image into many more segments. For each image, we first ranked the fitness values for each number of thresholds of each algorithm. After taking the average over the ranks of each algorithm, the highest averaged rank is bolded.

5.1 Kapur’s function

It is clear from Table 2 that PSO produced very competitive results in comparison to HFPSO; however, HFPSO overall surpasses PSO in performance evaluation. From Tables 2, 3, 4, it is seen that the HFPSO ranks first in fitness value, SD, PSNR, and SSIM. Likewise, GA, AMO, HFPSO, PSO, and FA rank first to last respectively in terms of FSIM. Also, Table 3 reveals that in most of the cases the obtained SD values of the HFPSO are zero. This shows that HFPSO is a more stable algorithm in comparison to other algorithms in this study. On the other hand, FA ranks worst in fitness value, SD, and PSNR. Moreover, AMO ranks fourth in all criteria, whereas GA ranks second, alongside PSO, in fitness value. PSO ranks second in SD, and GA’s rank is third. GA ranks second in PSNR, whereas PSO ranks third.

5.2 Otsu’s function

Table 5 shows that HFPSO and PSO outperform the other algorithms in fitness value and that FA performs the worst. GA and AMO rank second and fourth, respectively. Furthermore, based on Table 6, HFPSO is the most stable algorithm because its SD is 0 in all levels of segmentation in all the images. PSO ranks second in SD value, whereas GA, AMO, and FA rank third to fifth, respectively. For PSNR, GA performs the best and HFPSO ranks second, where PSO. AMO and FA rank fourth and fifth, respectively. Moreover, HFPSO and PSO rank first according to SSIM. Likewise, HFPSO ranks First alongside PSO in terms of FSIM (Table 7).

The experimental results show that our proposed method of an HFPSO algorithm achieves better fitness value and PSNR. It also shows better stability in comparison to the other optimization algorithms for multilevel image segmentation.

6 Conclusion

In this study, a multilevel image thresholding method has been proposed using a hybrid firefly and particle swarm optimization algorithm that maximizes Otsu’s and Kapur’s objective functions. The proposed method has been assessed by comparing it with four well-known optimization algorithms (AMO, PSO, GA, and FA) for six commonly used benchmark images. The visual experiments show that the thresholds are mostly located in valleys between two peaks.

The proposed HFPSO algorithm is extremely stable. The SD results are zero in Otsu’s objective function, and it was ranked first according to Kapur’s method. Moreover, HFPSO ranked first in fitness value for both Otsu’s and Kapur’s methods. In terms of the PSNR performance criterion, for Otsu’s objective function, HFPSO came in second, behind GA and for Kapur’s objective function, HFPSO ranks first. HFSPO performs well in terms of SSIM: it ranks first in Kapur’s method and first alongside PSO in Otsu’s method. On the other hand, for FSIM, HFPSO is tied with PSO for first under Otsu and is ranked third under Kapur. The comprehensive experiments reveal that this hybrid method achieves overall better results in term of fitness value and PSNR and that its SD is the best out of all the algorithms.