Design of digital IIR filter with low quantization error using hybrid optimization technique

Abstract

In this paper, a hybrid optimization technique based on particle swarm optimization (PSO) and artificial bee colony (ABC) algorithm is presented for the optimal design of infinite impulse response (IIR) filter with low quantization effect. In this method, different variants of PSO have been exhaustively tested, and the time varying coefficients-PSO (TVC-PSO) is used to formulate a new hybrid technique for better exploitation and exploration, which is further modified by sorting and replacement mechanism of Scout Bee from ABC algorithm. For designing IIR filter, an objective function is constructed that satisfies the absolute error including peak ripples in passband and stopband regions in frequency domain, while stability of designed filter is confirmed by exploiting the lattice form structure during iterative computation that also reduces computation complexity. Several attributes such as passband error $(e_{\mathrm{p}})$, stopband error $(e_{\mathrm{s}})$, and stopband attenuation $(A_{\mathrm{s}})$ are used to measure the performance of proposed algorithm. The simulation results presented in this paper evidence that this technique can be effectively used for designing digital IIR filter with higher filter taps, and low quantization effect for fixed number of bits.

1 Introduction

In recent decades, digital signal processing (DSP) has gained considerable attention due to its wide applications in numerous engineering fields of one-dimensional (1-D) and multidimensional signals. Typical applications include biomedical signal processing, adaptive filtering, harmonic estimation, satellite image processing, communication network, and power system (Kumar et al. 2012; Ahirwal et al. 2014a, b; Bhandari et al. 2015a). Digital filters are the frequency selective elements extensively used in various signal processing applications due to their reconfigurability and simplified design. In addition, digital filters have a sharper transient response, better stopband attenuation as compared to analog filters (Nongpiur et al. 2013). There are several applications such as system identification, adaptive filtering, biomedical signal processing (Hartmann et al. 2014) etc, where digital IIR filter is widely exploited. All such applications rely on efficient design of IIR filter, which can be designed either using a conventional method or by computer-based method. In early stage of research, conventional techniques based on analog filter design were exploited for the design of IIR filter due to their simplified design; however, they suffer from approximation error, quantization error due to truncation with a finite bit, poor efficiency, and the requirement of higher order of filter taps for prescribed behavior (Tang et al. 1998; Lang 2000). Therefore, a computer-based method using gradient-based optimization was presented in Kobayashi and Imai (1990), and subsequently, several algorithms were developed to explore the solution by either minimizing or maximizing the objective function formulated using different filter design objectives (Lang 2000; Kumar et al. 2010, 2013; Nongpiur et al. 2013). However, the gradient-based techniques are suitable for problem having unimodal surface. Thus, these techniques are not suitable for the design of IIR filter as its error surface is multimode.

A new technique using genetic algorithm (GA) was presented for the efficient design of IIR filter for system identification, and this was further improved in (Weidong and Fan; Etter et al. 1982). But, the limitation observed in all such techniques was dependence on initial population. If the diversity mechanism is not executed properly, GA converges into local minima/maxima, and high variation in the output parameters for each trial of execution is seen. Therefore, particle swarm optimization (PSO) inspired by fish schooling and bird flocking was used for designing IIR filters due to its fast convergence rate and better optimal solution as compared to GA (Krusienski and Jenkins 2003; Saha et al. 2011) and was further modified using different variants of PSO in (Bansal et al. 2011; Ahirwal et al. 2013). However, problem with PSO is still of getting trapped in local minima, and PSO is not immune to handle higher dimensional problems, which is order of filter in case of filter design.

Recently, artificial bee colony (ABC) algorithm inspired by intellectual scavenging conduct of honey bee swarm has emerged as a robust optimization for multimodal and unimodal error functions. Literature reviews reflects that this technique has been used for solving many complex engineering problems such as adaptive filtering (Ahirwal et al. 2014a), satellite image segmentation (Bhandari et al. 2015b), multirate system design (Kuldeep et al. 2015a, b). Karaboga (2009) have used ABC techniques for designing digital IIR filter using system identification, and this was further improved in Agrawal et al. (2015a). But, some hinders have been observed in ABC technique similar to PSO. As ABC technique adopts probabilistic mechanism for sorting best solution from the population and hence requires too many numbers of iterative trials and execution as compared to PSO. In conventional PSO, convergence is fast due to comparatively less function evaluation; however, it suffers to achieve global minima in large-scale optimization problems. To resolve this issue, an improved PSO based on scout mechanism of ABC algorithm was proposed, and used for designing filter bank (Rafi et al. 2013). Subsequently, several researchers have proposed improved PSO based on hybridization with other optimization techniques. Gong et al. (2010) and Zhang et al. (2014) have introduced a new hybrid PSO technique by governing mutation operation of DE on search space (particles) of improved bare-bones PSO (control parameter free) for power system optimization. The GA was also merged to develop another Hybrid PSO for determining design parameters for a higher-order sliding model controller (Cao et al. 2016). A new hybrid technique was also presented based on PSO and DE with binary search algorithm for designing optimal IIR filter (Sidhu et al. 2016). Thus, literature review on hybrid techniques for designing digital IIR filters evidences that several hybrid techniques discussed above, have been proposed. In these algorithms, complexity was quite high as both local and global search were conducted using two different optimization techniques. It is also evident that extensive work has been done toward design and development of IIR filter using various swarm-based techniques. However, there is no technique available in literature, which is applicable for higher filter taps, and has low quantization affect. Therefore, in this paper, a new hybrid technique is proposed with better exploration and exploitation abilities for designing improved IIR filter with simple stability constraint. This technique is also applicable for designing higher order filter with less quantization and truncation errors.

In above context, therefore, this paper presents an improved hybrid method based on ABC and PSO with time varying coefficients for solving nonlinear optimization constructed using the prescribed ripple in passband and stopband region. The presented technique gives a more stable design with less quantization effect.

2 Overview of swarm-based techniques

Swarm-based techniques are the subset of evolutionary computation and usually counted in artificial computation. The widely recognized algorithms are as follows.

2.1 Particle swarm optimization (PSO)

PSO algorithm was developed by inspiring from phenomena of communication behavior of birds, fish and insects, and successfully applied in optimal design of digital filter (Kennedy and Eberhart 1995; Sheng and Bing 2010; Shao et al. 2015; Sharma et al. 2016). In PSO technique, the optimal solution is obtained by following a random path governed by two variables: ‘Pbest’ (local best component), which is the solution corresponding to current best solution of objective function and ‘Gbest’ (global best component) which is the another solution corresponding to final best solution achieved in entire search. In first stage, initializing of population matrix of particles/swarm is performed. Each set of particle vector contains a possible solution of the given problem. In second stage, this swarm matrix is updated using (Kennedy and Eberhart 1995; Bansal et al. 2011):

$$\begin{aligned} v_n^{[i+1]}= & {} \chi \left( w\cdot v_n^{[i]} +C_1 \cdot { rand}_1 \cdot ({ Pbest}_n^{[i]} -{ pop}_n^{[i]})\right. \nonumber \\&\left. +\, C_2 \cdot { rand}_2 \cdot ({ Gbest}^{[i]}-{ pop}_n^{[i]} ) \right) \end{aligned}$$

(1)

In above Eq. (1), $v_{n}$ represents the velocity matrix, whose dimensionality is same as of the population/swarm matrix, w is the inertia weight that controls the search space by putting restriction on particles, $\chi$ is the constrained factor and $i{\hbox {th}}$ is the current iteration cycle. $C_{1}$ and $C_{2}$ are the cognitive and social scaling parameters; ${ rand}_{1}$ and ${ rand}_{2}$ is the random number vector. The velocity associated with particles of swarm has to be in limit of certain range., now the population matrix is updated as (Kennedy and Eberhart 1995):

$$\begin{aligned} { pop}_n^{[i+1]} ={ pop}_n^{[i]} +v_n^{[i+1]} \end{aligned}$$

(2)

In the end, greedy based selection procedure is conducted for sorting and updating of Pbest and Gbest. PSO has been formulated in CAD program by following the pseudocode explained in Ahirwal et al. (2013).

2.2 Artificial bee colony (ABC)

Artificial bee colony (ABC) algorithm is a global, meta-heuristic search and optimization method inspired by intelligent foraging behavior of honey bees (Karaboga and Basturk 2007a; Karaboga 2010). This method consists three segments: employed, onlooker, and scout bees with food sources. In ABC, problem formulation for optimization is constructed by searching the best parameter vector from population entitled as ‘Food’, which will minimize the objective function. In the first stage, search space is formed by initializing food matrix, and each row represents set of values (solution). Now, the employed bee phase starts with modification of search space, followed by evolution of fitness, and sorting of solution with best fitness value is performed. The second stage is onlooker bee phase, in which a solution is searched among the food particle left after employed bee phase based on certain parameter selection. During these phases, if a new solution has not been improved, then counter associated with each solution is incremented by one. In third phase (also known as scout bee phase), counter value is scanned, and if any of the counter value is found equal to permissible limit, then that corresponding solution (food particle) is initialized with a new value. The detailed analysis on ABC algorithm along with pseudocode can be found in many literature (Karaboga and Basturk 2007a; Karaboga 2010; Ahirwal et al. 2013).

3 Proposed hybrid technique

From literature review, it is evident that several attempts have been made to improve performance of conventional PSO for various applications. Rafi et al. (2013) have proposed the concept of hybridization of two optimization techniques for exploring the optimal solution. Originally, Rafi et al. (2013) have developed a hybrid technique for designing multirate filter banks based on PSO and ABC algorithms. In this technique, mechanism for updating population, and sorting of the best solution is governed by PSO, followed by replacement mechanism for unimproved solution of ABC algorithm. Here, after updating the velocity matrix, population matrixes are updated, and after execution of each steps, examination on velocity particles, and position matrix is carried out. If these values exceed the limit, they are brought back within the limits. After completion of evaluation of objective function using updated population matrix, quality of each solution is checked. This mechanism has been adopted from ABC algorithm in which concept of three bees such as employed bee, onlooker bee, and scout bee are used (Karaboga and Basturk 2007b; Karaboga 2010). During food search, if the employed bee could not succeed in an exploration of food source with improved quality in specified number of prescribed trails, known as the limit then it leaves the current food source and converts into a scout. Similarly, in hybrid PSO, if any vector whose solution is not improved, then its value is replaced by till known global best solution (Gbest) and the velocity vector corresponding to unimproved solution is replaced by the improved velocity recorded for Gbest In this way, swarm is updated in direction of an optimal solution that confirms the optimal point exploration, and resists the trapping in local minima. The proposed concept is a joint venture of two distinguishes robust meta-heuristic techniques, and thus, named as hybrid PSO. Later on, several researchers have developed improved hybrid techniques using different optimization techniques such GA and DE (Gong et al. 2010; Zhang et al. 2014).

Literature review on PSO technique evidences that several variant of conventional PSO such as constant weight inertia (CWI)-PSO, linearly decay inertia-PSO (LDI-PSO), dynamic inertia-PSO (DI-PSO) and time varying coefficients-PSO (TVC-PSO) have been proposed for different applications. A detailed discussion on these variants are given in Ahirwal et al. (2013) and the references therein. Therefore, in this work, a comparative study of performance of different variants of PSO, hybridizing with ABC algorithm is carried out. Based on performance, an improved hybrid technique is proposed using the time varying coefficients-PSO (TVC-PSO) and ABC algorithm for designing digital IIR filters. Following step are executed during the course of proposed hybrid PSO method, defined as:

• Step 1: The initial parameters are defined such as: desired solution value (D), dimension of solution (N), search space size (M).

• Step 2: The search space (particles) matrix is formulated by assigning a uniformly distributed random number in the range of lower limit $\left( {X_\mathrm{l}}\right)$ and upper limit $\left( {X_\mathrm{u}}\right)$, defined as:

  $$\begin{aligned} { pop}_n^{[i=0]} =X_\mathrm{l} +\left( {X_\mathrm{u} -X_\mathrm{l} } \right) \cdot { rand}_n \left[ {0,1} \right] \end{aligned}$$

  (3)

  where n is the index of search space vector, ranging from 1 to M.

• Step 3: Then, velocity matrix (v) associated with particles is formed within lower velocity limit $\left( {V_\mathrm{l}}\right)$ and upper velocity limit $\left( {V_\mathrm{u} }\right)$ as:

  $$\begin{aligned} v_n^{[i=0]} =V_\mathrm{l} +\left( {V_\mathrm{u} -V_\mathrm{l} } \right) \cdot { rand}_n \left[ {0,1} \right] \end{aligned}$$

  (4)

• Step 4: Each solution of search space (pop) vector is used for evaluation of fitness function, and the solution with best fitness function values is considered as Gbest, while the initially formed search space is considered as Pbest.

• Step 5: Now, algorithm enters into iterative computation stage ranging as ${\textit{i}}=1$ to ${\textit{i}}_{\max }$ Firstly, inertia weight (w) or coefficients $\left( {C_1 ,C_2}\right)$ are computed for suitable variant, then v is updated using Eq. (1), and at last, popis updated using Eq. (2).

• Step 6: Then, this updated pop is exploited for evaluation of fitness function. If newly generated solution vectors have gained better fitness value over previous vectors then, old solution in Pbest will be replaced with new one, and associated velocity is recorded correspondingly; else counter trail is increased by one.

• Step 7: Now check whether current fitness value of any Pbest solution vector is better than current Gbes t value or not. If yes, then, that Pbest solution will replace the current Gbest, and corresponding velocity is recorded as global best velocity (Best vel), otherwise old Gbest is kept unchanged.

• Step 8: Then, counter value corresponding to each individual solution vector of pop is checked. If it is equal to limit, then such solution is replaced by Gbest and v with Best vel.

• Step 9: Now, check whether fitness values achieve ‘Tol’ (tolerable fitness value) or iteration cycles are completed or not, if yes, then Gbest holds the optimal solution, otherwise, go back to step 5 and follow the next steps.

A flowchart for the proposed method is depicted in Fig. 1. The proposed hybrid algorithm is very useful for non-convex and non-differentiable design problem of IIR filter with acceptable fidelity parameters using finest swarm size, while the complexity remains same as for conventional PSO.

Fig. 1

A flowchart for the proposed Hybrid method (Rafi et al. 2013)

4 Design of IIR filter using proposed improved hybrid technique

Digital IIR filter can be designed using transformation method, but it suffers from inefficiency and quantization effect. Therefore, a new state of art has been practiced in which modern meta-heuristic techniques have been employed. In the design procedure using these techniques, the coefficients of required filter is searched according to required performance. IIR filter is characterized by a linear constant difference equation, defined as (Proakis and Manolakis 2006; Saha et al. 2013):

$$\begin{aligned} \sum _{i=0}^K {b_i y\left[ {n-i} \right] } =\sum _{j=0}^L {a_j x\left[ {n-j} \right] } \end{aligned}$$

(5)

where y(n) is the output sequence when excited by the input sequence of x(n), $a_{j}$ and $b_{i}$ are the coefficients that decide the nature of response. From Eq. (5), the output response can also be stated in frequency domain using z transform as (Proakis and Manolakis 2006):

$$\begin{aligned}&Y\left( z \right) \left[ {1+b_1 z^{-1}+b_2 z^{-2}+\cdots +b_K z^{-K}} \right] \nonumber \\&\quad =X\left( z \right) \left[ {a_0 +a_1 z^{-1}+a_2 z^{-2}+\cdots +a_L z^{-L}} \right] ,\nonumber \\&\qquad \hbox { where }b_0 =1 \end{aligned}$$

(6)

$$\begin{aligned}&\frac{Y\left( z \right) }{X\left( z \right) }=H\left( z \right) =\frac{a_0 +a_1 z^{-1}+a_2 z^{-2}+\cdots +a_L z^{-L}}{1+b_1 z^{-1}+b_2 z^{-2}+\cdots +b_K z^{-K}} \end{aligned}$$

(7)

Hence, Eq. (7) characterizes the frequency response of IIR filter, and governing parameters are the coefficients of denominator and numerator polynomials.

4.1 Problem formulation

In this paper, an efficient design of optimal IIR filters is carried out using ABC technique, variants of PSO, and Hybrid methods, in which coefficients of IIR filter are successively explored until the error between outputs of proposed filter and desired filter is minimized. The error function, which is absolute error with ripple has been adopted as an objective function, computed in frequency domain and minimized, given as (Saha et al. 2014):

$$\begin{aligned} J= & {} \left[ \sum \nolimits _{\omega \in \omega _p } {abs\left( {\left| {H(\omega )} \right| -D\left( \omega \right) -a_p } \right) }\right. \nonumber \\&\left. +\sum \nolimits _{\omega \in \omega _s } {abs\left( {\left| {H(\omega )} \right| -D\left( \omega \right) -a_s } \right) }\right] \end{aligned}$$

(8)

where $|H(\omega )|$ is the magnitude response of designed filter, $a_{\mathrm{p}}$ and $a_{\mathrm{s}}$ are the permissible ripples in passband and stopband, respectively. $D(\omega )$ is the desired frequency response, defined as:

$$\begin{aligned} D\left( \omega \right) = \left\{ {{\begin{array}{ll} 1,&{}\quad \omega \in { passband} \\ 0,&{}\quad \omega \in { stopband}, \\ \end{array} }} \right. \end{aligned}$$

(9)

4.2 Stability constraint

IIR filters are potential toward instability and thus require higher attention during their process of design. Stability of IIR filters are confirmed by ensuring the location of poles (roots of denominator polynomial), which should be lied in unity circle of z-plane. In early stage of research, researchers have utilized the concept of breaking higher-order transfer function into first and second order functions, in which denominator coefficients values are restricted in certain range for stability (Tang et al. 1998; Yu and Xinjie 2007). This method was suitable for lower-order filter design and was not computationally efficient. Moreover, the implemented filter in direct form was not able to achieve better optimal point. In this work, a different method is used in which instead of denominator polynomial in direct form, equivalent values in lattice form are substituted as shown in Eq. (10) (Sheng and Bing 2010):

$$\begin{aligned}&pop_{n,m}^{[i]}\nonumber \\&\quad = \left[ {a_{n,0} }\; {a_{n,1} }\; {a_{n,2} }\; \cdots \; {a_{n,L} } \;: \; {g_{n,L+1} } \; {g_{n,L+2} } \;\cdots \; {g_{n,L+K}} \right] ^{[i]}\nonumber \\ \end{aligned}$$

(10)

where n corresponds to index of a solution vector from population matrix, m corresponds to a solution vector element. Each solution consists of D elements $(D=2{ order}+1)$. The element $a_{n,m}$ are the numerator coefficients, whereas $g_{n,m}$ are the lattice equivalents of denominator coefficients. The use of lattice coefficients is adopted because it makes easy to handle the stability as their values should be in limit of $-1$ to 1 for stable design. The values of lattice coefficients are converted back into direct form and substituted in Eq. (7). The conversation is computed by recursive computation of the following set of equations for each lattice coefficient (Agrawal et al. 2015b):

$$\begin{aligned} U_m (z)= & {} U_{m-1} +g_m \cdot z^{-1}\cdot V_{m-1},\nonumber \\&m=1,2, \ldots ,N-1.\;\hbox { where U}_0 (z)=V_0 =1\nonumber \\ \end{aligned}$$

(11)

$$\begin{aligned} V_m (z)= & {} z^{-m}\cdot U(z^{-1}) \end{aligned}$$

(12)

Now, the polynomial coefficients of Eq. (11) are the equivalent direct form coefficients:

$$\begin{aligned} \left[ {g_{i,L+1} }\; {g_{i,L+2} } \; \cdots \; {g_{i,L+K} } \right] \rightarrow \left[ 1\; {b_{i,1} } \; \cdots \; {b_{i,K} } \right] \end{aligned}$$

(13)

Upon substitution in Eq. (7), N point frequency sample response $|H(\omega )|$ is calculated and used for computation of objective function using Eq. (8).

4.3 Designing of IIR filter using proposed hybrid method

In this paper, a comprehensive experimental study has been performed for designing an optimal digital IIR filter using improved hybrid swarm-based techniques. For this purpose, search space is formulated, in which $m \times n$ matrix is initialized with some pseudo random vector, defined in Eq. (10), where m is the total possible solutions considered, and $n\,(n=L+K)$ is the coefficient length of each solution, and a velocity matrix is also initialized with same dimension. The search space is modified by first updating the velocity matrix using Eq. (1), where scaling of w is performed in three distinguish ways, which result in constant weight inertia-PSO (CWI-PSO), linearly decay inertia-PSO (LDI-PSO), and dynamic inertia-PSO (DI-PSO), respectively. The detailed analysis of these variants can also be found in Ahirwal et al. (2013). In all these techniques, the value of $\chi$ is kept fixed at 1, while another variant of PSO, known as constrained factor inertia-PSO (CFI-PSO), is also exist, in which w is kept fixed to 1, and $\chi$ is initialized with 0.7213. Similarly, another variant has also been tested known as time varying coefficients-PSO (TVC-PSO), in which w and $\chi$ are fixed and $C_{1}$ and $C_{2}$ are made to swing accordingly (Sheng and Bing 2010):

$$\begin{aligned} \hbox {C}_1= & {} C_{1_{{\mathrm{initial}}} } -\left[ {\frac{\left( {\left| {C_{1 _{\mathrm{initial}}} -C_{1_{\mathrm{final} } }} \right| } \right) \cdot i}{i_{\max }}}\right] \end{aligned}$$

(14)

$$\begin{aligned} C_2= & {} C_{2 _{\mathrm{initial}}} +\left[ {\frac{\left( {\left| {C_{2_{\mathrm{initial} }} -C_{2 _{\mathrm{final}}}}\right| } \right) \cdot i}{i_{\max } }} \right] \end{aligned}$$

(15)

After velocity updating, its value is checked and if velocity associated with any element has been moved beyond the limit; then, it is enforced to stay back by initializing with a new value as shown in pseudocode-1 in Table 1. Experimentally the prescribed range of velocity matrix element is considered to be $-1$ to 1. After the velocity update, the position matrix (search space) is updated using Eq. (2), and similarly the elements corresponding to coefficients of lattice are checked. If the values of these coefficients moved beyond ${\vert }1{\vert }$, then they are reinitialized as shown in pseudocode-2 of Table 1. Now, the numerator and lattice coefficients from new positions are extracted, and used for evaluation of objective function using Eq. (8). The lattice coefficients are transformed in to direct form using Eqs. (11) and  (12), after which the frequency response is computed by substituting the coefficients in Eq. (7), and at last the objective function is evaluated.

Table 1 Pseudocode for checking and re-initializing the out of bound values

New solution (position) that has achieved better objective function value (J) with respect to previous local best solution (Pbest) is now accepted and copied over it. A similar analysis is carried out for the global solution, that is if the new any of ‘Pbest’ solution has better objective function value than current ‘Gbest’, then it replaces the previous global best solution (Gbest). Now in last stage the replacement mechanism is carried out in which scout bee concept of ABC technique is adopted. If any vector whose solution is not improved, then its value is replaced by till known global best solution that is Gbest. Also, the velocity vector corresponding to unimproved solution is also replaced by the velocity recorded for Gbest. In this way, the swarm is updated in the direction of an optimal solution. Thus, it confirms the optimal point exploration and resists the trapping in local minima.

5 Simulation results and discussion for efficient design of optimal IIR filter

In this section, the design of IIR filter using PSO, ABC and various developed hybrid PSO techniques are conducted. The comprehensive simulations have been carried out in order to analyze and select the suitable variant of PSO and then to develop the computationally improved hybrid version.

Table 2 Control parameters for the proposed technique and different algorithms

Table 3 Design specification for the prototype IIR filter

Table 4 mean, variance, standard deviation, best and worst values of J for each method for the 30 trials used to design the filter mentioned in Table 3 with the population size of 20

Table 5 mean, variance, standard deviation, best and worst values of J each method for the 30 trials used to design the filter mentioned in Table 3 with the population size of 30

Table 6 mean, variance, standard deviation, best and worst values of J for each method for the 30 trials used to design the filter mentioned in Table 3 with the population size of 40

5.1 Design examples and parameter specifications

Various control parameters of different optimization techniques required in the proposed methodology are summarized in Table 2, which has been taken from the extensive analysis of literature that reflects their practice in various optimization problems such as filter designing and numerical optimization testing (Karaboga and Basturk 2007b; Sheng and Bing 2010). These values involve cognitive and social scaling parameters, inertia weight, limit, and modulation index. In Table 2, the value of $C_{1}$ and $C_{2}$ are taken from work proposed in Sheng and Bing (2010) and Bansal et al. (2011). Value of linearly varying w for LDI-PSO and CWI-PSO has been considered from Bansal et al. (2011). The suitable value for limit, velocity element (v), and swarm size (population size) are incorporated by experimental analysis. The inertia weight strategy for DI-PSO is modified as it observed from the analysis made in Agrawal et al. (2015b) that inertia weight should be in between 0.1 and 0.7, as it make the algorithm stable for IIR filter design. Previously, w in DI-PSO was governed as:

$$\begin{aligned} w=0.5+\frac{{ rand}\left( \cdot \right) }{2} \end{aligned}$$

(16)

whereas, in the new strategy is adopted, in which w is regenerated as:

$$\begin{aligned} w={ ll}+\frac{\left( {{ ul}-{ ll}} \right) \cdot { rand}\left( \cdot \right) }{2} \end{aligned}$$

(17)

In above equation ll and ul are the suitable limits that are used to improve the convergence of algorithm during exploration, selected as 0.1 and 0.6, respectively (Kumar et al. 2012).

Table 7 mean, variance, standard deviation, best and worst values of J for each method for the 30 trials used to design the filter mentioned in Table 3 with the population size of 50

5.2 Comparison of Hybrid methods with swarm-based techniques.

In this section, the several examples are considered for the design of optimal IIR filter using PSO, ABC and various developed hybrid PSO techniques, which are listed in Table 3. The first experiment is carried out for analyzing the effect of search space matrix (swarm) on filter performance based on the value of J for the swarm-based techniques. In this experiment, all filters are designed using proposed methods for thirty independent trials, with different swarm size of 20, 30, 40 and 50. The necessary parameters are evaluated and summarized for respective swarm size in Tables 4, 5, 6 and 7. It is evident from Table 4 that, among CWI-PSO, LDI-PSO, CFI-PSO, MDI-PSO TVC-PSO and ABC algorithm, last three techniques of PSO family are able to achieve better value of J. Moreover, the standard deviation and mean value achieved by CFI-PSO is quite better than TVC-PSO and MDI-PSO, for the population size of 20 in lower-order filter examples. This leads to the development of three hybridized method using the concept of scout bee of ABC algorithm with these variant of PSO and entitled as Hybrid-1 (MDI-PSO with ABC), Hybrid-2 (CFI-PSO with ABC) and Hybrid-3 (TVC-PSO with ABC), respectively. The hybridization has resulted in significant improvement in preformation of the proposed techniques tabularized in Table 4. Similar analysis has been performed for other swarm sizes, and performances are summarized in Tables 5, 6 and 7, respectively. It is evident from these Tables that the proposed Hybrid-3 performs better, when compared with above discussed techniques due to their time varying strategy of the control coefficients, which not only helps it for better exploration and exploitation, but also leads it to possess less computation time with superior ability of handling a higher order design problem efficiently than others. It has observed that the proposed Hybrid-3 method provides sustainable performance for all population sizes, which is also reflected from the evaluation of mean, obtained for various methods as depicted in Fig. 2.

Fig. 2

a mean of J for quoted techniques for 10 used examples for the population size of 20. b mean of J for quoted techniques for 10 used examples for the population size of 30. c mean of J for quoted techniques for 10 used examples for the population size of 40. d mean of J for quoted techniques for 10 used examples for the population size of 50

Fig. 3

a Population size effect on LPF filter for different order. b Population size effect on HPF filter for different order

Table 8 Comparative analysis of best fidelity parameters obtained for proposed algorithm with different techniques

In second experiment, the effects of population size on filter performance for different orders are studied specifically for Hybrid-3 method. For this purpose, same filter specifications of example-5 and example-11 are utilized for different orders ranges from 2 to 15 with increment of 1, and in swarm sizes from 5 to 50 with linear increment of 5. It can be observed from Fig. 3 that the proposed technique works consistently efficient, for the entire range of filter orders and swarm sizes. On the basis of above discussed experiments, it is evident that the better performance in term of J can be achieved irrespective of filter order, which can be further utilized for designing the both LPF and HPF with less computation cost. In addition to this, the designed filter using proposed technique shows better fidelity parameter values than conventional ABC, PSO and its variant technique, which is summarized in Table 8.

Table 9 Computational complexity measured in terms of O-notation

5.3 Complexity of the algorithm

Computation complexity of the proposed method is measured in term of ‘O-notation’. It is evident from Table 9 that computation complexity of proposed hybrid method is ${\textit{O}}({\textit{n}}^{2})$, as only additional search and replacement of unimproved solution is executed. However, search mechanism of the proposed technique is improved as compared to other hybrid algorithm with less complexity. Computation time and total number of function evaluation (NFE) involved has been slightly increased compared to non-hybrid PSO.

Fig. 4

Effect of truncation and quantization of filter tap coefficients obtained by a, b for algorithm proposed in (Saha et al. 2013) c Hybrid-3 for LPF, d Hybrid-3 for HPF

5.4 Comparison with other existing methods

For the justification and significance of proposed technique in optimal design of IIR filter with other existing techniques, a numerical example has been taken and mentioned as example 6 and example 11 in Table 3 (Saha et al. 2013). The best performances corresponding to all developed methods are compared and tabularized in Table 8, which clearly indicates that the performance of proposed method is better than previously quoted versions of PSO and ABC algorithms. Moreover, the performance of the proposed technique is also evaluated by calculating the filter fidelity parameters such as: passband error, stopband error, and maximum stopband attenuation defined as (Rafi et al. 2013):

$$\begin{aligned}&e_p =\sum _{\omega \in \omega _p } {\left[ {D\left( \omega \right) -H\left( \omega \right) } \right] } ^{2} \end{aligned}$$

(18)

$$\begin{aligned}&e_s =\sum _{\omega \in \omega _s } {\left[ {D\left( \omega \right) -H\left( \omega \right) } \right] } ^{2} \end{aligned}$$

(19)

$$\begin{aligned}&{} { SE}=e_p +e_s \end{aligned}$$

(20)

$$\begin{aligned}&A_{{ stop}} =-20\cdot \log _{10} \left( {\left| {H(\omega )} \right| } \right) ,\,at\,\omega =\omega _s \end{aligned}$$

(21)

In previous techniques, the obtained solution for optimal IIR often suffers from quantization and truncation effect (Saha et al. 2011, 2012b, a, 2013), in which filter response is degraded during the truncating and quantizing the coefficients of numerator and denominator polynomials, as shown in Fig. 4a, b; therefore; computation cost would be a little bit higher for this system. However, the solution explored by proposed method has shown immunity to quantization effect, and sustained performance has recorded with lesser number of bits, as shown in Fig. 4c, d, respectively. These figures clearly indicate the better passband and stopband response with stability, where the poles lied inside the unity circle confirms the sustainable execution of stability mechanism, as depicted in Fig. 5. The power of exploring ability of proposed method is depicted in Fig. 6.

Fig. 5

a Poles and zero location for LPF designed using Hybrid-3. b Poles and zeros for HPF designed using Hybrid-3

Fig. 6

Convergence profile of various swarm-based methods. a HPF, b LPF

6 Conclusion

In this paper, an improved hybrid method is exploited for the design of optimal digital IIR filter based on minimization of nonlinear objective function constructed in frequency domain using prescribed passband and stopband ripples. The experiments based on statistical analysis evidence that the proposed hybrid method shows less deviation of fitness/ error function between best and worst values, when compared to other optimization methods. The proposed algorithm also helps for non-convex and non-differentiable design problem of IIR filter with acceptable fidelity parameters using finest swarm size, while the complexity remains same as for conventional PSO. Several design examples have been included to demonstrate the effect of swarm sizes and efficiency of proposed method to handle large-scale optimization problem. The simulation results illustrate that the proposed technique is efficient in term of stability, and designed filter does not suffer from degradation due to quantization effect. Therefore, the designed filter can be realized by quantizing using six bits only, hence helps in fast realization of filters. The incorporation of sorting scheme enables the system to be self-intelligent for selecting the best solution out of the executed trials. The proposed technique can be extended for fractional delay IIR filter design and also for reconfigurable IIR filter design.