Keywords

1 Introduction

Optimization refers to finding the best possible solution by varying the variables’ values with or without constraints. Due to the advent of computing machines and the speed of processing available today, solving complex optimization problems have become a reality. For smaller problems, evaluating the optimal solutions in a reasonable time is comparatively easier. When the number of dimensions and constraints increase, the computing time grows exponentially and many times, we will not be able to obtain the solution. As a result, heuristics and metaheuristics have been proposed by researchers which are capable of providing the results with acceptable accuracy in a reasonable time.

The development of optimization algorithms gained momentum since 1940. Earlier optimization algorithms generally make use of the gradient of a function and constraints to arrive at a solution. In such cases, the function shall be differentiable to move forward.

Gradient-based optimization makes use of the gradient to find the direction of the steepest increase. The stationary point will be reached when the gradient becomes zero. In such cases, the point need not be a global optimum. Based on the approach of choosing the search direction, several methods can be used like Newton, Quasi-Newton, Conjugate descent and steepest descent.

The first of its kind quasi-Newton concept was applied for solving non-linear problems in optimization by Davidon in 1959 [1] which was further developed by Fletcher and Powell [2]. The Sequential Quadratic Programming (SQP) was proposed by Robert Wilson which was made popular by the work of Powell [3].

Modeling the optimization problems using the process of natural evolution started in early 1950s. The pioneering work in this area can be credited to Holland [4] who proposed the Genetic Algorithm (GA).

Due to the difficulties faced in gradient-based algorithms, several gradient-free methods were proposed over the few decades. A wide range of classifications could be found in the “Engineering Design Optimization” book by Martins and Ning [5]. The classification is done based on the characteristics like search (local or global), optimal criteria and iteration criteria (mathematical or heuristic), function evaluation (direct or surrogate) and stochasticity (deterministic or stochastic). Rios and Sahinidis [6] considered a whopping 112,448 instances for analyzing different gradient-free algorithms. They concluded that the ability of these gradient-free solvers to yield better solutions diminishes when the problem size increases.

A gradient-free method termed as the “pattern search” was proposed by Hooke and Jeeves [7]. A revolutionary, new and fast method was proposed by Karmarkar in the year 1984 [8] to solve large scale linear programming problems.

Developments of evolution based algorithms are greatly influenced by a process that takes place in the society or nature. A majority of these nature inspired algorithms are population-based that evaluate several points during each iteration instead of one. These algorithms are stochastic in nature and assume different optimization problems as black boxes (Droste et. al., 2006 [9]) and hence, derivatives of functions are not required. The exploration (global search) is ensured by the widespread of the population within the bounds and their stochastic nature. The global minimum is not guaranteed and the convergence rate becomes poor with increasing dimensions.

Popular evolutionary algorithms include ant colony (ACO) optimization, bat (BA) algorithm, cuckoo (CS) search, firefly (FA) algorithm, hummingbirds (HBO) optimization algorithm, whale optimization (WOA) algorithm and teaching-learning based (TLBO) optimization. Rao [10] proposed the popular JAYA algorithm in 2016 which does not require any tuning parameter. It is one of the widely used and analysed population-based algorithms in this decade.

The Sine Cosine Algorithm (SCA) proposed by Mirjalili [11] is trigonometric functions based that generates a pool of candidate solutions initially using sine function and cosine function. During the solution process, three random numbers are generated. “The Whale Optimization Algorithm (WOA)” of Mirjalili and Lewis [12] is another efficient algorithm proposed in the same year. Two similar trigonometric algorithms were developed by Baskar [13, 14] and the performances were analyzed using 40 numbers of popular and time-tested benchmark test suite and, a few constrained engineering problems available in the literature. Abualigah et al. [15] proposed an efficient “Arithmetic Optimization Algorithm (AOA)” utilizing the distribution behavior of different mathematical operators (addition, subtraction, multiplication and division) during the optimization process. Agushaka and Ezugwu [16] further improved the AOA and solved a few mechanical engineering design problems.

A physics-inspired “Archimedes Optimization Metaheuristic Algorithm” was proposed based on the Archimedes principle and analyzed by Hashim et al. [17]. They claimed that the results obtained have outperformed a few well-known metaheuristic algorithms. CEC’17 test functions and four engineering real-world problems were used for analyzing the algorithm.

Any optimization algorithm may be a new one or an improvement over the existing one or a hybrid algorithm. Researchers apply all the three strategies effectively as it may not be possible to have a single algorithm that would be able to solve all kinds of optimization problems, as concluded by the ‘No Free Lunch Theorem’ proposed by Wolpert and Macready [18].

This paper proposes a hybrid local search algorithm (FP-AB) and the performance is analyzed using 93 test functions. The performances are then compared with the popular SCA, AOA, WOA, Grey Wolf Optimizer (GWO) [19], the hybrid WOAGWO [20], and WOA-BAT [21] algorithms. Finally, a statistical analysis has been carried out to compare FP-AB and AOA.

2 Optimization Problem Statement

The design variables and, single objective function with applicable constraints for a continuous optimization problem can be stated as:

$$ \begin{array}{*{20}c} {{\bf{Minimize}}\,{{\varvec{f}}}\left( {{\varvec{x}}} \right)\,{\text{such}}\,{\text{that,}}\,{\text{LB}} \le {{\varvec{x}}}_{{\varvec{i}}} \le {\text{UB}};{{\varvec{i}}} = {1}\,to\,n} \\ {{\text{Subject}}\,{\text{to}}\,{{\varvec{g}}}_{{\varvec{j}}} \left( {\bf{x}} \right){\bf{ \le 0}};\,{\text{for}}\,{{\varvec{j}}} = 1\,{\text{to}}\,{\text{g}}} \\ {{{\varvec{h}}}_{{\varvec{k}}} \left( {{\varvec{x}}} \right) = {\bf{0}};\,{\text{for}}\,{{\varvec{k}}} = 1\,{\text{to}}\,{\text{h}}} \\ \end{array} $$

LB and UB are lower and upper bounds for the design variables. The non-equality and equality constraints are denoted by g(x) and h(x) respectively. For a maximization problem the objective function is negated as:

$$ {\bf{Minimize}} - {{\varvec{f}}}({{\varvec{x}}}). $$

Generally; f, g and h are evaluated for all values of ‘x’ within the bounds.

3 New Hybrid Local Search Algorithm (FP-AB)

A local search algorithm that evaluates the cost functions at four-points around the existing one was applied by Baskar and Anthony Xavior [22] to locate any facility on different planar, spherical, and ellipsoidal surfaces. It selects the better one for the next iteration. Thus the location is iteratively moved towards the optimal / near-optimal location for a specific cost function.

Only the four point search strategy is borrowed from them and the newly proposed algorithm combines that with the population concept for effectively solving optimization problems which is an entirely different domain from the facility location. That is, the algorithm initially generates a population of 4*n (n = number of dimensions) solutions which are evaluated for their costs. The best solution is selected among them and then it is iteratively moved towards a better solution. Thus, this algorithm combines both the population and local search strategies to find an approximate solution which may be an optimal or near optimal one. Hence, this is called a “Hybrid Local Search Algorithm”. This could be demonstrated with an example.

Let,

Number of dimensions, dv = n (x1, x2, …, xn); Population size, PS = 4*n.

Consider a single objective optimization problem with three dimensions and the variables be; x1, x2 and x3. The solution bounds are LB (Lower Bound) and UB (Upper Bound).

Step 1: The initial population size will be, 4*3 = 12 and the solution set is randomly generated using the expression, LB + rand*(UB - LB).

Step 2: All the 12 solutions are evaluated for the cost function and the best one and its corresponding solution values, ‘Xbest [X1best, X2best and X3best]’ are selected.

Step 3: Now, pairs of solution values are selected; there will be ‘n’ number of pairs (n = 3 here).

$$ \left[ {{{\varvec{X}}}{\bf{1}}{{\varvec{best}}},\,{{\varvec{X}}}{\bf{2}}{{\varvec{best}}}} \right];\,\left[ {{{\varvec{X}}}{\bf{2}}{{\varvec{best}}},\,{{\varvec{X}}}{\bf{3}}{{\varvec{best}}}} \right]\,{{\varvec{and}}}\,\left[ {{{\varvec{X}}}{\bf{3}}{{\varvec{best}}},\,{{\varvec{X}}}{\bf{1}}{{\varvec{best}}}} \right] $$

Step 4: The new solution set (PS1 to PS12) is constructed by adding/ subtracting a small amount, “alpha” as follows:

  • Pair I: [X1best, X2best]:

    • PS1: {X1best + alpha}, {X2best + alpha}, X3best

    • PS2: {X1best - alpha}, {X2best + alpha}, X3best

    • PS3: {X1best - alpha}, {X2best - alpha}, X3best

    • PS4: {X1best + alpha}, {X2best - alpha}, X3best

  • Pair II: [X2best, X3best]:

    • PS5: X1best, {X2best + alpha}, {X3best + alpha}

    • PS6: X1best, {X2best - alpha}, {X3best + alpha}

    • PS7: X1best, {X2best - alpha}, {X3best - alpha}

    • PS8: X1best, {X2best + alpha}, {X3best - alpha}

  • Pair III: [X1best, X3best]:

    • PS9: {X1best + alpha}, X2best, {X3best + alpha}

    • PS10: {X1best + alpha}, X2best, {X3best - alpha}

    • PS11: {X1best - alpha}, X2best, {X3best - alpha}

    • PS12: {X1best - alpha}, X2best, {X3best + alpha}

Where, alpha = gamma*(0.5 + 0.4*rand())

$$ {\varvec{gamma = }}[{{\varvec{beta}}} - \left( {\left( {{{\varvec{beta}}}*{{\varvec{Current}}}\_{{\varvec{Iteration}}}\_{{\varvec{No}}}} \right)/{\varvec{ Max}}\_{{\varvec{No}}}\_{{\varvec{of}}}\_{{\varvec{Iterations}}}} \right)] $$

… (‘beta’ is a tuning parameter and varies from problem to problem).

The value of ‘gamma’ starts with ‘beta’ and approaches zero as the iteration progresses.

Step 5: The newly constructed solution set is evaluated for the cost and the best solution and its corresponding solution values are selected.

Step 6: Step 4 and Step 5 are repeated until the optimal solution is attained or the stopping criterion is reached.

For one dimensional and two dimensional problems, the population size should be 2 and 4 respectively and cannot be 1*4 = 4 and 4*2 = 8. However, in this paper, the population size, PS = 4*n is considered to have a wider solution set for one and two-dimensional problems as well. The population size cannot be a fixed one and, it is a function of the dimension. With the dimension, the population size also grows. This increases the computation time for higher dimension problems.

4 Benchmark Functions

For comparing the performance of any algorithm, numerous test functions are available in the literature. Several benchmark functions, their optimal values, search range and solution set used in this paper are collected [23]. Many functions are classified as tough in this portal and out of these, a few are analyzed separately. In addition, 10 instances of the 100 digit challenge of CEC2019 [24] and 13 instances from the paper, “The Arithmetic Optimization Algorithm” are also used in the analyses (Table 4 and Table 5). The test functions, their optimal values and search spaces are listed from Tables 1, 2, 3, 4, 5, and 6.

Table 1. Single Variable Functions
Full size table
Table 2. Popular Benchmark Functions
Full size table
Table 3. Hard Benchmark Functions
Full size table
Table 4. More Unimodal Benchmark Functions
Full size table
Table 5. More Multimodal Benchmark Functions
Full size table
Table 6. 100 Digit Challenge: Basic Test Suite (CEC2019)
Full size table

5 Computational Results and Analysis with Other Algorithms

In this section, the results of FP-AB are compared with a few other efficient algorithms like AOA, SCA. The original MATLAB codes used for the AOA are downloaded from the website [25]. The codes are run using MATLAB R2012b in an i5 PC with 4 GB RAM. The “Beta” values used in FP-AB are also given against each function.

Table 7 shows the results for 10 single variable functions. Sine Cosine Algorithm (SCA) and Arithmetic Optimization Algorithm (AOA) are used for this set of functions. FP-AB reports better results in 7/10 test functions when the ‘mean’ values are considered against 4/10 of AOA. When the ‘best’ values are considered, FP-AB could reach the optimal values in 9/10 cases whereas, for AOA and SCA, it is 8/10. The standard deviation is less in 7/10 problems for FP-AB whereas, the count is 3/10 for AOA and none for SCA.

Now, 49 numbers popular benchmark functions are analyzed for AOA and FP-AB. The ‘best’ (minimum), ‘mean’ (average), ‘worst’ (maximum), ‘SD’ (standard deviation) and time taken for 30 trials are computed.

FP-AB could reach the optimal values in 34/49 cases as against 29/49 of AOA. However, AOA dominates with 29/49 and 32/49 counts when the ‘mean’ and ‘standard deviation’ are considered. The total computation time for the execution of 49 test functions and 30 runs is 112.229784 s which is much less than the time taken (675.141266 s) by AOA.

For the 11 hard problems, FP-AB reports better results in 7/10 for the ‘mean’ and ‘best’ values followed by AOA which is accountable for 4/10 better results. However, the ‘standard deviation’ is better for AOA in 7/11 problems and for FP-AB, the count is 4/11. The performance of SCA is moderate here.

Table 7. One Dimensional Functions (500 iterations)
Full size table

Seven unimodal and six multimodal functions that were analysed in AOA paper are considered here also for 30 and 100 dimensions. Performance of AOA is better than FP-AB for the 7 unimodal functions whereas, FP-AB performs better in the case of multimodal test cases (Table 8).

Table 8. Count of Better Results: Unimodal and Multimodal Functions
Full size table

FP-AB could yield better values in 14/42 results of unimodal functions and 21/36 of multimodal cases when both 30 dimensions and 100 dimensions are combined. AOA performs better in the case of unimodal functions with 28/42 results and behind FP-AB with 15/36 results in the case of multimodal problems.

Table 9. Results of the 100 Digit Challenge (CEC2019) [30 Trials, 500 Iterations]
Full size table

Finally, ten test suites used in the popular 100-digit challenge of CEC2019 are evaluated for the ‘mean’ and ‘standard deviation’ metrics.

The program is run with AOA and FP-AB algorithms and the outputs are analyzed. Other results are taken from the paper of Mohammed and Rashid [20] for the purpose of comparison since the maximum iterations (500), population size (30) and trials (30) considered are the same in this paper also.

The results demonstrate the better performance of FP-AB when compared with WOA, WOA-GWO, GWO, WOA-BAT and SCA. FP-AB accounts for 7/10 better results followed by WOA-BAT with 2/10 and WOA-GWA with 1/10 results for the ‘mean’ metric. However, for the ‘standard deviation’, FP-AB could account for the 4 best values. In this case, GWO comes next with 3 best results and, WOA, WOA-GWO and SCA with 1 each.

When AOA and FP-AB are compared (Table 9), FP-AB performs well showing superior results in 9/10 cases (for ‘mean’ values) and, 8/10 cases (for the ‘standard deviation’). AOA could yield better values only in 2 cases each.

6 Statistical Analyses

The non-parametric Wilcoxon Signed-Rank Test is used for comparing the data and is used if the differences between pairs of data are not distributed normally. Since the sample size exceeds 16 in number, the test is conducted for the 49 popular benchmark functions. The differences between the ‘mean’ as well as ‘best’ values from the ‘optimal’ values are considered for the analysis. The results are presented in Table 10.

Table 10. Wilcoxon Signed-Rank Test
Full size table

The P-value is zero for both which indicates that the difference between the population median and the hypothesized median is statistically significant (rejecting the null hypothesis) when the ‘mean’ values are considered. However, the Wilcoxon Statistic for AOA is 468.0 against 635.5 of FP-AB which confirms the better performance of AOA over FP-AB.

When the ‘best’ values are analyzed, the P-value is 0.001 for both AOA and FP-AB which is again < 0.05. This shows that the difference between the population median and the hypothesized median is statistically significant here also. However, the Wilcoxon Statistic for FP-AB is 211.0 which is less than that of AOA indicating the better performance of FP-AB over AOA for the ‘best’ values.

The estimated median is smaller for FP-AB in both the cases of ‘mean’ and ‘best’ values. Table 11 shows the results of Mann-Whitney tests for both ‘mean’ and ‘best’ values.

Table 11. Mann-Whitney Test
Full size table

It can be concluded that there is no significant difference statistically in median engagement between AOA and FP-AB since the p-value reported is greater than 0.05. However, the p-value is less for the ‘best’ values.

7 Conclusions, Limitation and Future Work

This work proposed and analyzed a simple hybrid local search optimization algorithm, FP-AB using 93 unconstrained benchmark functions. The obtained results are compared with a few recent efficient algorithms available in the literature including the recent Arithmetic Optimization Algorithm (AOA). The analyses confirm the efficacy of the new hybrid algorithm; FP-AB which is comparable with AOA and a fair competitor to other optimization algorithms. The limitation of the new algorithm is that the computation time increases with increasing dimensions since the initial population size is dynamically fixed as 4*dimensions. Future work includes an analysis of the new algorithm for constrained and real-world engineering optimization problems.