1 Introduction

1.1 General introduction

To ensure the efficient operation of a modern power system (PS), the total power generated by conventional and other sources must be the same as the sum of load demand (\(P_{D}\)) plus transmission line losses (\(P_{L}\)). The increasing global population’s energy demands necessitate the expansion of electrical infrastructure and the ramping up of power production (Amiri and Hatami 2023). In the context of modern power systems, where multiple power control areas are interconnected, the structural complexity becomes apparent. Abrupt shifts in loading conditions result in a mismatch between the mechanical power generated by the turbines and the electrical power produced by the generators, consequently causing notable disruptions in key system parameters, including area frequency and tie-line power (FTP) flow. To mitigate these challenges and ensure the stability of the power grid, automatic generation control (AGC) plays a pivotal role in regulating system power output and maintaining FTP power at its designated values, especially during sudden load variations (Gouran-Orimi and Ghasemi-Marzbali 2023).

1.2 Literature survey

AGC is one of the best significant secondary services for controlling power supply among different sections of a complex interconnected power system while maintaining a consistent frequency (Vigya et al. 2021; Ranjan and Shankar 2022). The majority of power production is produced by conventional energy sources, hence authors have primarily examined the effectiveness of AGC in systems that use thermal, hydroelectric, and gas-producing units (Sahu et al. 2020; Aryan and Raja 2022). Due to changes in human lifestyles, declining availability of conventional sources, increased supply and demand, industrialization, and environmental concerns, there is a growing emphasis on integrating RES like solar, wind, biomass, and other renewables. Many national governments have set ambitious targets for promoting renewable energy (Saha et al. 2022). Currently, renewable energy accounts for 14% of the global energy supply. It is expected that by 2050 the electricity to be produced by RES will rise to 70–85%. But the output power of the RE sources is intermittent in nature, which means depends on the environmental conditions and changes during the daytime. As a result, a huge number of renewable energy resources have been incorporated into power networks, posing new difficulties and possibilities for grid management and planning (Prakash et al. 2019; Kumar Khadanga et al. 2021). High renewable energy penetration on the grid can lead to supply and demand mismatches, voltage fluctuations, and even instability of the network. For enhancement of the overall performance and system dependability, it becomes necessary to eliminate the intermittency of RES backup in the energy storage form.

Because of their quick response and precise management, energy storage systems (ESS) are particularly successful at adapting to a doubtful frequency fluctuation, according to several studies provided in the literature (Ramoji and Saikia 2022). By compensating for insufficient power and absorbing surplus power under dynamic conditions, ESS reduces the grid frequency deviations, hence enhancing the power quality of an electrical power system. Several studies have been conducted that show how ESS units can enhance the system dynamic behaviour of traditional and deregulated power systems in terms of smooth frequency regulation (Mishra et al. 2022). Numerous ESS are available in the market, but each unit has certain disadvantages like low power density, high cost, lower life span, etc. Therefore, it is important to move towards other feasible solutions to make the ESS more favourable in the LFC of investigated power system applications. The combination of different ESS, i.e. hybrid energy storage system (HESS) with strong points and shortcomings can be explored to regulate the high surge currents affordably. Few researchers (Behera and Saikia 2023) have proposed HESS in LFC modelling for improved system performance.

In thermal power plants, non-linearities similar to generation-rate-constraint, GDB, and BD are utilized to enhance the accuracy and realism of the PS (Parmar et al. 2014; Patel and Sahu 2021). However, these non-linearities also make the system more complicated, and, therefore, a good controlling system is essential to address stability issues related to tie-line power and frequency. Previous studies have focused on using regular control systems like I, PI, PD, and PID for AGC, but traditional controllers have drawbacks such as slower response time, poor convergence response, and higher susceptibility (Arya and Kumar 2017; Nayak et al. 2019). Some authors have suggested using fractional order (FO) controllers due to their greater flexibility in altering system dynamics (Dekaraja and Saikia 2022; Morsali 2022). Fuzzy logic controllers (FLC) are also recommended for dynamic controllers in uncertain environments, but they are time and space-intensive and sensitive to rule base and membership functions (Chen et al. 2020). Its weaknesses are time and space complexity, sensitivity to the rule base selection, and membership functions. For tracking setpoint and enhancing disturbance rejection, cascade controllers are preferable to classical controllers. Cascade controllers also have the ability to reject load disturbances, strengthen outcomes, accurately manipulate energy or mass, and compensate for uncertainties (Raj and Shankar 2023).

The secondary controller can function flawlessly with the appropriate calibration of its gain, and both classical and metaheuristic algorithms can achieve this calibration. Metaheuristic algorithms, such as the artificial gorilla troops optimizer (Ahmed et al. 2022), sailfish optimizer (Rai and Das 2022), arithmetic optimization algorithm (Anand et al. 2022), slim mode algorithm (Sharma et al. 2022), artificial bee colony (Taghvaei et al. 2022), Harris hawks optimization (Abubakr et al. 2021), quasi opposition Harris hawks optimization (Saxena et al. 2022), reinforcement learning algorithm (Yin and Wu 2022), sine augmented scaled arithmetic optimization algorithm (Kumar et al. 2023a) have been used recently in various studies for tuning controller gains.

1.3 Research gap and motivation

After reviewing the literature, it can be observed that many researchers have conducted studies on deregulated automatic generation control (AGC) systems, but only a few have focused on integrating energy storage systems (ESS) into the grid to enhance frequency response and tie-line stability. To the author's knowledge, no research has been done on the integration of HESS taking into account the fluctuating nature of solar and wind power in the proposed PS for frequency management. Additionally, there is a shortage of real-time HIL simulation analysis to evaluate the impact of RES and HESS in AGC. Some researchers have utilized cascade controllers to improve frequency regulation against various scenarios such as the intermittent nature of solar and wind, non-linearities, and random load conditions (Lin et al. 2022). To address the current topic, this study proposes a cascade fractional order proportional integral derivative with filter (FOPIDN) and a one plus tilted derivative controller (1 + TD) to effectively address AGC concerns (Choudhary et al. 2022). Additionally, optimization methods have been used to assess AGC's output performance. The productivity of the power system is significantly impacted by these optimization strategies. One of the various optimization techniques, the arithmetic optimization approach has shown significant improvement over other contemporary procedures due to its gradient-free mechanism and good local optimum avoidance capabilities. The HSA algorithm, which includes quasi-opposition learning, has also demonstrated improved convergence and the ability to avoid local minima (Shiva et al. 2022).

1.4 Contribution

The introduction and literature review in this research provided an outline of the need for an improved, intelligent controller to enhance AGC performance. The paper proposes the use of a Quasi-opposition-based AOA algorithm optimized cascade FOPIDN-(1 + TD) controller for frequency management of a multiple-zone PS, considering the effect of variable solar and wind. The hybrid energy storage element with SC and RFB is presented as a solution to mitigate unfavourable transient situations on load frequency and power-sharing in a RES-based DG system. The paper advances the study of the LFC mechanism in numerous aspects.

The paper's key findings can be summarized as:

  1. (1)

    The paper investigates the AGC of a three-area redesigned power structure using multiple-generation units such as thermal, gas, renewable energy, and hybrid energy storage, considering non-linearity caused by GRC, GDB, and BD.

  2. (2)

    A novel cascade FOPIDN-(1 + TD) controller for AGC is proposed and its superiority and feasibility over PID, FOPIDN, and (1 + TD) controllers are discussed under step and random load conditions.

  3. (3)

    The paper proposes a new modified quasi-opposition arithmetic optimization algorithm (QOAOA) to tune the scaling factor of the suggested controller. The dynamic response and figure of demerit (FOD) of the suggested algorithm relative to other recent evolutionary techniques are also investigated.

  4. (4)

    The impact of RFB, super-capacitor, and HESS on the dynamic performance of the investigated power system under the intermittent nature of RES is thoroughly investigated.

  5. (5)

    A comparative analysis of simulation results of the suggested controller with other relevant literature (Parmar et al. 2014) and (Shankar et al. 2016) on the same platform is performed.

  6. (6)

    Finally, the paper evaluates the practicability and effectiveness of the suggested controller on a real-time platform using a hardware-in-the-loop (HIL)-based OPAL-RT simulator.

1.5 Organization of the paper

The rest of the paper is organized as follows: Sect. 2 covers the power system analysis and examination. Section 3 outlines the proposed controller design approach. In Sect. 4, the AOA, the proposed QOAOA, and the superiority of the proposed controller are discussed. Section 5 presents the simulation and experimental results of different case studies. Finally, Sect. 6 provides the conclusion of the paper.

2 Proposed system under study

Figure 1 illustrates the proposed power system model. The paper considers a three-area smart grid system with an overall generation capacity of 2000 MW, where each area includes a conventional thermal power unit, a bio-gas generating plant, HESS, SPV, and a wind power turbine. The non-linearities, namely GDB, GRC, and BD, are taken into account for each plant (Saxena and Shankar 2022). The proposed Secondary Regulation (SR) controller is a cascade FOPIDN-(1 + TD) controller, which is designed to reduce deviations in zone frequency and interline power interchange with insignificant overshoot and a fast settling time. To tune the controller, a novel adaptation of the QOAOA optimization method is used. According to several research papers (Ahmed et al. 2022; Aryan and Raja 2022) Integral square error quadratic performance index provides a satisfactory balance between performance and control effort.

Fig. 1
figure 1

Overall diagrammatic representation of a proposed system

Therefore, it has been selected as the objective function in this study. The ISE equation is given by:

$$ F_{{{\text{ISE}}}} = \int\limits_{0}^{{T_{{{\text{sim}}}} }} {\left\{ {\left| {\Delta f_{1} } \right|^{2} + \left| {\Delta f_{2} } \right|^{2} + \left| {\Delta f_{3} } \right|^{2} + \left| {\Delta P_{{{\text{tie}}12}} } \right|^{2} + \left| {\Delta P_{{{\text{tie}}13}} } \right|^{2} + \left| {\Delta P_{{{\text{tie}}23}} } \right|^{2} } \right\}} dt. $$
(1)

This equation includes the variables \(\Delta f_{1}\), \(\Delta f_{2}\), \(\Delta f_{3}\) and \(\Delta P_{tie12}\), \(\Delta P_{tie23}\), \(\Delta P_{tie13}\), which represent the deviation in frequency and tie-line power flow for Area 1, Area 2, and Area 3, respectively.

2.1 Distributed generation (DG)

To bring a practical approach to RES integration in automatic generation control the investigated system is evaluated with the effect of constant and variable weather conditions. Solar-photovoltaic system and wind turbine system are connected in each area of a proposed system. In the case of variable weather condition solar photovoltaic and wind turbine provides an uncertain random input power to the system. The following sections go through the various constituents of the planned DG system in the AGC scheme (Kumar et al. 2023b).

2.1.1 Solar-photovoltaic system (SPV)

SPV refers to an energy source which involves the utilization of a photovoltaic system for the purpose of tapping into the sun’s energy. Solar insolation serves as the source of input for the photovoltaic array. The power produced by the PV array is as follows:

$$ P_{{{\text{SPV}}}} = \eta S\Psi \left\{ {1 - 5 \times 10^{ - 3} \left( {T_{a} + 25} \right)} \right\}, $$
(2)

where \(\eta\) is a conversion-efficiency measured in percentages ranging from 9 to 12 percent. The area of the SPV (\(S\)) array is measured in square meters (m2), the solar insolation (\(\Psi\)) is measured in kilowatts per square meter (kw/m2), and the ambient temperature (\(T_{a}\)) is measured in degrees Celsius (0c). For SPV power generation, the simplified SPV transfer function is expressed as:

$${{\text{G}}}_{{\text{SPV}}}=\frac{{K}_{{\text{SPV}}}}{1+{ST}_{{\text{SPV}}}},$$
(3)

where \({K}_{{\text{SPV}}}=1\) and \({T}_{{\text{SPV}}}=1.8 {\text{s}}\) are the tuning parameters and time constant of SPV.

2.1.2 Wind turbine system (WTS)

The power production of WTS is proportional to the wind speed at any given moment. WTS might be a viable alternative to conventional fossil fuels because it produces no greenhouse emissions. These systems are constantly fitted with an energy-storing device because of the variable nature of wind. The wind turbine system generates the following amount of output power:

$$ P_{{{\text{WTS}}}} = 0.5\rho A_{r} C_{p} (\lambda ,\alpha )V_{w}^{3} . $$
(4)

The WTS transfer function is expressed below, where the symbols \(\rho\), \(Cp(\lambda ,\alpha )\),\(A_{r}\), \(\lambda\), \(\alpha\), and \(V_{w}\) used signify the air density, the power conversion coefficient, the swept area of the blade, the tip speed ratio, the pitch angle of the blade, and the wind speed. The gain and time constant of WTS are denoted by \({K}_{{\text{WTS}}}=1\) and \({T}_{WTS}=1.5s\):

$${{\text{G}}}_{{\text{WTS}}}=\frac{{K}_{{\text{WTS}}}}{1+{{\text{ST}}}_{{\text{WTS}}}}.$$
(5)

2.2 Energy storage system (ESS)

In the test system, an ES device is installed to offset the negative effects of RES inertia reduction. Though there are numerous ES on the market, battery energy storage with a higher energy density is the most popular due to its low cost (Rangi et al. 2022). The RESs have a variable power output compared to batteries, which have a relatively low power density. Batteries have a very tough time continuing to operate after sharp power fluctuations. Hence in this section, RFB and super-capacitor are hybridized to counter the variable nature of renewable energy sources.

2.2.1 Redox flow battery (RFB)

The conversion of energy (from electrical to chemical) takes place in a specific kind of electrochemical cell called an RFB. Vanadium ions and sulphuric acid act as electrolytes in RFB. The following is a list of basic chemical reactions (Sharma et al. 2021b):

$$ \begin{gathered} {\text{At positive electrodes}}\quad V^{4 + } \Leftrightarrow V^{5 + } + e^{ - } , \hfill \\ {\text{At negative electrodes}}\quad V^{3 + } + e^{ - } \Leftrightarrow V^{2 + } . \hfill \\ \end{gathered} $$
(6)

This technique allows for the controlled backup use of the energy that has been stored as needed. RFB is more well-known than other conventional batteries because of a variety of attributes and features. RFBs stand out for their better efficiency, quick response times during charging and discharging, portability, low cost, and no negative effects on life when regularly charged or discharged. The ability to handle a broad range of operational powers and discharge times makes redox flow batteries (RFBs) a fitting option for sustaining electricity generation from renewable sources. The most favourable power ratings and storage periods for RFBs fall within the range of KW to MW and 2–10 h, respectively. Figure 2a illustrates the transfer function model of RFB.

Fig. 2
figure 2

Transfer function model of: a redox flow battery, b hybrid energy storage system

2.2.2 Super-capacitor (SC)

The ultra-capacitor is a novel electrochemical-ESS commonly referred to as a super-capacitor or electric double-layer capacitor. It consists of an ion-exchange layer, a potassium hydroxide electrolyte, and two permeable anodes. The thinness of the double layer in SC results in a higher specific capacitance, which explains its exceptional capacitance (100–1000 times greater than that of typical capacitors) (Sharma et al. 2021a). Compared to conventional capacitors and batteries, SC has several advantages, such as small size, high energy density, longer lifespan, low maintenance, and quick charging and discharging cycles. Therefore, super-capacitors are well-suited for frequency control in a multi-zone smart grid system. The transfer function of SC is presented in the following equation:

$$ G_{SC} = \frac{{K_{SC} }}{{1 + sT_{SC} }}. $$
(7)

2.2.3 Hybrid energy storage system (HESS)

In a perfect world, an energy storage (ES) device would need high power density to survive sharp power changes and high energy density to supply power for a longer period of time. To achieve this, multiple energy storage technologies are used in the modeling of the HESS to create an efficient ES tool. To supplement the low power density of batteries, additional energy storage systems with higher power densities, like superconducting SMES, SC, and flywheels can be combined with them. For energy storage (ES) units to operate correctly, the total energy discharged and stored must balance each other out. Consequently, if the power usage surpasses the rated capability of an ES unit, the ES control becomes ineffective. Thus, for HESS, the bounds for \(p_{\max }\) and \(p_{\min }\) have been established as 0.005 puMW for RFB and SC, respectively, taking into account the system’s base and rated power. Figure 2b describes the transfer function model of the hybrid energy storage system (HESS).

2.3 Stability of the proposed system

According to classical control theories, the effective performance of a feedback control system depends on the system's stability. The power system under investigation exhibits non-linear behavior due to various system components such as GDB, BD, and GRC. When assessing the reliability of the proposed power system, one crucial parameter that demands analysis is its stability. In this section, transfer functions for different segments of the proposed system are computed using the MATLAB platform. The open-loop gain transfer function for Area 1 can be determined using MATLAB, as shown in Eq. (8). Where \({{Gp}}_{1} (s)\) and \({{Gec}}_{1} (s)\) are the plant and controller transfer function. The open-loop transfer functions for Area 2 and Area 3 can also be determined using a similar approach. In Fig. 3, a Bode diagram is generated for the open-loop transfer function in a distinct area to assess the closed-loop system’s stability. A comprehensive stability analysis considers two scenarios: one with controllers and one without controllers. The analysis indicates that the system is inherently stable with gain margin and phase margin of (42.035, 28.32), (53.43, 30.72), and (43.45, 32.684) in Area 1, Area 2, and Area 3, and the implementation of the recommended controllers improves the system’s frequency-domain stability margin, as depicted in Fig. 3. In Table 1, key parameters such as gain margin (\({\text{GM}}\)), phase margin (\({\text{PM}}\)), gain, and phase crossover frequencies (\(\omega_{gc}\), \(\omega_{pc}\)) are presented.

$$ {{Gec}}_{1} (s){{Gp}}_{1} (s) = \frac{{\left( \begin{gathered} 3.2 \times 10^{3} S^{19} + 329.2s^{18} + 4.32 \times 10^{6} s^{17} + 1.8 \times 10^{5} s^{16} + 3.23 \times 10^{8} s^{15} + 2.361 \times 10^{12} s^{14} + 2.36 \times 10^{12} s^{14} + 6.79 \times 10^{13} s^{13} \hfill \\ + 3.63 \times 10^{15} s^{12} + 3.35 \times 10^{15} s^{11} + 3.843 \times 10^{6} s^{10} + 1.207 \times 10^{17} s^{9} + 2.347 \times 10^{17} s^{8} + 2.832 \times 10^{17} s^{7} \hfill \\ + 1.958 \times 10^{17} s^{6} + 6.868 \times 10^{16} s^{5} + 1.23 \times 10^{16} s^{4} + 8.178 \times 10^{14} s^{3} + 2.147 \times 10^{13} s^{2} + 1.68 \times 10^{11} s + 3.023 \times 10^{8} \hfill \\ \end{gathered} \right)}}{{\left( \begin{gathered} 1.32 \times 10^{5} s^{21} + 1.74 \times 10^{4} s^{20} + 2.9 \times 10^{5} s^{19} + 1.68 \times 107s^{18} + 5.6 \times 10^{12} s^{17} + 1.2 \times 10^{12} s^{16} + 2.34 \times 10^{13} s^{15} + 4.82 \times 10^{14} s^{14} \hfill \\ + 4.889 \times 10^{15} s^{13} + 3.078 \times 10^{16} s^{12} + 1.22 \times 10^{17} s^{11} + 3.011 \times 10^{17} s^{10} + 4.485 \times 10^{17} s^{9} + 3.83 \times 10^{17} s^{8} + 1.769 \times 10^{17} s^{7} \hfill \\ + 4.125 \times 10^{16} s^{6} + 4.851 \times 10^{15} s^{5} + 2.896 \times 10^{14} s^{4} + 8.403 \times 10^{12} s^{3} + 1.024 \times 10^{11} s^{2} + 4.081 \times 10^{8} s + 3.874 \times 10^{5} \hfill \\ \end{gathered} \right)}}. $$
(8)
Fig. 3
figure 3

Bode-plots stability of the proposed system considering loop-transfer function: a Area 1, b Area 2, c Area 3

Table 1 Stability margin of closed-loop system

3 Designed control strategy of cascade FOPIDN-(1 + TD) controller

The proposed system controller comprises of two cascaded controllers: FOPIDN and (1 + TD) controller. While the industry commonly employs the PIDN controller due to its simple design and limited tunable parameters, it exhibits performance degradation in nonlinear systems because of its linear and symmetric nature.

In contrast, the FO controller offers several benefits over the integral-order (IO) conventional controller as it has additional tuning parameters, \(\lambda\) and \(\mu\), provides more degrees of freedom (Sahu et al. 2020; Aryan and Raja 2022). Overshoot and settling times are best handled by FO but computational time increases due to the presence of more parameters. FO controllers are better suited for handling overshoot and settling times but require increased computational time due to the presence of more parameters. The N-filter parameters are used in conjunction with a differential controller to reduce system noise. An extension of the FO controller, the tilted-integral-derivative controller, is analogous to the proportional integral derivative controller, apart from the measurement parameter \(k_{p}\) is replaced by \(k_{t} s^{ - b}\), where \(b\) is a tilt parameter and \(k_{p}\) = \(k_{t}\) (Saxena and Shankar 2022). The proposed cascade FOPIDN-(1 + TD) controller is designed for the investigated frequency regulation mechanism, which benefits from the advantages of both controllers. Cascade controllers are preferable over traditional controllers as they enhance disturbance rejection, accurately manipulate energy or mass, reject load disturbance, compensate for uncertainties, and track set-points (Prakash et al. 2020). A simplified representation of cascade controller’s is presented in Fig. 4 where the FOPIDN and (1 + TD) controller's transfer-function is written as [17, 28]:

Fig. 4
figure 4

Structure of proposed cascade FOPIDN-(1 + TD) controller

$$ G_{{{\text{FOPIDN}}}} (S) = \left( {k_{p} + \frac{{k_{i} }}{{s^{a} }} + \frac{{N_{1} k_{d} s}}{{s + N_{1} }}} \right), $$
(9)
$$ G_{(1 + TD)} (S) = \left( {1 + \frac{{K_{t} }}{{s^{b} }} + \frac{{N_{2} K_{D} s}}{{s + N_{2} }}} \right). $$
(10)

The optimization process aims to minimalize the objective function, which is the ISE as defined in Eq. (1), while meeting all the constraints listed below.

\(k_{p,\min }\) ≤ \(k_{p}\) ≤ \(k_{p,\max }\); \(k_{i,\min }\) ≤ \(k_{i}\) ≤ \(k_{i,\max }\); \(k_{d,\min }\) ≤ \(k_{d}\) ≤ \(k_{d,\max }\); \(a_{\min }\) ≤ \(a\) ≤ \(a_{\max }\); \(K_{t,\min }\) ≤ \(K_{t}\) ≤ \(K_{t,\max }\);\(K_{D,\min }\) ≤ \(K_{D}\) ≤ \(K_{D,\max }\); \(b_{\min }\) ≤ \(b\) ≤ \(b_{\max }\); \(N_{1,\min }\) ≤ \(N_{1}\) ≤ \(N_{1,\max }\); \(N_{2,\min }\) ≤ \(N_{2}\) ≤ \(N_{2,\max }\) are the gain tuning parameter and the subscript min, and max represents its minimum and maximum values. The range of best controller gain setting obtained using QOAOA is between [0 10] for integer gains (\(k_{p}\), \(k_{i}\), \(k_{d}\), \(K_{t}\) and \(K_{D}\)), [0 1] for fractional-order integrator (\(a\)) and tilt parameter (\(b\)), and [0 400] for the derivative filter (\(N_{1}\) and \(N_{2}\)).

4 Optimization technique

4.1 A brief review of meta-heuristic-based optimization techniques

In recent decades, the escalating complexity of real-world problems has spurred the demand for more dependable optimization techniques, with a particular emphasis on meta-heuristic optimization algorithms. These approaches, largely stochastic in nature, excel at approximating optimal solutions across a wide spectrum of optimization challenges. Their ascendancy over traditional optimization algorithms stems from their unique features, such as the absence of gradients and a remarkable capacity to sidestep local optima. A multitude of meta-heuristic-based optimization algorithms is available, catering to both single and multi-objective optimization scenarios. Moreover, some of these algorithms have been adapted for continuous optimization problems, while others have been tailored to address discrete optimization challenges. Hosseini Shirvani (2020, 2021) used the PSO algorithm to solve single-objective and multi-objective continuous optimization problems in cloud computing which is a broad and cost-efficient IT industry. However, Ekhlas et al. (2023) and Alaie et al. (2023) proposed a discrete grey wolf optimization algorithm and multi-objective discrete cuckoo search optimization to solve single-objective and multi-objective discrete optimization problems. In addition, hybrid approaches are there to enhance the gained solutions. For instance, Tanha et al. (2021) proposed a hybrid genetic algorithm thermodynamic simulated annealing (GATSA) algorithm to solve complicated and large-scale engineering problems.

4.2 Arithmetic optimization algorithm (AOA)

Seyedali Mirjalili proposed an AOA (Abualigah et al. 2021) that leverages the dispersion properties of commonly used mathematical operations such as addition (A), subtraction (S), multiplication (M), and division (D). However, AOA may sometimes get stuck at a regional optimum, despite its effectiveness compared to other conventional methods. To address this issue, the current study proposes to enhance AOA by incorporating quasi-opposition-based learning (QOBL) techniques. As a result, computational performance and the likelihood of not getting stuck in local optima are improved. Thus, both computing efficiency and the likelihood of avoiding local optima are increased. Various steps in the modified QOAOA algorithm are shown in Fig. 5.

Fig. 5
figure 5

Flowchart of modified QOAOA algorithm

4.3 Proposed modified quasi-opposition arithmetic optimization algorithm (QOAOA)

The aim of this study is to introduce the concept of quasi-oppositional-based learning (QOBL) (Shankar et al. 2016; Vigya et al. 2021) and how it can be utilized to accelerate convergence and achieve a globally optimal solution in the context of the AOA. Unlike opposition-based learning which only generates opposite values, QOBL generates both opposite and quasi-opposite values of the initial population to eliminate solutions that deviate from the optimum result at the initial stage. This leads to improved computational speed and a lower likelihood of getting stuck in local optima. The modified QOBL algorithm involves several steps, as shown below.

4.3.1 Origination of opposite-point

If \(X(X_{1} ,X_{2} ,X_{3} ,.......X_{n} )\) is a point in n-dimensional exploration space such as \((X_{1} ,X_{2} ,X_{3} ,.......X_{n} ) \in R\). Then, for this point, the search space’s center is written as:

$$ M_{j} = \frac{{Ub_{j} + Lb_{j} }}{2}. $$
(11)

The origination of the opposite point Xo is now written as:

$$ X_{Oj} = \left[ {Lb_{j} + Ub_{j} - X_{j} } \right]_{1 \times n} ,\quad {\text{where}}\, j = {1},{2},{3}, \ldots \ldots .n. $$
(12)

where \(Lb_{j}\) and \(Ub_{j}\) are lower and upper bound.

4.3.2 Generation of quasi opposite point

The quasi-opposite point \(X_{qo}\) for n-dimensional search is expressed by:

$$ X_{qoj} = rand(M_{j} ,X_{oj} ),\quad {\text{where }}j = {1},{2},{3}, \ldots \ldots n. $$
(13)

4.3.3 Updating of Xj

By linking opposite and quasi-opposition principles into a particular vector, the search space is updated:

$$ X_{j} = \left[ {X_{oj} ;X_{qoj} } \right]_{1 \times 2n} . $$
(14)

4.3.4 The pseudocode for computing the quasi-opposition point

\(M_{j} = \frac{{Ub_{j} + Lb_{j} }}{2}\) (Generate centre of search space from n dimension vector \((X_{1} ,X_{2} ,X_{3} ,.......X_{n} )\))

If \((X_{oj} \prec M_{j} )\) (Check if production of the opposing point \(X_{Oj}\) is less than the centre of search space)\(X_{qoj} = M_{j} + (X_{oj} - M_{j} ) \times r_{1}\) % \(r_{1}\)\(\in\)[0,1] (Generate Quasi-opposite point \(X_{qoj}\) if the above condition certifies)

else

\(X_{qoj} = X_{oj} + (M_{j} - X_{oj} ) \times r_{1}\) (If the above condition is not certified then generate a Quasi-opposite point \(X_{qoj}\))

end if-else

4.4 Justification of modified algorithm

In this section, the suggested controller gain parameters are optimized using several optimization procedures like PSO, Artificial Bee Colony (ABC), Wild Horse Optimizer (WHO), Volleyball Premier League (VPL), AOA, and the proposed QOAOA. The initial population and number of iterations are set at 30 and 10, correspondingly. The objective function is to reduce ISE for the three-area systems under consideration. The convergence characteristics of each algorithm are evaluated by calculating the FOD shown in Fig. 6. Table 2 presents the best optimal values of ISE obtained from each optimization technique over a simulation time of 100 s. The outcomes indicate that the suggested QOAOA algorithm converges faster than the other algorithms after 10 iterations, as shown in Fig. 6. Furthermore, the dynamic response of the three-area scheme under diverse algorithms is presented in Fig. 7, demonstrating that QOAOA outperforms other algorithms in terms of disturbance resolution and has a lower maximum deviation.

Fig. 6
figure 6

FOD of different optimization techniques

Table 2 Best optimal values of integral square error
Fig. 7
figure 7

Frequency regulation and tie-line power response of different algorithms

4.5 Justification of proposed controller

To estimate the effectiveness of the suggested controller, a comparative analysis was conducted using MATLAB and SIMULINK Toolbox 2021 b on a connected system that does not have a DG and hybrid energy storage system. Nonlinearities were incorporated to enhance the practicality and realism of the PS. The GRC and GDB values were set at 3% per minute and 0.0006 pu, respectively. The QOAOA method was utilized to optimize the PID, FOPIDN, 1 + TD, and proposed cascade FOPIDN-(1 + TD) controllers for optimal tuning. The controller gain settings were limited to a range of 0–10 for integer gains (\(k_{p}\), \(k_{i}\), \(k_{d}\), \(K_{t}\), and \(K_{D}\)), 0–1 for FO integrator (\(a\)) and tilt parameter (\(b\)), and 0–400 for the derivative filter (\(N_{1}\) and \(N_{2}\)). The dynamic responses of these controllers were then compared for the three area system.

4.5.1 System dynamic response for step load perturbation

The response of the system is analyzed when subjected to step-load perturbations of 1% or 0.01pu. Figure 8 depicts the system’s dynamic behavior as determined by the various controllers, i.e., PID, FOPIDN, 1 + TD, and the proposed controller for step load perturbation. From the frequency response curve PID and (1 + TD) have oscillatory response, which is not acceptable. In comparison to PID and (1 + TD) controllers, the FOPIDN controller’s response is more consistent. However, when compared to the proposed controller, it shows increased undershoot and settling time. Table 3 illustrates that the suggested controller surpasses the above-mentioned controller measured by transient and steady-state dynamics.

Fig. 8
figure 8

Comparative dynamic performance of different controllers for 0.1 p.u SLD in Area 1

Table 3 Comparative of the dynamic performance of various controller schemes for step load

Table 4 shows the optimal tuned value of controller parameters obtained using QOAOA for different controllers. The improvement in the performance of the proposed controller is indicated by maximum variation in frequency (\(\Delta f_{1}\), \(\Delta f_{2}\), \(\Delta f_{3}\)) and power change on the tie-lines (\(\Delta P_{{{\text{tie}}12}}\), \(\Delta P_{{{\text{tie}}13}}\)) over FOPIDN controller is (99.13%, 98.33%, 98.44%) and (81.48%, 78.26%).

Table 4 Best gain parameters of different controllers acquired by the QOAOA algorithm for step load

4.5.2 System dynamic response for random load perturbation

The response of the system is examined with random load perturbations indicated in Fig. 9a. Figure 10 depicts the system’s dynamic behavior as determined by the various controllers, i.e., PID, FOPIDN, 1 + TD, and the proposed controller for random load perturbation. The frequency response curve depicted in Fig. 10 demonstrates that the suggested controller surpasses the performance of the other tested control methods for the investigated system. Table 5 illustrates that the suggested controller surpasses the above-mentioned controller measured by transient and steady-state dynamics. For unsystematic load the improvement in the performance of the proposed controller indicated by maximum variation in frequency (\(\Delta f_{1}\), \(\Delta f_{2}\), \(\Delta f_{3}\)) and power change on the tie-lines (\(\Delta P_{{{\text{tie}}12}}\), \(\Delta P_{{{\text{tie}}13}}\)) over FOPIDN controller is (98.65%, 48.78%, 52.52%) and (75.0%, 76.1%).

Fig. 9
figure 9

Relative dynamic performance of different controllers for random load in Area 1

Fig. 10
figure 10

Profile of a random load pattern, b random SPV output, and c random WT output

Table 5 Comparative of the dynamic performance of various controller schemes for random load

5 Simulation and experimental results

The analyzed three-area test system is simulated using MATLAB/Simulink toolbox 2021 b. Non-linearity is taken into account, for example GDB, GRC, and BD. The effect of RFB, Super-capacitor, and HESS on the dynamic behavior of the studied power system under intermittent solar irradiation and wind turbine power is thoroughly studied. A comparative analysis is also carried out between the simulation results obtained from the proposed controller and other previously published works on the same system. To ensure the practical implementation and efficacy of the suggested controller, a real-time hardware-in-the-loop (HIL) simulation is conducted using the OPAL-RT OP4510. The proposed system is illustrated in Fig. 1.

5.1 Effect of hybrid energy storage system with constant and variable DG

The overall frequency response structure of the PS under investigation, as depicted in Fig. 1, is used to evaluate the dynamic performance of the system in the presence of different energy storage systems. A step load perturbation of 0.01 pu is introduced to the Area 1 of the system. The effectiveness of the proposed hybrid energy storage system (RFB + SC) is compared to other energy storage systems, including RFB, SC, and without energy storage systems.

5.1.1 Effect of HESS with constant DG having 1% step load disturbances in Area 1

In the test system, RESs such as SPV and WT are inserted in each area under constant weather conditions. In this context, the power input for solar photovoltaic and wind turbines is considered as a 0.001 pu which remains constant throughout the simulation. HESS increases transient performance by delivering stored energy at peak demand and drawing energy from the grid during light load. The dynamic response of the proposed PS with different ESS under the variation of DG is illustrated in Fig. 11. The system’s dynamic response demonstrates that the use of HESS results in a quicker reduction of undershoot and settling time compared to other energy storage systems. The maximum deviation in frequency regulation and tie-line power is presented in Table 6, indicating that the HESS configuration results in a lower deviation in both parameters. Furthermore, as illustrated in Fig. 11, the response is stable and free of oscillations. The improvement in performance indicated by maximum variation in frequency (\(\Delta f_{1}\), \(\Delta f_{2}\), \(\Delta f_{3}\)) and power change on the tie-lines (\(\Delta P_{{{\text{tie}}23}}\)) by incorporating HESS in the system is (45.21%, 33.33%, 29.59%) and 46.95% over WES. The proposed controller gain parameter when HESS is connected to the test system is given in Table 7.

Fig. 11
figure 11

Influence of HESS on system dynamic response with constant DG

Table 6 Percentage improvement of system with HESS
Table 7 Optimal gain settings of the suggested controller with HESS

5.1.2 Effect of HESS with variable DG having 1% step load disturbances in Area 1

The test system includes SPV and WT as renewable energy sources in each area under varying weather conditions. Initially, the power input to the SPV is set at 0.0025 pu from 0 to 40 s, 0.0035 pu from 40 to 60 s, 0.001 pu between 60 to 80 s, and 0.002 pu after 80 s. Similarly, the power input to the WT is set at 0.002 pu from 0 to 20 s, 0.003 pu from 20 to 60 s, 0.004 between 60 to 80 s, and 0.001 after 80 s. As a result, solar photovoltaic and wind turbines generate random input power to the scheme as presented in Fig. 9.

The system's dynamic response to variable DG and different ESS is depicted in Fig. 12, and the highest deviation is reported in Table 6. It can be shown that connecting both HESS and DG to the system reduces the maximum overshoot in frequency-deviation and the maximum undershoot in inter-line power. The system response is smooth and oscillation-free. The improvement in performance indicated by maximum variation in frequency (\(\Delta f_{1}\), \(\Delta f_{2}\), \(\Delta f_{3}\)) and power change on the tie-lines (\(\Delta P_{{{\text{tie}}23}}\)) by incorporating HESS in the system is (40.86%, 34%, 41.53%) and 45.922% over WES. The optimal proposed controller gain parameters with HESS are shown in Table 7.

Fig. 12
figure 12

Impact of integration of HESS with variable DG on system dynamic response

5.2 Result comparison with previous published work

The efficiency of the newly proposed QOAOA-optimized cascade FOPIDN-(1 + TD) controller was evaluated and compared to the works of Parmar et al. (2014) and Shankar et al. (2016) on the same atmosphere analyzed in the latter study. Figure 13 results show that the suggested controller demonstrated a superior transient response. Additionally, Table 8 indicates an advancement in the highest deviation in Zone 1 frequency, Zone 2 frequency, and tie-line power of 9.822%, 23.425%, and 13.90%, respectively, in comparison to the findings of Shankar et al. (2016).

Fig. 13
figure 13

Comparison of the dynamic response of the suggested controller scheme with published literature

Table 8 Comparison of maximum deviation of the proposed controller with published literature

5.3 Experimental test analysis

To verify the suggested controller’s practicality and performance, a real-time analysis was carried out using the OPAL-RT OP4510. The methodical technique for the implementation of the HIL-based OPAL-RT experiments is represented in Fig. 14. The hardware-in-the-loop (HIL) simulator is primarily required to achieve the suggested controller’s accuracy and robustness in a real investigated system.

Fig. 14
figure 14

Flowchart for implementation of real-time MATLAB simulation utilizing OPAL-RT

The RTS operates as a parallel computation, enabling both extremely accurate and affordable real-time execution of complicated simulation models. To replicate glitches and delays that may not be evident in conventional offline simulations, an OPAL-RT-based RTS was employed (Mishra et al. 2022). The experimental configuration of the HIL setup, as shown in Fig. 15, comprises three components: (1) an OPAL-RT RTS that models the system under study, (2) a host PC that generates the Matlab/Simulink model-based code performed on the OPAL-RT, and (3) a Mixed Signal Oscilloscope (200 MHz) used to visualize different waveforms.

Fig. 15
figure 15

Real-time experimental setup

5.3.1 Effect of hybrid energy storage on system performance using OPAL-RT hardware setup

Figure 15 demonstrates the utilization of OPAL-RT OP4510 for conducting a real-time investigation to verify the performance and practical implementation of the proposed controller. To ensure the accuracy and robustness of the suggested controller in a real system, a hardware-in-the-loop (HIL) simulator is utilized. The proposed system is sampled at a rate of 50 microseconds during the real-time study. Figure 16a and b depicts the real-time dynamic frequency and tie-line response of the system under investigation for case 5.1.1, with and without hybrid energy storage, respectively. Additionally, Fig. 16 illustrates a comparison of the dynamic response of the system with and without a hybrid energy storage system. The results obtained from OPAL-RT demonstrate that hybrid energy storage significantly reduces undershoot and settling time compared to the system without hybrid energy storage. Moreover, Fig. 16 illustrates that the response is smooth and free of oscillations.

Fig. 16
figure 16

HIL OPAL-RT dynamic response of the investigated scheme in Zone 1, Zone 2, and Zone 3: a without hybrid energy storage, b with hybrid energy storage

5.3.2 Experimental validation

This section compares the dynamic response of the suggested controller in the investigated system using both MATLAB simulation and OPAL-RT-based real-time simulator (RTS) (Ramoji and Saikia 2021). The proposed three-area system with constant distributed generation and hybrid energy storage system was first developed in MATLAB/Simulink R2021b platform for a simulation time of 100 s. The proposed system is then validated via RTLab software integrated with MATLAB, with a sampling time of 50 microseconds during the real-time study. Figure 17 depicts the dynamic response of the system obtained from both MATLAB Simulation and OPAL-RT-based RTS. Analysis of the results reveals that the responses from both platforms are almost identical, thus validating the practical viability of the proposed system using the suggested control strategy.

Fig. 17
figure 17

Dynamic response assessment of OPAL-RT and Matlab/Simulink

6 Conclusion and future direction

This work developed a novel intelligence-based cascade FOPIDN-(1 + TD) controller to address the difficulties of AGC design in smart grids with nonlinearities. A modified quasi-opposition-based arithmetic optimization (QOAOA) algorithm is used to optimize the controller gain. The time domain simulation result leads us to claim that, in comparison to other common controllers, the recommended controller regulates the smart grid frequency quite effectively over-tested controllers. Simultaneously, the enhanced computational speed and error-reducing abilities of QOAOA have been demonstrated excellently in comparison to numerous popular algorithms. To bring a practical approach to RES integration in automatic generation control the investigated system is evaluated with the influence of solar and wind under uncertain weather conditions. The improved dynamic performance of the scheme demonstrates that the suggested controller is considered so that it can minimize the influence of power uncertainty of load disturbances and renewable energies. The analysis has been further carried out to check system performance when hybrid energy storage devices containing super-capacitors and redox flow battery (RFB) are integrated into the proposed three-area system considering the varying nature of wind speed and solar irradiation. The occurrence of HESS in the control areas produces inspiring outcomes and further improves the power system performance. A step ahead, the performance of the suggested controller’s dynamic response is compared to that of previously published literature on the same platform. Finally, utilizing a hardware-in-the-loop (HIL)-based OPAL-RT simulator, the practicability and effectiveness of the suggested controller are evaluated on a real-time platform.

This study is dedicated to tackling the specific challenges encountered in the realm of smart grid systems. In future research, further investigation can be carried out in coordination with a hybrid energy storage system considering the effects of cyber-attacks on frequency regulation services of smart grid power. In the future, the proposed control scheme will be enhanced with some advanced knowledge-based controllers using fuzzy logic. To improve frequency and tie-line power, the exploration and exploitation characteristics of the QOAOA algorithm may be further enhanced by the hybridization process with several advanced techniques.