1 Introduction

Integration of Renewable Energy Sources (RES) is important from both environmental perspective and prospective scarcity of fossil fuel to be witnessed in near future, for running the traditional thermal power plants (Weitemeyer et al. 2015; Owusu and Asumadu-Sarkodie 2016; Swain et al. 2017). The RES can strengthen the Power Grid through local generation and Micro grids which lessens the burden of power transmission, minimises subsequent losses and investment involved (Alsaif 2017). The incorporation of RES, however has certain drawbacks like intermittency of generation, degradation of power quality, reduction of storage energy of the grid and high investment associated with these projects (Dilshad et al. 2020; Hossain et al. 2018). The advancement of electronics and cyber physical sensor technology has already however opened a lot of scope of improving the existing power grid, fostering it to be compatible with RES (Butta et al. 2021). The new Smart Power Grid is now utilizing these techniques like advanced metering infrastructure, Real Time digital Monitoring and control, Demand Response, data analytics, high speed communication between the power market players, Phasor Measurement Units (PMUs)etc. (Nafi et al. 2016; Touzene et al. 2019). Demand side management through proper implementation of data analytics and demand response is one of these potential areas which can ensure optimisation of the hindrances of integration of RES in the power grid (Nigam et al. 2019; Huang et al. 2019). Use of demand response is of utmost importance since the RES is highly intermittent in nature and efficient and optimal demand management is necessary for maintaining the supply demand balance and the same also need to be implemented at optimal price of electricity (Tahir et al. 2019). In smart power market, thus the generation of generators and load demands of Load Dispatch Centres/ consumers are scheduled with the availability of generator cost characteristics and consumer cost benefit function data, a typical demand response (Yang et al. 2015; Arias et al. 2018), so that RES can be incorporated in the grid. The demand response programs are objected towards delivering maximum load as per willingness to pay of the LDCs/ Consumers and to provide incentives to the consumers who volunteers for load curtailment through demand response (Yu et al. 2018). Implementation of demand response is not free from challenges. Since it is a new concept, there is lack of knowledge and a lot of assumptions are to be made for its modelling and subsequent design of its objective and algorithm (Thoelen 2019). Absence of a perfect framework for all the conceptualized demand response is also an obstacle of its implementation. More over effects of demand response on vital parameters of the power grid also need to be studied and ways of mitigation of undue changes need to be devised. In a works presented in Chai et al. 2019a; Narimani et al. 2015; Bajool et al. 2017; Gao et al. 2020; Stawskaac et al. 2021), it has been proven that the implementation of demand response can influence the voltage stability, maximum line flow that is line congestion and distribution and transmission line active power loss. These difficulties arise from the fact that demand response characteristics of the LDCs/ Consumers are traditionally price based and its social or grid level impact is not estimated and compensated. The Independent system Operator (ISO) however has to ensure no limit violation as the same may be detrimental to Transmission Companies (TRNSCOs) and in long run the same may affect the operation of Distribution Companies (DISCOs) and Generation Companies (GENCOs). Hence ISO must have the regulatory monopoly to allow the techniques like innovative demand response algorithm in the optimisation program up tothe extent that it ensures benefit of all the power market players for Social Welfare (Balamurugana et al. 2015; Chanda and De 2014; Sen et al. 2015). This work presents such an algorithm and a supporting demand response framework to schedule both generators and LDCs/Consumers to optimise the price of electricity and operating condition of power system networks.

1.1 Literature survey

Some recent contributions in the same field (Shigenobu et al. 2017; Chai et al. 2019b; Dong et al. 2018; Jabir et al. 2018) was found in incentive-based approaches, where the customer / LDC was given incentives for choosing its own demand response for sustenance of power grid system parameters within specified limit contributing to Social Welfare. The work presented in Hirotaka et al. (2018) shows a synergism of social Welfare optimisation and incentive-based demand response. But in the work, the incentive was only calculated as per customer’s comfort not on deviation of system parameters. Works reported in Long et al. (2019) concerning system parameter optimisation for social welfare but the same used FACTs devices support. The work in Nainar et al. (2021) addresses the issue and resolves the same by Battery Energy storage. The work presented in Mosaddegh et al. (2018); Viet et al. 2018) depicts the concern of limit violation and how smart loads can help in producing optimal solution to system operation. Through the review of these works, it has been found that an appropriate framework for data analytics of Social Welfare optimisation only has not been developed yet (Zhang et al. 2018) which can assess each of the GENCOs and DISCOs/LDC/consumers as per their ability to deliver or modulate consumption respectively for reaching a particular solution or schedule of the state variables of the grid, at minimal price and with no limit violation and which will provide incentives to the market players in terms of maximising the dispatch of power and appropriate nodal pricing of each of the LDCs/Consumers for keeping system vital parameters within limit. The Independent system Operator (ISO) is generally responsible for this Social Welfare analytics while the Market Players (GENCOs, TRANSCOs and DISCOs) are responsible for Individual Welfare Analytics (Munshiab and Mohameda October 2017; Lucas Melo et al. 2019; Palmintier et al. 2017). All the market players are thus, in this scenario, are presumes of data (Both producer and consumer). ISO however as per the bids and characteristics received from the participants executes an optimisation programme which ensures maximum possible dispatch of power at minimum price and cost of generation (Alotaibi et al. 2020). For the hour ahead market, in presence of RES it is important that the market players not only maximise their own benefit but also, they should contribute to the benefit of all the participants that is to cause Social Welfare and the same is possible only when the system parameters are also given the same importance in scheduling and load management (Mahanty and Singh 2018). Table 1 presents a summary of the related works in this field and the scope of developing the proposed Social Welfare optimisation algorithm.

Table 1 Summary of previous works justifying the need of the proposed Framework and Algorithm

In pursuit of developing such an algorithm, this work enunciates in Sect. 2 about the present framework of the power market and the proposed framework of security constrained social Welfare. In Sect. 3, the mathematical model of the proposed framework has been presented in contrast to the traditional Security Constrained Optimal Power Flow (SCOPF) (Mohammadi et al. 2018). The case studies in IEEE 30 bus system, considering worst possible contingencies and loading scenarios has been presented in Sect. 4, using the proposed DEQPSO (Liu et al. 2019) based Security Constrained Social Welfare Optimisation algorithm and its performance has been presented against traditional SCOPF. The major contributions of this research work can be summarised as follows:

  1. i.

    A novel Framework for Security Constrained Social Welfare maximisation has been proposed in the paper, introducing various penalties and incentives from price responsive demand response for small deviation of operating point to encourage LDCs/Consumers to shift their demands from peak to lean periods.

  2. ii.

    A novel method of determination of minimum load curtailment has been proposed for the LDCs based on generation uncertainty imposed by RES and the prevailing price of electricity decided by the DR of consumers.

  3. iii.

    A novel method for maintaining required level of system inertia has been proposed through additional incentive to the consumers to maintain optimal dependence on grid power and RES.

2 Present and proposed social welfare optimization framework

This section presents the prevailing framework of social welfare optimisation and develops the proposed framework and its mathematical model.

2.1 Present frame work of social welfare optimisation in smart grid

The present framework for Social Welfare maximisation is presented in Hwang et al. (2018) as shown below in Fig. 1. Here the distribution Energy management (DEM) section optimises the benefit of both GENCOs and DISCOs or End Users (EU) for optimal pricing.

Fig. 1
figure 1

Present framework of Social Welfare Optimisation with demand response

All the actions however in this framework is by virtue of price-based demand and generation response of the grid and the consideration of limit violation and security issue is only to the extent of willingness to pay of the LDCs. Thus, this framework is to be modified for the proposed Security Constrained Social Welfare Optimisation to ensure required level of security margin.

2.2 The proposed social welfare optimisation framework for smart grids

The Fig. 2 shows the proposed framework and its mathematical model. The proposed optimisation model of security constrained social welfare optimisation is crucial in presence of RES. The ISO in this model is the central data analytics authority and the market players like GENCOs, DISCOs submit their cost, consumption hour ahead characterises to ISO. In this dynamic market, ISO in this optimisation, as shown in fig., can consider the security constraints of social welfare.

Fig. 2
figure 2

Optimisation model of proposed security constrained social welfare optimisation

3 Mathematical modelling of the proposed framework

In this section the mathematical model of the proposed Security Constrained Social Welfare optimisation has been developed compared to SCOPF model. This section also depicts some novel methods for estimation of penalties and incentives for transmission line Congestion, uncertainty of Generation due to RES, increase in power loss of the network and poor power factor. The incentives and penalties formulated in this section directly relates the consumers with the Social Welfare optimisation problem.

3.1 The traditional security constrained optimal power flow (SCOPF) in smart grid

The fundamental objective of any optimal power flow program is to minimize the total generation cost as shown in Eq. 1.

$${\text{Minimize }}f\left( {C_{i} } \right) = \mathop \sum \limits_{n = 1}^{{n_{g} }} C_{n} \quad {\text{Rs}}{\rm{./hour}},$$
(1)

where \({C}_{n}={a}_{n}{P}_{n}^{2}+{b}_{n}{P}_{n}+{c}_{n}\)

In Smart Grid along with minimizing the generation cost, the operational standard in terms of reliability, efficiency and quality of power has to be maintained. Combing all the constraints, the objective function for security constrained optimal power flow (SCOPF) is given by

$${\text{Minimize}}\;f(C_{n} , P_{{{\text{lmaxj}}}} T_{Lmaxj} ,V_{min} ) = \mathop \sum \limits_{n = 1}^{{n_{g} }} C_{n} + \mathop \sum \limits_{ij = 1}^{nlns} P_{lmaxij} .p_{1ij} + \mathop \sum \limits_{ij = 1}^{nlns} T_{Lmaxij} .p_{2ij} + V_{min} .p_{3} + \left( {\mathop \sum \limits_{n = 1}^{ng} P_{gn} - P_{D} - T_{L} } \right).p_{4 }$$
(2)

3.2 Limitations of SCOPF in representing constraints of optimization

In SCOPF only the maximum line flow is considered for the penalty to be applied to the objective function. This makes all the line flows below the maximum limit but makes the other lines underutilized. Similar in case of Transmission line loss, in SCOPF, a single limit of transmission line loss is traditionally assumed. Thus, the scope of SCOPF becomes confined and fails to motivate the solution with optimal demand side management. Hence, general expression for the limits also to be estimated.

Considering lumped circuit model Fig. 3, the power loss in transmission line is given by (Eq. 3)

Fig. 3
figure 3

Lumped circuit model

$${T}_{Lij}={g}_{ij}\left({V}_{i}^{2}+{V}_{j}^{2}\right)-2{V}_{i}{V}_{j}Cos{\delta }_{ij}$$
(3)

If Eq. 3 is differentiated with a change in \({V}_{i}\) (Eq. 4)

$$\frac{d{T}_{Lij}}{d{V}_{i}}=2{V}_{i}{g}_{ij}-2{V}_{j}Cos{\delta }_{ij}$$
(4)

For minimum transmission line active power loss in line i–j (Eq. 5)

$${V}_{i}=\frac{{V}_{j}Cos{\delta }_{ij}}{{g}_{ij}}$$
(5)

Substituting the value of \({V}_{i}\), the minimum possible active power loss in transmission line i–j (Eq. 6)

$$T_{Lijmin} = V_{j}^{2} \left( {g_{ij} - \frac{{Cos^{2} \delta_{ij} }}{{g_{ij} }}} \right) = g_{ij} \left( {V_{j}^{2} - V_{i}^{2} } \right)$$
(6)

Considering Fig. 3 the expression for line flow is given by (Eq. 7)

$${P}_{ij}={V}_{i}{V}_{j}{Y}_{ij}Cos\left({\theta }_{ij}+{\delta }_{ij}\right)-{V}_{i}^{2}{Y}_{ij}Cos{\theta }_{ij}$$
(7)

Differentiating the equation with respect to \({V}_{i}\) (Eq. 8)

$$\frac{d{P}_{ij}}{d{V}_{i}}={V}_{j}{Y}_{ij}Cos\left({\theta }_{ij}+{\delta }_{ij}\right)-2{V}_{i}{Y}_{ij}Cos{\theta }_{ij}$$
(8)

For maximum line flow as shown below (Eq. 9)

$${V}_{i}=\frac{{V}_{j}Cos\left({\theta }_{ij}+{\delta }_{ij}\right)}{2Cos{\theta }_{ij}}$$
(9)

Substituting the value of \({V}_{i}\) (Eqs. 10, 11 and 12)

$${P}_{ij}=\frac{{V}_{j}^{2}{Cos}^{2}\left({\theta }_{ij}+{\delta }_{ij}\right){Y}_{ij}}{4Cos{\theta }_{ij}}$$
(10)
$$=\frac{{V}_{i}{V}_{j}Cos\left({\theta }_{ij}+{\delta }_{ij}\right){Y}_{ij}}{2}$$
(11)
$$={P}_{lmaxij}Cos\left({\theta }_{ij}+{\delta }_{ij}\right)$$
(12)

where

$$P_{lmaxij} = \frac{{V_{i} V_{j} Y_{ij} }}{2}$$
(13)

This expression can be treated as maximum line flow limit for general conditions (irrespective of no. of buses and lines). This formula can be however approximated as Surge Impedance loading (SIL) of the line as \({V}_{i}\), \({V}_{j}\) are the maximum values and if the rms values of the bus voltages \({V}_{i}{,V}_{j}\) should be assumed as 1 pu. Hence the line flow limit for testing congestion may be assumed as (Eq. 14)

$$P_{lmaxij} = Y_{ij} \,{\text{Mw}}$$
(14)

3.3 Augmenting demand side management in the proposed optimization framework

In the model expressed in Eq. 1, in traditional SCOPF, the distribution companies were participating with a fixed forecasted demand. This mode of participation can however be changed if LDCs can participate with a flexible demand which are depicted (Huang et al. 2019). In the present work such a demand response model is proposed and implemented where as per the price of electricity in every hour, the LDCs alter their consumption for staying in the market with their willingness to pay characteristics or price responsive characteristics.

In this market, electricity price is discovered by the intercepting point of aggregated generation cost curve and individual cost benefit function curve of each LDC or consumer. As willingness to pay of different customers are different, the load curtailment and the price at different node or LDC will be different. (Fig. 4).

Fig. 4
figure 4

Nodal pricing with price elasticity of demand

3.3.1 Incentive on line utilization through DR

In demand response scenario, it is expected that all the transmission lines will be used to their fullest capacity without exceeding the maximum allowable line flow limit. For the measurement of this ability for each and every load bus the transmission line utilization factor may be proposed as (Eq. 15)

$${\text{Line utilizationfactor }}({\rm{LUF}}) = \frac{{P_{ij} }}{{P_{lmaxij} }}$$
(15)

The maximum value of this utilization factor is obviously unity. For offering this incentive, nodal pricing should be influenced in the power system by LUF. The buses which are responsible for the improvement of the factor should be eligible for this incentive.

Considering Fig. 5 The active power flow in transmission line i–j is (Eq. 16), Eqs. 17 and 18 can be obtained

$$P_{lij} = P_{gi} - P_{di} - P_{gj} + P_{dj}$$
(16)
$$\frac{{P_{lij} }}{{P_{lmaxij} }} = \frac{{P_{gi} - P_{di} - P_{gj} + P_{dj} }}{{P_{lmaxij} }}$$
(17)
Fig. 5
figure 5

Elementary power system

$$\frac{d\frac{{P}_{lij}}{{P}_{lmaxij}}}{d{P}_{di}}=-\frac{1}{{P}_{lmaxij}}$$
(18)

Considering \({P}_{dj}\) constant. As the effect of demand response has been considered, \({P}_{gi}\) and \({P}_{gj}\) have been assumed to be constants.

Now, Considering \({P}_{di}\) and its characteristics (Fig. 6) with the price of electricity. The base case refers to an operating point where there was balance between generations and demand and \({P}_{di}\) was constant. Considering small change in demand a tangent can be drawn about the operating point and from the slope of the tangent \(\frac{\Delta {P}_{ri}}{\Delta {P}_{di}}\) can be calculated.

Fig. 6
figure 6

Small variation in demand characteristics

Considering quadratic price responsive demand, as per willingness to pay, the price responsive demand characteristics of LDCs / Consumers, can be assumed as (Eq. 19)

$${P}_{ri}={\alpha }_{i}{P}_{di}^{2}+{\beta }_{i}{P}_{di}+{\delta }_{i}$$
(19)

where \({\alpha }_{i},{\beta }_{i}\), \({\delta }_{i}\) are the coefficients of consumer cost benefit function as shown in the Fig. 8 for the LDC or consumers of ith Bus. For a small variation of the demand the slope of the quadratic curve can be found as (Eq. 20)

$$\frac{{\mathrm{dP}}_{\mathrm{r}}}{{\mathrm{dP}}_{\mathrm{di}}}=2{\mathrm{\alpha }}_{\mathrm{i}}{\mathrm{P}}_{\mathrm{di}}+{\upbeta }_{\mathrm{i}}$$
(20)

Penalty for change in price due to change in line utilization factor

$$\left( {P_{LUF} } \right) = \frac{{dp_{ri} }}{dLUF}$$
(21)
$$=\frac{d{P}_{r}}{d{P}_{di}}.\frac{d{P}_{di}}{dLUF}$$
$$=(2{\alpha }_{i}{P}_{di}+{\beta }_{i})(-\frac{1}{{P}_{lmaxij}})$$
(22)

Thus, the DR program proposed, in every iteration will be calculating the \({P}_{LUF}\) using equation no.21 to 22 and will appropriately move the solution towards the best solution where the consumers / LDCs that are more responsible for line utilization through DR will be provided incentives by reduced nodal pricing.

3.3.2 Penalty imposed to minimise load curtailment

The weather dependant unpredictable generation capacity of RES may create immense difficulty in balancing power generation with demand. The uncertainty of generation can be measured by generation surplus of the upcoming hour. The generation surplus for the upcoming hour may be given as (Eq. 23).

$$P_{gsurplus} = \mathop \sum \limits_{i = 1}^{ng} P_{gimax} - \mathop \sum \limits_{i = 1}^{ng} P_{gi}$$
(23)

The factor of generation uncertainty can be represented as Generation Uncertainty factor (Eq. 24)

$$(\mathrm{GUF})=\frac{\sum_{i=1}^{ng}{P}_{gimax}-\sum_{i=1}^{ng}{P}_{gi}}{\sum_{i=1}^{ng}{P}_{gimax}-\sum_{i=1}^{ng}{P}_{gimin}}$$
(24)

The proposal for load curtailment in each LDC/customer in this work is that when the generation uncertainty factor is zero, the load curtailment should be minimum that is the dispatch should be \({P}_{dimax}\) and when the GUF is 1 then the curtailment can be maximum (\({P}_{dimax}\)\({P}_{dimin})\) that is where \({P}_{dimax}\) is the maximum value of dispatch requested by the consumer in ith bus when the price of electricity is appropriately low and \({P}_{dimin}\) is the sum of minimum dispatch. Thus, the expression for allowable curtailment in the ith bus can be written as (Eq. 25)

$$P_{ci} = \left( {P_{{{\text{dimax}}}} {-}P_{{{\rm{dimin}}}} } \right) \cdot {\text{GUF}}$$
(25)

Thus, the higher of the GUF, the higher is the curtailment. During intermittency of generation, the maximum value of generation of RES reduces and subsequently the GUF increases. Thus, this load curtailment strategy ensures minimum despatch for ith bus if the value of GUF remains between 0 and 1. If however, the value increases above unity due to uncertain generation the curtailment may increase accordingly.

The change of allowable curtailment \({P}_{ci}\) in the ith bus with respect to \({P}_{di}\) is given by (Eq. 2628)

$$\frac{{dP_{ci} }}{{dP_{di} }} = \frac{d}{{dP_{di} }}\left( {\left( {P_{dimax} {-}P_{dimin} } \right) \cdot \frac{{\mathop \sum \nolimits_{i = 1}^{ng} P_{gimax} - \mathop \sum \nolimits_{i = 1}^{ng} P_{gi} }}{{\mathop \sum \nolimits_{i = 1}^{ng} P_{gimax} - \mathop \sum \nolimits_{i = 1}^{ng} P_{gimin} }}} \right)$$
(26)
$$= \frac{d}{{dP_{di} }}\left( {\left( {P_{dimax} {-}P_{dimin} } \right) \cdot \frac{{\mathop \sum \nolimits_{i = 1}^{ng} P_{gimax} - \mathop \sum \nolimits_{i = 1}^{ng} P_{di} - T_{L} }}{{\mathop \sum \nolimits_{i = 1}^{ng} P_{gimax} - \mathop \sum \nolimits_{i = 1}^{ng} P_{gimin} }}} \right)$$
(27)
$$= \left( {P_{dimax} {-}P_{dimin} } \right)\left( { - \frac{1}{{\mathop \sum \nolimits_{i = 1}^{ng} P_{gimax} - \mathop \sum \nolimits_{i = 1}^{ng} P_{gimin} }}} \right)$$
(28)

It has been assumed under small variation change of active power transmission line loss is TL with respect to small change in \({T}_{L}\) is negligible.

Considering small variation of demand from the operating point, the change in charge for demand with respect to the change in load curtailment can be expressed as (\({p}_{5})\) (Eq. 29)

$$\frac{{dP_{r} }}{{dP_{ci} }} = \left( {2\alpha_{i} P_{di} + \beta_{i} } \right) \cdot \left( {P_{dimax} {-}P_{dimin} } \right)\left( { - \frac{1}{{\mathop \sum \nolimits_{i = 1}^{ng} P_{gimax} - \mathop \sum \nolimits_{i = 1}^{ng} P_{gimin} }}} \right)$$
(29)

This penalty in every iteration may be applied to the curtailment of ith bus in order to guide the solutions towards optimality.

3.3.3 Incentive for maintaining bus power factor

The bus power factor may get affected by the changes in demand as suggested in optimal solution. This may affect the bus voltage, and in turn may increase the losses and regulation of bus voltage may be lost. The power factor for a particular ith (Fig. 7) bus can be represented as (Eq. 30)

Fig. 7
figure 7

Elementary bus power flow

$${PF}_{i}=\frac{{P}_{gi}-{P}_{di}}{\sqrt{({P}_{gi}-{P}_{di}{)}^{2}+({Q}_{gi}-{Q}_{di}{)}^{2}}}$$
(30)

Change in the value of \({PF}_{i}\) with respect to change in \({P}_{di}\) can be expressed as (Eq. 31)

$$\frac{d{PF}_{i}}{d{P}_{di}}=\frac{{P}_{gi}-{P}_{di}}{\sqrt[3/2]{({P}_{gi}-{P}_{di}{)}^{2}+({Q}_{gi}-{Q}_{di}{)}^{2}}}$$
(31)

The penalty for change in power factor can be expressed as Eq. 32 (\({p}_{6})\)

$$\frac{d{P}_{r}}{d{PF}_{i}}=\left(2{\alpha }_{i}{P}_{di}+{\beta }_{i}\right).\frac{\sqrt[3/2]{({P}_{gi}-{P}_{di}{)}^{2}+({Q}_{gi}-{Q}_{di}{)}^{2}}}{{P}_{gi}-{P}_{di}}$$
(32)

This penalty will be reflecting for the consumers / LDCs with reduced nodal pricing and will motivate them to keep a good power factor in order to get lower nodal priceas they manage demand.

3.3.4 Incentive for improving line stability

To secure a stable operation of power system, the line stability index should be maintained below unity. The Eq. 33 shows the relation between line stability index and power system parameters.

$${LQ}_{p}=4\left(\frac{x}{{V}_{i}^{2}}\right)\left(\frac{x}{{V}_{i}^{2}}.{P}_{i}^{2}+{Q}_{j}\right)$$
(33)

where the line reactance of the line is \(x ,\) connected between bus i and bus j. \({V}_{i}\) is the bus voltage of the ith bus.

Now bus injection power\(,\)

$${ P}_{i}={P}_{gi}-{P}_{di}$$
(34)
$${LQ}_{p}=4(\frac{x}{{V}_{i}^{2}})(\frac{x}{{V}_{i}^{2}}.{({P}_{gi}-{P}_{di})}^{2}+{Q}_{j})$$
(35)
$$\frac{d{LQ}_{p}}{d{P}_{di}}=4\left(\frac{x}{{V}_{i}^{2}}\right)\left(\frac{x}{{V}_{i}^{2}}\right).2.\left(-1\right)=\frac{-8{x}^{2}}{{V}_{i}^{4}}$$
(36)

Now considering Eq. 37,

$$\frac{d{P}_{r}}{ d{P}_{di}}=2{\alpha }_{i}{P}_{di}+{\beta }_{i}$$
(37)

Thus \(\frac{d{P}_{r}}{d{LQ}_{p}}\) =Penalty for change in price, due to change in \({LQ}_{p}\) due to change in demand (considering Eqs. 3437, Eq. 38 can be obtained)

$$p_{7} = \frac{{dP_{di} }}{{dLQ_{p} }}.\frac{{dP_{r} }}{{dP_{di} }} = - \frac{{V_{i}^{4} }}{{8x^{2} }}.\left( {2\alpha_{i} P_{di} + \beta_{i} } \right)$$
(38)

Hence determination of this penalty is necessary in order to penalise the calculated value of \({LQ}_{p}\) in every iterations to reach its optimal value in the given set of constraints. As stated earlier this technique will influence the nodal pricing in a way that the LDCs / consumers responsible for poor values of \({LQ}_{p}\)

3.3.5 Incentive for power utilities to maintain high inertia constant

The operation of the grid and the corresponding optimal generation and demand schedule should be able to maintain substantial or optimal value of the inertia constant in order to combat unexpected disturbances or contingencies. From Newton’s law, the force balance equation in a rotational system is given by (Eq. 39),

$$J\frac{dw}{{dt}} = T_{m} - T_{e}$$
(39)

The inertia constant is expressed as the ratio of kinetic energy stored at synchronous speed to the volt Ampere rating of the machine (\({S}_{b})\), with \({W}_{0}\) is the angular velocity of the rotor at synchronous speed (Eq. 40)

$$H=\frac{1.J{w}_{0}^{2}}{2.{S}_{b}}$$
(40)

Now, the synchronous frequency is given by (Eq. 41)

$$f=\frac{P}{2}{f}_{r}$$
(41)

where \(P=\) No. of poles and \({f}_{r}\) = Rotor frequency and \(w=2.\pi .{f}_{r}\)

From these equations, an expression for the change in frequency is given by (Eq. 42)

$$\frac{df}{dt}=\frac{-\Delta P}{2.H.{S}_{b}}{f}_{0}$$
(42)

\(\Delta P\) is the change of active power due to loss of intermittent renewable energy source and \({f}_{0}=\) the frequency before the loss of active power. For the nth generator unit this change or loss of active power generation will be reflected in change in the generation surplus and thus the minimum limit of load curtailment as expressed in (25). The loss of active power can be expressed as

$$\Delta P=\sum_{n=1}^{{n}_{g}}({P}_{gn0}-{P}_{gn})-\sum_{i=1}^{{n}_{d}}({P}_{di0}-{P}_{di})$$
(43)

The lowest or optimal value of inertia constant may be assumed from the base case operation of the grid. The optimal power flow objective function should be motivated towards minimum value of \(\Delta P\), corresponding to maximum value of inertia constant. This will maximise the generation of other generators which are not subjected to intermittency and will try to minimise the demand of each bus to retain the inertia constant of the base case. Thus, penalty should be added in the objective function to motivate the solutions towards a higher inertia constant, even in case of intermittency of generation.

3.4 Formulation of the proposed security constrained social welfare optimisation objective function

For the benefit of all the power market participants, Social Welfare Optimisation could be taken as the objective function of the grid with the suitable modifications as explained in previous section. A more potential objective function is explained in (Eq. 44).

$${\text{Maximise}}\,f(C_{n} , P_{lmaxj} ,T_{Lmaxj} ,V_{min} ) = \mathop \sum \limits_{i = 1}^{{i = n_{d} }} \alpha_{i} P_{di}^{2} + \beta_{i} P_{di} + \delta_{i} - \left( \begin{gathered} \mathop \sum \limits_{n = 1}^{{n_{g} }} C_{n} + \mathop \sum \limits_{j = 1}^{nlns} P_{lmaxij} .p_{1ij} + \mathop \sum \limits_{ij = 1}^{nlns} T_{Lmaxij} .p_{2ij} + V_{min} .p_{3} \hfill \\ + \left( {\mathop \sum \limits_{n = 1}^{ng} P_{gn} - P_{D} - T_{L} } \right).p_{4} + \mathop \sum \limits_{i = 1}^{{i = n_{d} }} \left( {P_{dimax} {-}P_{di} } \right).p_{5} + \mathop \sum \limits_{i = 1}^{{i = n_{d} }} PF_{i} .p_{6} \hfill \\ + \mathop \sum \limits_{j = 1}^{nlns} L_{Qpj} .p_{7} + \mathop \sum \limits_{n = 1}^{{n_{g} }} (P_{gn0} - P_{gn} ) - \mathop \sum \limits_{i = 1}^{{n_{d} }} \left( {P_{di0} - P_{di} } \right).p_{8} + LUF_{ij} .P_{LUF} \hfill \\ \end{gathered} \right)$$
(44)

The left part of this objective function contains the consumer /LDC cost benefit function. This characteristics or function can be derived from the willingness to pay of the LDCs and their bid data. The right-hand side is the operational standard constrained generation cost function. The first term of it is the total generation cost of all the generators or GENCOs participating. The second term is the transmission line congestion management cost, the third term is the transmission line power loss cost, the fourth term is the cost for maintenance of bus voltage at a minimum value (0.95 PU), the fifth term is the generation and demand balance management cost, the sixth term is the optimal load curtailment cost, the seventh term is the power factor management cost, the eighth term is the line stability index management cost, the ninth term is the higher inertia maintenance cost and the 10th term is for transmission line utilization.

4 Implementation of the proposed optimization framework—case study on IEEE 30—bus system

In this section of the work, the proposed Differential Evolution modified Security Constrained Social Welfare Optimisation Algorithm has been described step by step. This section also depicts the case studies conducted to demonstrate the efficacy of the proposed algorithm with respect to the traditional SCOPF. The mathematical model of DEQPSO algorithm and description of modified IEEE 30 bus System have been presented in the Appendix section.

4.1 The proposed DEQPSO based social welfare optimisation framework

The step-by-step descriptions of the proposed optimisation algorithm are as follows:

  • Step 1. Appropriate parameters of DEQPSO, error gradient (0.01X 10–25), Dimension of the swarm (27 (6 generator bus, 19 load bus)), number of iterations (T = 300), Population size(35), random numbers (\(\varepsilon =0.35)\), Contraction and Expansion factor (λ = 0.25) are chosen. For the first iteration arbitrary values of penalties are assumed.

  • Step 2. Position of the solution particles which are random combinations of 6 generation in Mw and 19 Load in Mw within the maximum and minimum limit is calculated by equation no. 3A and 4A.

  • Step 3. For each of the solution particle, the fitness function or the Social Welfare objective function is calculated as per Eq. 44. For the 1st iteration positive arbitrary values of penalty is assumed. The base case data is either available or assumed to be same as 1st iteration random data.

  • Step 4. The average of all the best positions is calculated using Eq. 5A as available in appendix section. As the objective is maximisation of social welfare, solution particles with maximum values of objective function are the best values.

  • Step 5. All the positions of the particles are updated as per Eqs. 6A and 7A with the knowledge of mbest in the next iteration. Before moving to the next iteration, the penalties for present iteration is calculated which are applied in the next iteration.

  • Step 6. In each iteration the fitness or the objective function as in Eq. 44 is calculated. The maximum value of the objective function corresponds to the best solution and the pbest, gbest and mbest are calculated corresponding to this best solution. In the next iteration, the optimiser looks for better solution particle with respect to the previous best solution. The error gradient between two consecutive iterations is also calculated which is the stopping criteria.

  • Step 7. In each iteration the incentives and penalties are estimated to be imposed in the objective function.

  • Step 8. In each iteration load curtailment limit is also calculated to penalise the objective function accordingly so that minimum dispatch requested by the LDC/Consumer through Demand Response can be ensured.

  • Step 9. In each iteration the donor vectors are calculated in as per Eqs. 8A and 9A. The difference vectors add versatility to the solution particles so that solutions with better fitness can be obtained.

  • Step 10. The knowledge of crossover probability (CR) in this step decides the ultimate position of the solution particle which may be better than or equal to QPSO results as per Eq. 10A.

  • Step 11. The solution particle meeting step 6 criteria are tested for its feasibility of implementation in the power network.

5 Implementation of the proposed framework in IEEE 30-bus system

In the base case (Sl no. 1, Table 2), the DEQPSO based optimizer, with the SCOPF objective (Eq. 2) produces minimum generation cost at standard operating condition in respect of bus voltage, transmission line Power Loss and maximum power flow. But as Active power loading (Sl. No. 2, 3, 4 of Table 3), reactive power loading (Sl. No. 5, 6, 7) and (N-1), (N-2) and (N-3) contingencies worsen, the generation cost increases for sustainable operation of the grid.

Table 2 Case studies with security constrained optimal power flow (SCOPF)
Table 3 Case studies with security constrained social welfare optimization with demand response

The Table 3 shows the results for the proposed security constrained Social Welfare optimization technique with demand response. Imposing the same loading conditions with social welfare as objective and with the incentives on maintaining better Transmission line stability and demand response, better operating condition with remarkably less generation cost was found. However, there is a considerable load curtailment which is a threat to reliability of supply. In Eq. 25, a formula related to maximum allowable load curtailment has been derived in this paper. Hence although the LDCs/consumers are subjected to load curtailment, the system can operate within safe limit of reliability if the allowable load curtailment is not violated. The same fact is substantiated in Fig. 8 which shows that although the nodal price of Social Welfare optimization is less than that of SCOPF, the load curtailment in case of former is more but the same is within safe limit as formulated in Eq. 25.

Fig. 8
figure 8

Load curtailment with the proposed algorithm with respect to ISO set limit of each bus

In addition, the proposed social welfare optimization algorithm in stressed condition of the power network produces improvement in Bus PF (Fig. 9), reduction in nodal price (Fig. 10), improvement in LSI (Fig. 11) creating a more secure operational zone of the network. The reduction in nodal price is due to the incentives provided by the solution being driven by penalties against each social contribution of the LDCs/Consumers.

Fig. 9
figure 9

Improvement of PF in proposed social welfare optimization

Fig. 10
figure 10

Nodal or bus-wise price of electricity per MW

Fig. 11
figure 11

Improvement of LSI with proposed algorithm

The proposed algorithm also is capable of producing generation and demand schedule with higher degree of dynamic rotor angle stability. In the extreme stressed condition of active power loading, it was examined that due to the incorporation of generation demand balance in the objective function, the result produced also optimises the rotor angle and thus contributes to the improvement of stability margin of the grid (Fig. 12). All the simulations are performed in MATLAB 2018a.

Fig. 12
figure 12

Improvement of rotor angle stability

6 Conclusion

For balancing the generation with load demand, smart grids use techniques like demand response to ensure Optimal Power Flow leading to maximisation of Social Welfare in terms of reduced electricity prices. However, existing OPFs are generally more biased towards reduction of electricity prices for consumers, often at the cost of violating operational constraints and challenging system security. It is imperative that a well-thought of Social Welfare Optimisation Framework must fulfil objectives likeoptimal load curtailment, maintaining adequate stability margins, maintaining high system inertia and line stability as well as maintaining acceptable bus voltages and power factor, apart from minimizing the nodal prices of electricity. This work proposed an optimisation framework which strived to achieve security constrained social welfare optimisation through DR technique, taking into account several operational issues such as intermittency of Renewable Generation, lowering of system inertia due to RES, degradation of bus PF, line stability, deviation of voltage etc. The algorithm developed in this paper ensured optimized load curtailment, reduction of network losses, improvement of operating power factor and mitigation of line congestion, among several other benefits, while ensuring nodal prices of electricity to be optimal. Differential Evolution modified Quantum Particle Swarm Optimisation (DEQPSO) was used in this research work to achieve the proposed objective. The case study on modified IEEE 30 Bus system presented in the paper established that the proposed algorithm outperformed the existing Social Welfare optimization algorithms delivering encouraging results.