An Ensemble Classifier Based Scheme for Detection of False Data Attacks Aiming at Disruption of Electricity Market Operation

Abstract

Wide area monitoring and control of modern power network demand real-time estimation of state variables from sensor measurements. Maintaining a high degree of reliability and accuracy in the state estimation process is important in avoiding any disruption in the electricity market operation. The market operation in power networks aims at providing a win-win situation for both the utility and consumer. The exposure and vulnerability of cyber components in smart grids allow for manipulating the electricity market by falsifying the state variables. The attacker can cause intentional profit/loss to the utility/consumer by misdirecting the estimated states through the injection of false data into the sensor information. Hence, maintaining integrity in the market operation demands a mechanism for detecting false data injection attack (FDIA). This paper proposes a classification-based approach for detecting FDIAs aiming at electricity market disruption. For any variation in the predicted and real-time nodal electricity price, the proposed decision tree (DT) based ensemble classifier is executed using state information to identify the prevailing scenario as a contingency or FDIA. The effectiveness of the proposed scheme has been extensively validated for various contingency and FDIA scenarios in IEEE 14 bus, 39 bus, and 57 bus test power systems.

1 Introduction

The traditional power system throughout the world is undergoing rapid changes with the increasing digitalization of system components, extensive accumulation of distributed energy resources, prevalent demand response, and inclusion of advanced protection and control equipment [1, 2]. The overall operation of the complex system through innovative monitoring of the system states has led to the development of smart grid. Reliable and resilient operation of the smart grid relies on the internet of things (IoT) technology-based monitoring and computing operations carried out at the control center [3, 4]. The dependence on cyber infrastructure, intelligent devices, and information and communication tools (ICT) has increased the vulnerability of smart grids to intimidating cyber-attacks [5,6,7]. Among the different cyber-attacks reported in the literature, FDIA is considered the most pertinent and severe attack [8]. Through FDIA, the attacker attempts to introduce malicious fake data into meter measurements, thereby manipulating the state estimation process. The manipulation is aimed at causing volatility in smart grid operation, which may even lead to a regional blackout.

The stable, secure and reliable operation of the smart grid is highly dependent on the real-time monitoring of the power system scenarios. Supervisory control and data acquisition (SCADA) systems receive real-time measurements from remote terminal units (RTUs) and transmit suitable control commands to the circuit breakers (CBs) and transmission system operators via a communication network. The overall monitoring and control of the smart grid are carried out by the energy management system (EMS). In contrast, the market management system (MMS) is responsible for performing all the economic operations in the smart grid [9]. The operations performed under EMS and MMS are highly dependent on the reliability of the states estimated from the sensor measurements. In a classical power system, the control actions rely solely on sensor measurements. In a smart grid, the inclusion of state information allows for improved monitoring and control during stressed scenarios like loss of sensor measurements and sensor failure.

The MMS executes the financial operations through the day-ahead market (DAM) scheduling and real-time market (RTM) price allocation [10]. The DAM involves purchasing and selling wholesale electricity for the next operating day through virtual bidding by the generation utilities to avoid price volatility. The final nodal price is evaluated in RTM operation as per the operating parameters executed in real-time. The final price settlement is performed in RTM in terms of locational marginal price (LMP) [11]. The LMP changes as a result of variation in the estimated states of the power system. Any disturbance arising out of load-generation imbalance or contingencies will impact the MMS operation through state variables. However, any contingency will have a more significant impact on the system states and hence, LMP. The possible contingencies include a change in network topology, generation failure and sudden loss of loads. To maintain accuracy in the state estimation task, bad data detection (BDD) methods are employed to filter out spurious meter measurements and/or noise arising out of faulty meters [12].

The attacker can exploit the high dependence of MMS on sensor information to cause financial mismanagement in the grid operation by falsifying the measurement data. The vulnerable cyber layer allows an attacker to manipulate the market operation by increasing/decreasing the profit/loss margin to a targeted utility/consumer. It can be achieved by injecting false data into the measurement within the endurable limits of BDD, thereby falsifying the state estimation process and LMP calculation. Precise information regarding the grid configuration eases the task of an attacker to launch FDIA aiming at the disruption of MMS operation [13]. The electricity market’s financial misconduct due to FDIA causes economic loss to the consumers and power suppliers due to incorrect electricity price. Simultaneously, the attacker makes a profit from the gap in the actual and manipulated price. Thus, ensuring integrity and dynamic balancing in the market operation demands a mechanism for the early detection of FDIA.

In spite of the wide volume of work reported for detecting FDIA, no technique has addressed the detection of FDIAs aiming at the disruption of MMS in the smart grid. Early detection enables the independent system operator (ISO) to take necessary control action towards eliminating the financial mismanagement in the electricity market. Motivated by the significance of maintaining fairness in MMS operation and the possibility of launching FDIA in the cyber layer, the present work proposes a classification-based scheme for detecting market-oriented FDIAs. Considering the significant impact of state variables on the market operation, the proposed scheme is formulated by mapping the system states with the prevailing market scenario (healthy, power system contingency or FDIA). The proposed FDIA detection scheme is processed in two stages. In the first stage, the LMP is monitored in real-time to detect any substantial variation between the day-ahead price allocated and the real-time settlement price. For any significant variation in the price, the classifier is executed in the second stage to detect the prevailing scenario as a contingency or FDIA. The classifier is executed by feeding the states estimated in real-time as input. Before detecting FDIA in real-time, the classifier is trained with a simulated dataset comprising state variables for various contingency cases and FDIAs.

Considering the wide range of operating scenarios of the grid and the ineffectiveness of a single stand-alone classifier in providing accurate and unbiased results for complex multi-dimensional datasets, an ensemble of multiple classifiers has been used to solve the classification problem [14]. The ability of DTs in attaining high accuracy with increased robustness and low computational cost has been utilized to implement the ensemble classifier by a set of DTs. The effectiveness of the ensemble classifier in achieving high classification accuracy for small and multi-dimensional data set has been outlined in [15]. Unlike an isolated classifier with the possibility of overfitting and biasness towards a particular class, in ensemble, the output of a set of classifiers is aggregated using a majority voting strategy to provide the final output.

The proposed scheme has been extensively validated for varying operating scenarios of IEEE 14 bus, 39 bus and 57 bus power systems. The effectiveness of the scheme has been analysed in terms of its ability to discriminate between contingency and FDIA. The major highlights/ contributions of the proposed work can be summarized as

• 1. Assessing the impact of FDIA towards financial mismanagement of electricity market operation in terms of deviation in nodal price between day-ahead and real-time electricity market.

• 2. Formulating the market oriented FDIA detection scheme as a classification problem.

• 3. Solving the classification task using an ensemble of DTs while considering the real-time estimated state variables as discriminatory attributes.

• 4. Validation of the proposed attack detection scheme for varying scenarios of contingency and FDIA in IEEE 14 bus, 39 bus and 57 bus test power system.

The rest of the paper is organized as follows. In Sect. 2, the fundamentals of state estimation process and FDIA have been discussed. Section 3, demonstrates the electricity market mechanism in smart grid along with the formulation and impact of FDIA in causing financial mismanagement. In Sect. 4, the proposed FDIA detection scheme is discussed, and the methodology is validated in Sect. 5. Section 6 summarizes the concluding remarks of the proposed work.

1.1 Related Work

A number of FDIA detection schemes have been proposed in the literature. A cumulative sum (CUSUM) algorithm based on the generalized likelihood approach for FDIA detection has been presented in [16]. The other notable detection techniques include schemes based on collaborative intrusion detection [17], sparse optimization [18], unknown input observer (UIO) [19], Go decomposition approach [20], spatial-temporal correlations [21] and Kullback-Leibler distance estimation [22]. In addition to the analytical approaches, the effectiveness of machine learning in identifying intricate patterns from the multidimensional data of complex systems has been utilized for detecting cyber-attacks. In [23], an intrusion detection system has been proposed using the deep learning technique. In [24], a random forest classifier based attack detection scheme based on information and logs obtained from phasor measurement units (PMUs) has been proposed. Compromised meters in data integrity attacks have been identified using artificial intelligence in [25]. In [26], neural network and naive Bayes classification scheme is proposed for anomaly detection in load forecasting during cyber-attacks. In [27], the intrusion detection capability has been enhanced by employing a two-level hybrid anomaly detection mechanism for the smart grid. The cyber-attack is classified against normal fault scenarios by training the classifier with a dataset comprising physical and cyber layer information [28]. In [29], the consumer energy consumption pattern is classified from a malicious pattern to detect energy theft using an ensemble-based classifier. In [30], dynamic state prediction using an ensemble-based DT approach is proposed to compare the predicted result with the real-time values for security analysis of a power system network. Although many works have been reported on the detection of FDIA launched to disrupt different operations carried out at the control center, no work has addressed the detection and impact of FDIA on MMS operation. Since the electricity market operation is not continuously monitored by protective relaying equipment, it is difficult to detect any malicious intrusion in the power network in real-time. However, the intrusion is reflected in the form of irregularity in MMS operation, which further causes economic losses/profit to the utility/consumer. Thus, availing the economic benefits of smart grid demands a reliable electricity market operation with robustness against possible manipulation of sensor information. The summary of all the notations/variables used in describing the proposed work is given in Table 1.

Table 1 Summary of notations/variables

2 False Data Injection Attack on State Estimation

2.1 State Estimation and Bad Data Detection

State estimation is the core operation performed at EMS to identify the precise operational state of the system. The measurement set $(Z_{meas})$ is collected from the sensors deployed throughout the power system network and is transmitted to the SCADA system through the communication channel. The measurement set comprises of bus voltage magnitude ($V_{b}$), real and reactive bus power injection ($P_{inj},Q_{inj}$), and the real ($P_{fl}$) and reactive ($Q_{fl}$) branch power flows. The states estimated in a $N_{b}$ bus power system consists of $2N_{b}-1$ states i.e. $N_{b}$ voltage magnitudes and $N_{b}-1$ bus angles [12]. The measurement set can be represented as

$$\begin{aligned} Z_{meas}=Hx+e \end{aligned}$$

(1)

The states estimated using weighted least squares (WLS) algorithm can be evaluated as

$$\begin{aligned} {\hat{x}}=(H^{T}RH)^{-1}H^T R Z_{meas} \end{aligned}$$

(2)

Since the measurements are sensitive to noise, calibration of instruments and several internal/ external factors or any inconsistency in estimation pose a threat to the integrity of estimated states. Hence, it is subjected to bad data detection (BDD) test to identify the occurrence of any abnormality in the measurement set. According to the largest normalized residual (LNR) test, a sensor measurement is considered valid only if the measurement residuals, i.e. the difference between the measured value and estimated value, are found to be less than the threshold value $(\alpha )$ i.e. [12]

$$\begin{aligned} r=\vert |Z_{meas}-H {{\hat{x}}}\vert |\le \alpha \end{aligned}$$

(3)

However, if an intruder can obtain the information regarding H matrix, a measurement set can be developed that can pass the BDD test. Manipulated sensor measurements passing the BDD test are known as false data.

2.2 False Data Injection Attack (FDIA)

Cyber-attack aims at manipulating the sensor measurements by injecting falsified sensor information into the communication channel during transmission from the remote terminal unit (RTU) to the control center. It can either be a random cyber-attack or a cyber-topology attack , where an arbitrary set of measurements or network connectivity status is manipulated to disturb the state estimation process. In FDIA, an intruder possessing information of the system configuration develops a properly formulated measurement dataset, that can falsify the estimated states.

In a measurement set $Z_{meas}$, the attack vector $Z_{a}$ can be injected such that $Z_{attack}=Z_{meas}+Z_{a}$; so that the states can be deviated to $x_{attack}={\hat{x}} + x_{a}$. The states estimated after the launch of FDIA is given as

$$\begin{aligned} \begin{aligned} x_{attack}&=(H^{T} RH)^{-1} H^{T} R Z_{attack} \\&=(H^{T} RH)^{-1}H^{T} R(Z_{meas}+Z_{a}) \\&={\hat{x}}+x_{a} \end{aligned} \end{aligned}$$

(4)

and the measurement residual evaluated after the FDIA is given as

$$\begin{aligned} \begin{aligned} r(x_{attack})&=\vert |Z_{attack}-H x_{attack}\vert | \\&=\vert |Z_{meas}-H {{\hat{x}}}+(Z_{a}-Hx_{a})\vert | \end{aligned} \end{aligned}$$

(5)

If $Z_{a}$ is injected equal to $Hx_{a}$, then $r(x_{attack})=r({\hat{x}})$, and hence, it can pass the BDD test. This allows the intruder to send false information for desired value of deviation in estimated states [31].

3 Formulation of False Data Injection Attack (FDIA) for Electricity Market Disruption

3.1 MMS Based Electricity Market Operation in Smart Grid

The deregulation in the electricity market industry imparts economic benefits to the end-user by providing cheaper electricity. It also provides bilateral involvement of utility and customer in the energy buying and selling mechanism to increase the satisfaction level at each stage of the power transmission process. The market operation performed by ISO is executed in two stages. The DAM is performed by solving OPF for the forecasted network scenario. The solution schedules the hourly energy price as per the generation offers, purchase/selling through bids by market participants, and physical constraints of the power network. It is solved by formulating the power delivery as a cost-minimizing optimization problem satisfying load generation balance while considering the physical and operating system constraints. The second stage of market settlement is performed in real-time and solved as per the states estimated from the sensor measurements obtained from the SCADA system and the network configuration identified by the network topology processor (NTP). The state estimation directs the optimal power flow (OPF) to allocate the generation schedule, and branch power flows, considering the system constraints and dispatch schedule cleared during the DAM settlement. Figure 1 depicts the block diagram of the state-dependent MMS operation performed to allocate the final nodal price in the real-time market settlement.

The nodal price calculated can be formulated as an optimization problem that aims at minimizing the following generation cost and is formulated as [13]

$$\begin{aligned} \small \min _{P_{gi}^{*}} \quad \sum _{i=1}^{N_{g}} C_{gi}(P_{gi}^{*}) \end{aligned}$$

(6)

subjected to

$$\begin{aligned}&\lambda : \sum _{i=1}^{N_{g}}P_{gi}^{*}=\sum _{d=1}^{D} L_{d}^{*} \nonumber \\&P_{gi}^{min}\le P_{gi}^{*} \le P_{gi}^{max},\quad \forall i = 1 , 2 , \ldots N_{g} \nonumber \\&P_{fl}^{min}\le P_{fl}^{*} \le P_{fl}^{max},\quad \forall fl = 1 , 2 ,\ldots N_{tl} \end{aligned}$$

(7)

The nodal price calculated during RTM can deviate from that of DAM allocated price by a physical or cyber-attack which falsify the sensor measurements or topology status. The attack results in an OPF solution different from real scenario. The incremental cost of generation and transmission resulting due to the new OPF creates unethical market mismanagement by increasing the cost of energy delivery.

3.2 False Data Injection Attack (FDIA) in Electricity Market

In this section, the possible attack scenarios aiming at the disruption of electricity market operation has been analysed. The analysis is carried out in terms of quantifying the impact of the attack on the deviation in nodal price between DAM and RTM settlement. As mentioned earlier in MMS, financial misconduct can be achieved by manipulating the sensor measurements while bypassing BDD. For launching the FDIA, a strong adversary possessing the following capabilities has been considered.

• 1. Complete knowledge about the system configuration.

• 2. Information regarding the optimal states and dispatch schedule for the day-ahead market settlement.

• 3. Accessibility and hence ability to manipulate all the sensor measurements. In case of a set of sensors being protected, accessibility is assumed for the unprotected sensors.

With the knowledge of system configuration and line parameters, the intruder can construct the H matrix (1). Thus, an attack model can be formulated to inject a false data vector such that the residual condition of (3) can be satisfied, thereby circumventing the BDD test. The difference between nodal price evaluated during DAM $(\lambda _{da})$ and RTM $(\lambda _{rt})$ reflects the profit acquired by the market participant by selling/ buying electricity during virtual trading. The deviation in nodal price at bus i is

$$\begin{aligned} dev_{i}=\lambda _{rt_{i}}-\lambda _{da_{i}} \end{aligned}$$

(8)

The deviation $dev_{i}$ can be manipulated by injecting false data to cause intentional profit/loss to a utility. Such attacks refrain the ISO from taking desired financial decisions and hence, the integrity of MMS is compromised.

Fig. 1

Block diagram of electricity market operation

3.3 Cyber Attack Formulation

The FDIA can be launched in the communication channel by injecting false data to either manipulate the sensor measurements (random cyber-attack) or/and network connectivity status (cyber topology attack). However, to design the corresponding attack vector, a set of constraints needs to be satisfied depending on the capability of the adversaries to access and manipulate sensor data. The formulation of possible attack vectors for different scenarios of intruder accessibility are dealt in the subsequent sub-sections.

3.3.1 Random Cyber Attack

The random cyber-attack aims at manipulating only the analog sensor measurements by injecting false data into the communication channel. The attack can be executed under the following constraints of intruder accessibility.

(a) Intruder having access to all the sensors All the sensors are assumed to be vulnerable for injecting false data into the measurement vector. The attack vector $Z_{a}$ can be generated with the knowledge of measurement Jacobian matrix H. With the injected attack vector, the measurement vector is manipulated as [8]

$$\begin{aligned} Z_{a}=Hc \end{aligned}$$

(9)

where c is an arbitrary integer vector. The generation of $Z_{a}$ can be framed as an optimization problem which aims at manipulating the measurement vector $Z_{meas}$ for maximizing $dev_{i}$, such that the branch power flows are constrained within the limits. Thus, for manipulating the MMS operation, the objective function can be framed as

$$\begin{aligned} \max _{Z_{a}} (dev_{i}) \end{aligned}$$

(10)

subject to

$$\begin{aligned} P_{gi}^{min}\le P_{gi}+\Delta P_{gi} \le P_{gi}^{max}, \forall i = 1 , 2 , \ldots N_{g} \\ P_{fl}^{min}\le P_{fl}+\Delta P_{fl} \le P_{fl}^{max}, \forall fl = 1 , 2 ,\ldots N_{tl} \end{aligned}$$

(b) Intruder having access to limited sensors Often the availability of limited budget and protection of certain strategic sensor through necessary security protocol hinders launch of FDIA on all the sensors. For such cases of limited sensors accessibility, the attack vector can be generated if the following condition is satisfied [32].

$$\begin{aligned} k_{m}\ge (m-n+1) \end{aligned}$$

(11)

where $k_{m}$ is the set of compromised meters out of m sensors and n is the total number of states. Under such condition the attack vector can be formulated as per (9).

$$\begin{aligned} Z_{a}=Hc \end{aligned}$$

where $c\,=\,b*a$ such that a is an integer vector and b is the binary vector in which the protected and unprotected sensors are represented by ‘1’ and ‘0’ respectively, i.e.

$$\begin{aligned} b_{i}&= {\left\{ \begin{array}{ll} 1 \ \ \quad \ if \ i \in k_{m} \\ 0 \ \ \quad \ if \ i \notin k_{m}\\ \end{array}\right. } \nonumber \\ c &=\, [a]_{m*1} [b]_{1*m} \end{aligned}$$

(12)

3.4 Cyber-Topology Attack

Based on the digital information received regarding the status of circuit breakers/switches, the processor continuously monitors the network information through NTP. Any inconsistency in analog/digital sensor information can be easily identified from the estimated states. However, a well-formulated cyber topology attack aiming at manipulation of network topology can evade possible attack detection by falsifying both the analog and digital sensor information. The NTP processes the network configuration in real-time using the line status information as [33]

$$\begin{aligned} A[i,j]={\left\{ \begin{array}{ll} \ \ 1 \quad \ if \ line\ connects\ from\ i\ to\ j \\ -1 \quad \ if \ line\ connects\ from\ j\ to\ i \\ 0 \ \quad if\ no \ line\ connection\ exists \end{array}\right. } \end{aligned}$$

(13)

For manipulating the network topology, the adversary falsifies the circuit breaker/switch status by altering 1s to 0s in matrix A or vice versa. The change in the information fed to the NTP is simultaneously accompanied by injecting false data in the analog measurements to mask the impact of topology attack at the control centre. The false data injection vector for cyber topology attack can be formulated as

$$\begin{aligned} Z_{a}=-\sum _{(i,j)\in \Delta \zeta } P_{fl,i-j}A(i,j) \end{aligned}$$

(14)

where $\Delta \zeta$ represents the changed network topology.

For initiating an attack under the above scenarios, an adversary can obtain the system matrix by accessing parameter and configuration information from an internal employee or using the day-ahead market data available on the website. Depending upon the objective to disrupt the market operation and constraints on the part of an adversary, either of the above three attacks can be executed. The BDD passed attack vectors can mislead the ISO by falsifying estimated states, thereby refraining the operator to take any control action against the unethical practice. This urges a mechanism to detect the attack by discriminating any disturbance at the control centre as a contingency or FDIA scenario.

Fig. 2

IEEE 14 bus system

Table 2 State estimation under pre-attack and post-attack scenario for IEEE 14 bus test system

3.5 Impact of False Data Injection Attack (FDIA) on Electricity Market Operation

The impact of FDIA on the state estimation task and further on the market operation has examined in this section. For the IEEE 14 bus test system shown in Fig. 2, the measurement set for state estimation comprises all the bus power injections and branch power flows obtained by solving the OPF under various operating conditions. The DAM operation is executed under a normal or anticipated contingency scenario (pre-attack), while the RTM operation is performed considering various FDIAs (post-attack) as discussed in Sect. 3.3. For the IEEE 14 bus test system (Fig. 2) with the attack vector $Z_{a}$ (9), the states estimated (bus angles) for actual ($Z_{meas}$) and falsified ($Z_{attack}$) measurements are shown in Table 2. The nodal electricity prices at all the buses under attack scenarios are estimated higher than the healthy condition. Hence, for the attack scenario, the market participant can buy electricity at various nodes during the DAM scheduling and sell during RTM to make an unethical profit. For both the examined state vectors, the corresponding market operation is performed, and the nodal price at different buses are depicted in Fig. 3.

4 Ensemble Classifier for FDIA Detection

In this section, the market-oriented FDIA detection task is formulated as a classification problem. The DAM and RTM operational procedure in allocating nodal electricity price is processed through a classification scheme to detect intrusion of any inconsistency in the data received at the control centre (Fig. 4). Any marginal deviation ($dev_{i}$) between the DAM and RTM allocated nodal price is processed by the classifier to categorize the prevailing scenario as an actual or FDIA induced contingency.

Fig. 3

Nodal price at different buses for Pre-attack and Post-attack states of Table 2

Fig. 4

Flow chart for proposed methodology

4.1 Decision Tree (DT) Based Ensemble of Classifier

In recent times, DT has emerged as a powerful machine learning tool for solving complex classification problems in multi-dimensional space. The learning capability of DT has been successfully applied to solve complex regression and classification tasks in different domains of power systems, i.e. cybersecurity, transient stability, intrusion detection, and islanding detection. Being a rule-based approach, DT is more transparent and human friendly as compared to black-box solutions like neural networks. For a given dataset, a DT considers all the possible mapping in feature space to designate a particular class/ category for a given input. Further, logical operations to correlate the features with the class allow for straightforward interpretation and easier real-time implementation on a digital platform. The effectiveness of DT in achieving high classification accuracy for high dimensional complex dataset has been outlined in the literature. Motivated by the same, the present binary classification problem of categorizing the status of the power network as attack/healthy scenario has been performed using DT.

However, in spite of their effectiveness in achieving high classification accuracy, quite often, DTs tend to overfit, which leads to improper generalization during validation [34]. Also, often DTs fail to incorporate intricate information embedded in the dataset during the mapping because of low bias and high variance property. The same leads to wider variation in the classification accuracy for a minor change in the learning variable. Such instability in the prediction due to the biasness of individual DTs can be overcome by incorporating the learning ability of a set (ensemble) of DTs. With the consideration of the mapping characteristics of a set of DTs, the limitation pertaining to the weak learning ability of an individual standalone DT is avoided. Thus, by employing an ensemble of DTs, the high variance and overfitting of an individual classifier are nullified by utilizing the mapping characteristics of multiple DTs [35].

For a given dataset, the classification accuracy is improved by combining the individual output of all the classifiers using a voting strategy [36]. Among the ensemble algorithm, the bootstrap aggregation/bagging scheme has been widely used for complex classification tasks aiming at low variance because of its reduced sensitivity to noisy dataset and simpler implementation. The bagging algorithm initiates by partitioning the training set into a group of subsets of the same size by sampling with replacement [37]. Further for each of the N subsets (Fig. 5), a DT model is fitted to the desired level of classification accuracy. The output (predicted class) derived from all the ‘N’ DTs are combined using a weighted algorithm. Each DT contributes to the final output in direct proportion to its classification performance. The overfitting of an individual DT does not impact the final output, since the same is offset by other DTs of the ensemble.

Fig. 5

Proposed DT based ensemble classifier for FDIA detection

4.2 DT Based Ensemble Classifier Design for FDIA Detection

Following the formulation of the FDIA detection task as a classification problem, different disturbances in the smart grid market operation is classified as a FDIA or contingency using an ensemble of DT classifier. Figure 5 outlines the sequence of operations employed to classify a disturbance by the ensemble classifier as a FDIA or contingency. It is to be noted that, the classifier is activated only if a deviation in between the nodal price for day-ahead $(\lambda _{da-i})$ and real-time $(\lambda _{rt-i})$ is observed (Fig. 4). Following the partitioning of data samples, all the dataset is trained simultaneously by a set of DTs. After training to the desired level of pre-defined accuracy, the individual output of each DT is combined to form the ensemble of classifier. For arriving at the final output, the widely used weighted majority algorithm is adopted. For the present binary classification problem involving two classes i.e. FDIA and contingency, considering the output of the ith DT being represented as $d_{i,j}\in [0,1]$, where $i=1,2,\ldots ,N$ and $j=1,2$. For FDIA, the output is assigned as 1, while $d_{i,j}=0$ for contingency. The values of $d_{i,j}$ derived for all the ‘N’ DTs are accumulated and fed to the weighted majority algorithm. For every classifier, a weight $(w_{t})$ is assigned in proportion to the classification accuracy. The class receiving the maximum outcomes is assigned as the final class of ensemble of classifier. The final output is represented as [34]

$$\begin{aligned} \sum _{i=1}^{N} w_{t} d_{i,j}= \max _{j=1}^{2} \sum _{i=1}^{N} w_{t} d_{i,j} \end{aligned}$$

(15)

Post-training of an ensemble of DT, any new scenario of MMS operation satisfying $\lambda _{da-i} \ne \lambda _{rt-i}$ is processed by the ensemble classifier with the corresponding state variables for detection of FDIA (if any). For the scenario being classified as a contingency, the final nodal price as calculated using (7) is allocated by the operator. In the case of a FDIA, necessary control measures are initiated to mitigate the impact of the data falsification on the market operation. The pseudo-code for the proposed ensemble of DT classifier for FDIA detection is given in Algorithm 1.

Table 3 Size of different training and testing data samples

5 Results and Discussion

The effectiveness of the proposed FDIA detection scheme is validated using IEEE 14 bus, 39 bus and 57 bus test power systems. The measurement dataset has been generated by simulating the system under varying contingency and attack scenarios using MATPOWER 7.0 [38]. The measurement set $(Z_{meas})$ is further used for state estimation using WLS algorithm. With the dataset of state vector for different contingency and attack scenarios, the ensemble classifier based FDIA detection scheme has been executed to identify any disturbance in the market operation as a FDIA or contingency.

As outlined in the previous section, for formulating the attack detection task as a binary classification problem, the state variables estimated from the sensor measurements are considered input to the classifier. The state variables for varying operating scenarios of the smart grid involving both healthy and cyber-attack cases have been estimated from the corresponding sensor measurements. Further, the estimated state variables have been used to generate the dataset for training and validating the DT based ensemble classifier. The choice of state variables for detecting any market-oriented cyber-attack is motivated by the fact that any deviation in the state vector will have a significant impact on the electricity market operation and hence, LMP. The increased dependence of market dynamics on the state estimation task allows for proper mapping between the state variables and the operating scenario (Attack/ Healthy).

The training dataset for FDIA is generated using the formulations outlined in Sect. 3.3, considering load variation between 0.8 p.u. to 1.1 p.u. of the rated value. For contingency scenarios, the states are estimated from measurements during a change in the network topology. The effectiveness of the ensemble classifier has been evaluated for detecting manipulated sensor measurements. Further, the detection accuracy of the proposed classifier is compared with the classical standalone Support vector machine (SVM) and DT based classifiers. Support vector machine (SVM) has been widely used for binary classification problems with known labels. However, the SVM necessitates a large amount of data for training to achieve higher accuracy, and its applicability is restricted to datasets with higher noise or data overlapping [39]. For discrete applications, DTs are known to be more effective than SVM [40, 41].

5.1 Classifier Performance Metric

In this sub-section, the effectiveness of the proposed attack detection scheme has been quantified using different performance indices and metrics.

Table 4 Training performance of DT based ensemble classifier

Table 5 Testing performance of DT based ensemble classifier in detecting cyber-attacks

(a) Accuracy The accuracy of the classifier represents the efficacy of the ensemble algorithm in classifying a falsified data from healthy data obtained at the control centre. It is quantified as [42]

$$\begin{aligned} Accuracy=\frac{TP+TN}{Total \ data} \end{aligned}$$

where TP and TN are the true positive and true negative samples, indicating the actual number of attacked and normal data being accurately classified, respectively.

The measurement data set comprising of healthy and attacked measurements are generated for different network configurations. The classification task is performed for the data encompassing both healthy and compromised sensor information. The training and testing dataset are respectively divided into $80\%$ and $20\%$ of the total data. The size of the training and testing data sets are demonstrated in Table 3. Each data sample comprises of estimated system states, i.e. 13, 38 and 56 load angles for IEEE 14 bus, 39 bus and 57 bus system, respectively. The training and testing results are estimated by means of 10-fold cross-validation, where the dataset for each fold are chosen randomly in 80 : 20 ratio. The ensemble classifier is trained for 20 learners with a maximum number of splits confined to 30. The training accuracy of the proposed ensemble classifier is identified to be $98.56\%, 98.80\%$, and $99.33\%$ for IEEE 14 bus, 39 bus, and 57 bus respectively (Table 4). The detection accuracy for the attack cases in the testing dataset is reported in Table 5 along with the performance on the type of attack (random cyber and cyber-topology attack).

Fig. 6

Accuracy of the classifiers with varying training samples for IEEE 14 bus test system

Fig. 7

Accuracy of the classifiers with varying training samples for IEEE 39 bus test system

Fig. 8

Accuracy of the classifiers with varying training samples for IEEE 57 bus test system

Table 6 Comparison of performance metric of different classifiers

To evaluate the impact of training data size on the classification accuracy, the ensemble algorithm has been executed by varying the sample size. The training dataset is divided into subsets comprising $20\%$ of the entire samples. In each step, the classification accuracy has been evaluated by sequentially adding the data subsets into the training dataset. A similar process is carried out for the SVM and DT classifier on the benchmark test systems. The variation in the classification accuracy on the testing dataset with an increase in the size of the training dataset is depicted in Figs. 6, 7 and 8 for IEEE 14, 39, and 57 bus system, respectively. The increase in the classification accuracy with an increase in the number of training samples is because of the improved extent of generalization achieved between the input (state variables) and operating scenarios (healthy/ attack). The higher proportion of the training dataset allows for incorporating more diverse operating scenarios in the mapping performed by the classifier. It can be observed that the effectiveness of the ensemble classifier over DT and SVM is more pronounced even when the data size is less. For a lesser number of training samples, the ensemble approach avoids the limitation of base classifiers of increased biasness toward a particular class. The test results (Table 6) reflect the effectiveness of the proposed ensemble classifier in achieving high classification accuracy for a multi-dimensional dataset. The improvement over standalone DT and SVM classifier is attributed to the proposed scheme’s ability to avoid possible over-fitting on the training data.

${\textit{(b)}\;F_{1} Score}$ In addition to accuracy, the performance evaluation of the proposed scheme has been carried out for precision and recall. Precision indicates the effectiveness of the classifier in detecting attacks only. The higher value of precision pertains to a lesser number of false alarms from the detection mechanism, while recall refers to the number of attacks being not detected by the classifier. Using the precision and recall index, the effectiveness in attack detection is quantified by $F_{1} Score$ as [39]

$$\begin{aligned} Precision(P_{r})=\frac{TP}{Predicted \ Positive} \\ Recall (R_{e})=\frac{TP}{Actual \ Positive} \\ F_{1} Score= 2 \frac{P_{r}R_{e}}{P_{r}+R_{e}} \end{aligned}$$

The appropriateness of the proposed classifiers in maintaining integrity in market operation has evaluated and compared with DT and SVM based schemes using the above indices in Table 6. The increase in the classification accuracy and precision with the size of the network is due to the increased volume of the dataset used for training the classifier. The improved performance of the classifier on the testing dataset reflects its generalization ability for cases not learned during the training phase.

Fig. 9

ROC Curve of the proposed ensemble DT based classifier in standard IEEE 14, 39, and 57 bus test systems

(c) Receiver Operating Characteristics (ROC) Receiver operating characteristics (ROC) plot is a vital tool for estimating the performance of a discrete classifier. It plots the graph between true positive rate (TPR) and false positive rate (FPR) to establish the trade-off between sensitivity and specificity of the classifier. True positive rate (TPR) and false positive rate (FPR) are evaluated as [43]

$$\begin{aligned} TPR= & {} \frac{TP}{TP+FN} \\ FPR= & {} \frac{FP}{TN+FP} \end{aligned}$$

where TPR and FPR indicate the number of attacked samples being correctly detected and false alarms, respectively. The ROC plot for the ensemble scheme demonstrates how efficient the classifier is for detecting the attacked scenarios. The ROC plots for the proposed ensemble DT-based classifier in IEEE 14, 39,  and 57 bus test system is shown in Fig. 9. A higher area under the curve of the ROC plot, authenticates the improved performance of the classifier.

(d) Confidence interval Confidence interval is used to quantify the uncertainty in the response of a classifier. It is estimated by providing bounds to the estimated results (classification accuracy). A lesser interval corresponds to higher precision in the classifier performance. The confidence interval can be estimated as

$$\begin{aligned} Confidence \ interval=s * \sqrt{\frac{accuracy(1-accuracy)}{n_{s}}} \end{aligned}$$

where $n_{s}$ is the size of sample and s is the number of standard deviations for Gaussian distribution (for $95\%$ confidence interval, $s=1.96$) [44]. With the proposed scheme, the confidence interval for IEEE 14 bus, 39 bus, and 57 bus test system is estimated to be (0.92±0.05), (0.95±0.04), (0.97± 0.03). The same has been illustrated in form of error bars in Fig. 10.

Fig. 10

Error bar plots of the proposed ensemble DT based classifier in standard IEEE 14, 39, and 57 bus test systems

5.2 Complexity Analysis

Effective performance of any cybersecurity mechanism necessitates a computationally cheap algorithm to detect any false data attack. The overall complexity of any classifier based attack detection is composed of the computational cost associated with two components i.e., feature extraction and classification. Unlike the reported techniques [29, 39], the proposed scheme involves estimation of features/attributes (state variables) using a non-iterative procedure, thereby avoiding the complexity associated with the feature extraction stage. The use of DT-based ensemble classifier for attack detection and classification allows for achieving high robustness and accuracy with the reduced computational cost compared to other classifiers like artificial neural network (ANN), SVM, and adaptive neuro-fuzzy inference system [34].

6 Conclusion

With the aim of maintaining the integrity of market operation performed by ISO, this paper proposes a scheme for detecting FDIA intended at the disruption of the MMS in the smart grid. The dynamic behaviour of system states and its impact on market operation has been used to identify FDIAs. Detecting the factiously developed nodal price allows the ISO to take the necessary steps so as to avoid market mismanagement. The market behaviour over all possible operating scenarios has been considered to discriminate the FDIA from actual contingency using an ensemble classification-based approach. The scheme involves continuous monitoring of the market operation by analysing the deviation in the nodal electricity price. For any substantial deviation, the classifier identifies the corresponding scenario as an actual contingency or FDIA. The falsified data generated within the physical constraints are found to impact the market operation, while benefiting a particular market participant (utility/consumer) at the cost of others. The effectiveness of the proposed methodology has been extensively validated for IEEE 14, 39, and 57 bus systems. For different operations of the MMS, the proposed DT based ensemble classifier is found to effectively detect FDIAs of varying type and magnitude. It has been observed that, with the proposed scheme, the attack detection accuracy for IEEE 14, 39, and 57 bus test system is observed to be $92.25\%, 95.13\%,$ and $97.08\%$, respectively. The improvement in the performance over other states of the classifiers (SVM, DT) is found to be more significant for systems of increased size. The uncertainty of response with regard to a false alarm is found to lie under $5\%$ for all the test systems. In the present work, the network scenario for the DAM and RTM is assumed to be similar, and the nodal price deviation between them has been considered as a threshold to activate the classifier. Future work in this direction would include a possible change in network configuration or sensor failure between the execution of DAM and RTM.