Definition and Theory of Transmission Network Planning

Abstract

With increasing interest in building large-scale solar parks and wind farms and the implementation of new environmental regulations that will result in retirement of some conventional power plants, the need for building new transmission is inevitable even in regions with low demand growth. Planning of such network expansion is therefore increasingly important, particularly because the cost of new transmission is typically higher in real terms than historical costs. This chapter contains five sections. In the first section, the impact of different factors affecting transmission expansion planning (TEP) is investigated, and different models for transmission investment financing and coordination are reviewed. The second section covers a literature review on TEP studies. Different TEP optimization formulations and available decomposition techniques are explained in the third section. Then, a general framework for solving large-scale TEP studies is reviewed, and computational challenges/potential solutions are investigated from different perspectives in section four. The chapter concludes with a section on numerical analysis, in which large-scale network models are used to demonstrate capabilities of optimization-based TEP studies.

1 Introduction

The transmission network is the backbone of the electric power system. Increasing penetration of renewable resources, energy storage devices, mobile and flexible demand, along with new public policies makes the future much more uncertain for transmission expansion planning (TEP). As the transmission network is a monopoly infrastructure, it is critical to expand and operate this network at minimum cost while keeping a high level of reliability. This is particularly the case in jurisdictions such as Electric Reliability Council of Texas (ERCOT) where investment and operation costs are distributed between all electricity users in the region.

Transmission expansion planning is the process of deciding which equipment should be selected, where it should be installed, and when is the best time to install it. Villasana et al. (1985) provide a hierarchy of three questions that should be answered in transmission planning:

• (a) What new facilities should be installed so that future operation will not be limited by transmission capacity?

• (b) What new transmission facilities can be economically justified versus the higher operation costs if new facilities were not installed?

• (c) What new generation sites can be justified versus new transmission facilities or higher operation costs?

These three questions specify main components of the objective function in TEP. In question (a), the objective function is to invest in the transmission network as much as we need to supply all demand without the transmission network affecting generation dispatch or demand supply. It is sometimes called reliability planning, in which the main concern is satisfying network reliability criteria. Unit operation set points are mainly defined based on experience or least cost. In the case of using lower operating cost units as much as possible, we will have the least operation cost but we may need to invest highly in transmission expansion, posing the question of whether the investment is cost-effective.

In the next hierarchy level (question b), the impact of operation cost on decision making for TEP is considered, which means it might be economical to dispatch some expensive power plants to supply demand instead of building some new transmission lines to dispatch all cheap power plants. The second question provides a better modeling property compared to the first one as it economically adjusts transmission investment cost and power systems operation cost, but it is computationally more expensive.

In question (c), which has the highest rank in the hierarchy, not only the impact of operation cost but also the impact of investment in generation sector on TEP is evaluated. In other words, it might be economical to invest on the generation side (for example, building new power plants close to demand centers) instead of the transmission side to supply the demand. It provides a better expansion plan (from economical perspective); however, it is much more computationally expensive, and planners would need to have the authority to make decisions about the location/capacity of new power plants.

Since generation expansion decisions are usually made by individual private investors in vertically unbundled electricity industries, the consideration of generation investment may be beyond the control of transmission planners. In this chapter, our main focus will be on the second question, and we assume we know the location and capacity of future generation units (with uncertainties). In principle, generation expansion could be added to the formulation.

1.1 Factors Affecting Transmission Expansion Planning

TEP studies are performed for different timescales, including, for example, near-term (for five years or shorter) and long-term (for more than five years), and for each timescale different parameters with different level of detail are considered. The main issues that affect TEP can be categorized into four groups, namely environmental issues, policy and regulatory issues, uncertainties, and network modeling, and these are explained briefly in the following.

1.1.1 Environmental Issues

Environmental concerns/limitations may directly affect transmission planning especially for line routing in particular areas such as regions with wildlife and endangered species, wetlands, national parks, historic areas, and military areas.

Furthermore, there are some environmental concerns that indirectly affect transmission planning such as limitation on pollution production by power plants in different areas that will shift future generation mix toward more renewables, and access to water resources necessary for building and operating power plants. These factors will directly affect the generation expansion (both generation mix and location), and consequently, transmission expansion planning will be affected.

1.1.2 Policy and Regulatory Issues

Policy-makers can affect TEP in several different ways such as who should pay for transmission network upgrades, how the cost should be distributed among them, what the transmission usage tariffs should be, electricity market price caps, and penalties for pollutions. This is discussed in more detail in Sect. 1.2.

1.1.3 Uncertainties

There are several uncertainties that affect TEP and should be addressed during the planning stage. They mainly can be categorized as micro- and macro-uncertainties:

• Macro-uncertainties such as future changes in economic growth, market rules, carbon emission issues, fuel price, generation mix/location and capacity, technology revolutions, etc.

• Micro-uncertainties such as load and intermittent resource variations, availability of power plants and transmission lines in real time, market price, behavior of market participants, etc.

The micro-uncertainty may be well represented by probability distributions, and an expected cost framework may be sufficient to capture main issues. In contrast, the macro-uncertainties may not have well-defined probability distributions, and risk may be much more important in this context, motivating approaches such as robust optimization (Bertsimas et al. 2011; Ruiz and Conejo 2015).

1.1.4 Power System Modeling

The modeling of the power system can have a significant impact on TEP studies. It affects the accuracy of results and computational time required for solving the problem. Main modeling factors are briefly reviewed in the following:

• Steady-state power flow formulation: It can be divided into three main categories: transportation model in which only the first Kirchhoff’s law is satisfied; the DC model that satisfies both first and second Kirchhoff’s laws, while ignoring network losses and reactive power requirements; and the AC model, which is the most accurate model for power system steady-state modeling and considers network losses and reactive power requirements as well as the first and the second Kirchhoff’s laws. There are also some hybrid models that are mainly driven from one of these three main models such as DC model with linear approximation of network losses or linearized AC model with loss and reactive power modeling.

• Transmission network model: Transmission network can be modeled as non-controllable or controllable. In the non-controllable model, the topology of the network is fixed, and in the controllable model, it is possible to use switching, phase shifters, FACTS devices, special protection schemes, and other available tools to control and manage flow on branches.

• Generation model: There are several parameters that affect a power plant’s operation, i.e., its maximum and minimum capacity limits, ramp rate capability, minimum up and down time, and some limits that are driven by specific generation technologies like total energy limit for hydropower plants (based on their reservoir capacity).

• Demand model: There are two different ways to model load, i.e., elastic or inelastic. In the elastic model, demand can be controlled with different signals such as the market price, but in the inelastic model, demand is modeled as a fixed quantity that should be supplied, if possible, and only curtailed in case of scarcity.

• Operation states: Normal and under contingency are two different types of operation states that can be evaluated in power system analysis (for both steady-state and transient analysis).

• Market model: There are several different aspects in market modeling like ideal versus real markets, day-ahead versus real time that may affect system operation costs and TEP.

1.2 Transmission Investment Financing and Coordination

Transmission system operation and expansion are heavily regulated because of their critical role in power system reliability and their natural monopoly. Although it might be owned/operated by different companies/organizations, the transmission network is an interconnected infrastructure in many countries and regions; therefore, coordination between owners/operators for efficient expansion and operation is critical to maintain power systems reliability and security while economically modeling future uncertainties. In this section, we briefly overview different regulatory schemes for coordination between transmission owners for capacity expansion, investment financing and cost recovery. For discussion regarding generation and load interconnection regulations and procedures, interested readers are referred to Regairz et al. (2017) for a more detailed review.

1.2.1 Transmission Organization Models

As discussed in (Regairz et al. 2017), transmission network ownership and operation model can be divided into three main organizational structures as follows:

• Vertically Integrated Utility (VIU) Model: In this model, which was a dominant model before electricity industry deregulation/restructuring, one company owns all generation, transmission, and distribution grid assets in a particular geographical area, and is a solely responsible for supplying its customers.

• Transmission System Operator (TSO) Model: In this model, which is common in the Europe, generation and customer supply are separated from transmission system to maintain the full independence of TSOs. In this model, a TSO is the owner and solely responsible for operation and expansion of the grid in its area.

• Independent System Operator (ISO) Model: In this model, which is common in the USA, not only is the transmission sector separated from generation and supply, but also its operation is separated from its ownership to enhance the independence of the system operator. In this model, ISO is responsible for systems and market operation, short-term and long-term resource adequacy and transmission expansion planning; however, the ISO does not own any transmission, generation or supply assets. Transmission owner companies own transmission assets and are responsible for maintenance and for most transmission operations, under the authority of the ISO.

1.2.2 Transmission Coordination: Planning and Investments

In this section, we briefly overview responsibility for transmission expansion plans development and investment financing (for each transmission organization model from Sect. 1.2.1) and how these activities are coordinated when expansion projects cross multiple transmission owners’ territories.

• For vertically integrated utilities, the utility is responsible to perform transmission and generation expansion studies for its area and will select/approve the cost-effective expansion plans to be built. Their performance might be overseen by a local government or a regulatory agency. Depending on their interconnection with neighboring networks, they may be required to meet some external reliability and security requirements as well. For example, vertically integrated utilities in the USA, connected to the bulk power system network, should meet NERC reliability requirements for power system planning and operation. The North American Electric Reliability Corporation (NERC) is a not-for-profit international regulatory authority whose mission is to assure the effective and efficient reduction of risks to the reliability and security of the grid. It develops and enforces reliability standards that span the continental USA, Canada, and the northern portion of Baja California, Mexico (NERC 2019). Moreover, in this structure, the vertically integrated company itself is responsible for financing selected plans usually on the basis of a state-regulator approved rate of return on investment, based on cost of service to be discussed below.

• TSO performs planning studies for the network within its area and will send the results to a regulatory board for approval. TSO makes the investment to build approved expansion projects and will operate and maintain them. In Europe, the Ten Year Network Development Plan (TYNDP) presents a forward-looking non-binding proposal for electricity transmission infrastructure investments across 34 European countries (Regairz et al. 2017). For projects between countries, investment decisions are made based on specific agreements between parties who benefit from or are affected by the project.

• In the ISO model, ISOs are mostly responsible for performing transmission expansion planning studies. However, all stakeholders including generation owners, load serving entities, and transmission owners can participate in the planning process by submitting their proposals for transmission upgrades to the ISOs for their review/selection. The final expansion plans are sent to transmission owners for construction after they are approved by a board of directors or a regulatory agency. In the USA, CAISO, SPP, ERCOT, MISO, PJM, NYISO, ISO-NE are major ISOs. The Federal Energy Regulatory Commission (FERC) is a federal government agency that regulates the interstate transmission of natural gas, oil, and electricity (FERC 2019). Except for ERCOT, all other ISOs in the USA are under FERC’s jurisdiction.

1.2.3 Tariffs and Regulatory

Transmission and distribution sectors of electric power systems remained regulated in many countries even after thirty years of electricity industry reform. Traditionally, a regulated firm’s budget constraint was formed based on the assumption that regulators have perfect information about technologies, costs, and consumer demands. However, in reality, regulators have imperfect information about costs and service quality opportunities, and in many cases, the regulated firm has more information than the regulator, which is a disadvantage for regulators. Four main approaches for compensating a regulated firm’s costs, namely cost of service, price cap, incentive, and merchant-regulatory mechanisms, are briefly discussed in the following.

Cost of service is one of the widely used approaches for compensating regulated firms. In this method, it is effectively guaranteed that essentially all operation and investment costs that actually occurred will be compensated. Although it provides incentive to invest more on the grid maintenance and expansion, it does not provide any incentive for improving performance and reducing costs. Price cap regulatory mechanism is designed to provide incentives for managers to reduce costs and improve performance. Because of uncertainties in firm’s actual realized costs, a low price cap may not cover all their costs. As regulators should consider financial viability, a high price cap should be selected to cover uncertainties, but this may decrease the efficiency of this approach (Joskow 2006). Incentive regulatory mechanism is designed to address this issue by providing a menu of options for different situations. A comprehensive review of incentive regulatory mechanism is provided in (Armstrong and Sappington 2005; Blackmon 1994; Sappington and Sibley 1988). Merchant-regulatory mechanism allows a combination of regulated and merchant investments. It provides more flexibility on planning and project approval stages but introduces cost recovery risks for merchant-based projects as there is no guarantee for their cost recovery. Hogan et al. (2010) and Rosellon and Weigt (2011) discussed this mechanism in detail.

For transmission network investment cost recovery, different mechanisms can be used. In the Europe with TSO model, regulated tariffs using incentive-based mechanism is used to recover transmission related costs (including investment and operation). In the USA with ISO models, a combination of regulated tariffs and merchant regulatory is used to recover investment and operation costs at transmission level. In regions with vertically integrated utility model, cost of service and price cap regulatory mechanisms are mainly used to guarantee cost compensations. For more details, interested readers are referred to references (Vogelsang and Finsinger 1979; Vogelsang 2001; Hesamzadeh et al. 2018).

Whatever the regulatory mechanism, there is an implicit assumption that the system is planned according to some criterion to achieve a particular objective. Historically, transmission planning has not, however, utilized systematic optimization approaches, but rather has involved expert knowledge and trial and error. The rest of this chapter focuses on systematic transmission expansion planning that is aimed at explicitly finding an optimal plan with less reliance on expert knowledge. It is organized as follows: in Sect. 2, a literature review on TEP studies with major focus on different TEP formulations, reliability, and uncertainty modeling are provided. In Sect. 3, stochastic and robust TEP optimization formulation along with different decomposition techniques are discussed. Then, we review a general framework for solving large-scale TEP studies and evaluate computational challenges from different perspectives in Sect. 4. In Sect. 5, numerical results on solving real-size networks are discussed. This chapter is based, in part, on (Majidi-Qadikolai 2017).

2 Literature Survey

As discussed in Sect. 1.1, there are significant factors affecting transmission expansion planning, which make TEP a multi-dimensional and very complex problem. A major question is how to model/formulate all those parameters, and more importantly how to solve TEP for large-scale networks. Making assumptions and simplifications are inevitable, and we seek to do so in a way that does not fundamentally invalidate the analysis. Environmental, legal, policy, and regulatory issues mostly can be considered in near-term TEP/line design stage and can be partially addressed in developing candidate lines for long-term TEP. Therefore, we can model their impacts outside of TEP optimization formulation and thereby significantly reduce TEP problem size. Uncertainties can be captured either by developing different possible scenarios or by developing uncertainty boundaries and using robust optimization techniques. Villasana et al. (1985) discussed different levels of complexity of the TEP optimization problem as follows:  

Level I::

Considering all quantities deterministic (future load, generation, and fuel price), static model (one planning horizon), single operation condition (normal operation), all variables as continuous (continuous line capacity for expansion);

Level II::

Deterministic quantities, static model, single operation condition, mixed-integer problem (MIP) statement (binary decision variables for building transmission lines);

Level III::

deterministic quantities, static model, multi-operation conditions (normal and under contingency operation states), MIP statement;

Level IV::

Deterministic quantities, dynamic model (multi-planning horizons), multi-operation conditions, MIP statement;

Level V::

Stochastic quantities (uncertainties in load, generation, and fuel price), dynamic model, multi-operation conditions, MIP statement.

  By moving from level I to level V, the model will be more accurate and closer to reality, but much more complicated and challenging to solve. By using the DC model, stage I represents a continuous linear optimization problem. Adding integer variables makes it a mixed-integer programming (MIP) problem in level II. Level III adds contingency analysis into TEP that significantly increases the problem size and can easily make TEP optimization problem intractable. TEP moves from static to dynamic in level IV that increases the number of binary variables in the optimization formulation, and TEP is modeled as stochastic dynamic TEP in level V.

2.1 Solution Methods

Using some expert knowledge (EK) for solving large-scale TEP optimization problem is inevitable with current existing machines and software. But there are different points of view on how EK should be integrated into the transmission planning decision making process. Historically, decisions are mainly made by experts based on their expertise instead of using an optimization-based method. A second approach integrates EK into the TEP decision-making process by using EK to choose the worst case for planning, choose the list of possible contingencies, or reduce the list of candidate lines. A third approach converts EK into some criteria (where applicable) and tries to integrate them into a TEP optimization framework. Compared to the second approach, this method is systematic and tractable on the one hand, and more challenging from the modeling perspective on the other hand. The fourth approach tries to use EK as little as possible and solve the problem through pure mathematical formulation. These purely mathematically driven methods are usually computationally very expensive and are not practical for large-scale problems.

In heuristic models, approaches one and two, the TEP problem is solved through several steps of generating, evaluating, and selecting expansion plans, with or without the user’s help (Latorre et al. 2003). One of the common heuristic methods is to use sensitivity analysis to select additional circuits (Pereira and Pinto 1985; Latorre-Bayona and Perez-Arriaga 1994; Majidi-Qadikolai and Baldick 2015; Monticelli et al. 1982). MISO Midcontinent ISO (2016), ERCOT ERCOT System Planning (2016), and CAISO Market & Infrastructure Development (2016) are three examples of independent system operators in the USA that use different heuristic methods for TEP.

In optimization-based methods, approaches three and four, a mathematical formulation for TEP is developed and the problem is solved using classical optimization programming techniques. Optimization-based methods are computationally very expensive and have historically been thought to be impractical for large-scale TEP problems (Latorre et al. 2003; Munoz et al. 2015). However, modern computing systems and optimization software, together with novel formulations, have begun to make optimization-based methods practical for large-scale planning. Several methods are proposed to formulate the TEP problem.

Using linear approximation of AC power flow equations is one of the most popular simplifications for modeling nonlinear power flow equations in high-level TEP studies. The accuracy of linear approximation of power flow equations (DC model) is evaluated in (Van Hertem et al. 2006; Baldick et al. 2005; Overbye et al. 2004). In Van Hertem et al. (2006), authors compared the results of AC and DC power flow results for the IEEE 300-bus system and showed the error between DC and AC results will be less than 5% when the assumptions of DC power flow are satisfied. Baldick et al. (2005) performed sensitivity analysis in power systems with DC and AC models and demonstrated that it provides a relatively reliable approximation of the behavior of the system. Overbye et al. (2004) showed that locational marginal prices (LMPs) that drive the economic analysis of power system operation will not be significantly affected when the AC model is approximated with the DC model so long as various assumptions are satisfied.

In Villasana et al. (1985) and Garver (1970), transmission planning is formulated as a simple linear programming (LP) problem with continuous decision variables. Villasana et al. (1985) proposed a LP method with continuous variables for optimal transmission planning by minimizing load curtailment. As transmission line capacity is lumpy, considering capacity to be a continuous variable is not accurate. Villanasa (1984) proposed a mixed-integer programming (MIP) formulation using binary decision variables for selecting new lines with DC power flow approximation. This method is more accurate in representing new line capacities, but the proposed formulation is not computationally efficient.

Kirchoff’s second law is represented with two inequalities in a mixed-integer disjunctive model, each related to one possible flow direction in (Bahiense et al. 2001). This technique increases the number of constraints and provides better conditioning properties by tightening constraints. Bahiense et al. (2001) also used GRASP meta-heuristic method to provide an upper bound feasible solution. In Alguacil et al. (2003), power network losses are integrated into TEP optimization problem using piecewise linear loss function for each line. It provides more accurate power system model for planning purpose while preserving linearity and may affect the selected expansion plan for networks with relatively high losses such as systems with long transmission lines. However, the simulation time for this case is increased around five times compared to the case without losses.

Benders decomposition (BD) is used in several contexts as a powerful tool for decreasing simulation time for solving large-scale optimization problems. Mathematical formulation for implementing Benders decomposition for transmission and generation expansion planning was developed by EPRI in 1988 (Granville et al. 1988). Gomory cuts are added to Benders cuts in (Binato et al. 2001) to improve the performance of BD for large-scale MIP problems. To overcome the non-convexity of transmission planning problem  Romero and Monticelli (1994); Rosellon and Weigt (2011) proposed a three-phase hierarchical decomposition method to find the global optimal answer. They used BD to solve each phase and transferred Benders cuts into the next phase to integrate different phases. Park and Baldick (2013) considered load and wind as dependent and uncertain variables and used a two-stage stochastic model and sequential approximation technique to solve TEP optimization problems with BD. A dynamic transmission expansion planning is formulated in (Munoz et al. 2014) and authors compared the performance of stochastic programming with deterministic and heuristic methods. Munoz et al. (2013) evaluated the impact of different approximations on TEP with renewable portfolio standards. Munoz et al. (2014) and Munoz and Watson (2015) proposed a new approach for multi-regional transmission and generation expansion planning with Benders decomposition technique, which is enhanced by developing new lower bounding constraints that increase convergence speed. They applied the model to large-scale networks with a relatively large number of scenarios to capture uncertainties and evaluated the impact of optimality gap on simulation time. A complex mathematical model for centralized transmission planning and decentralized generation expansion planning is developed in (Jin and Ryan 2014). To represent the interaction between generator and transmission planners during transmission and generation expansion planning, game theory-based approaches are used by (Tohidi and Hesamzadeh 2014; Tohidi et al. 2017a, b; Ruiz and Contreras 2007; Yen et al. 2000). To decrease computational efforts, all above-mentioned references ignored contingency analysis in their proposed methods for transmission planning. So, there is no guarantee that selected optimal plans by these papers satisfy reliability requirements.

2.2 Power System Adequacy and Reliability

The power system should be adequate and reliable. Based on North American Electric Reliability Corporation (NERC) definition “Adequacy is the ability of the electric system to supply the aggregate electric power and energy requirements of the electricity consumers at all times, taking into account scheduled and reasonably expected unscheduled outages of system components” and “Operating reliability is the ability of the electric system to withstand sudden disturbances such as electric short circuits or unanticipated loss of system components” (NERC 2007). In standard 51, NERC categorized system adequacy and security into four levels A-D (NERC 2005). Level A refers to system performance under normal conditions (no contingency), and in level B, system performance following the loss of a single bulk system element is evaluated. In Levels C and D, system performance under loss of two or more bulk system components and extreme events are evaluated, respectively. Categories A-C should be evaluated for near-term and long-term planning, and category D should be considered for near-term planning only.

The power system should be planned and be operated in a way to be able to supply all loads under normal conditions and in case of a single outage in system components (levels A and B). This is called the $N-1$ criterion (NERC 2005; Electric Reliability Council of Texas 2014). To satisfy this standard, system operators usually use security-constrained optimal power flow (SCOPF) or security-constrained unit commitment (SCUC) to dispatch/commit power plants. Post-contingency re-dispatch (Monticelli et al. 1987), congestion management (Majidi et al. 2008), transmission switching (Ruiz et al. 2012b; Hedman et al. 2008; Majidi-Qadikolai and Baldick 2015; Ruiz et al. 2012a), or using FACTS devices (Majidi et al. 2008; Ziaee et al. 2017) are techniques used to add flexibility to transmission operation and subsequently reduce operation costs. In Monticelli et al. (1987), a new algorithm for security-constrained optimal power flow (SCOPF) is proposed that considers post-contingency corrective rescheduling to decrease dispatch costs. To integrate transmission switching in the system operation, Ruiz et al. (2012a) used the flow cancelation technique to model switching. They showed that this technique is faster than using binary variables to change the status of lines in topology control when the number of switching lines in limited.

Various researchers use either the $N-1$ criterion or probabilistic approaches such as loss of load probability (LOLP) or loss of load expectations (LOLE) for power system adequacy and security evaluation. Leite da Silva et al. (2010) explained drawbacks of each method and evaluated the impact of considering different reliability criteria on TEP. They performed numerical analysis for the Garver 6-bus system (Garver 1970) to compare the performance of these methods. The result shows that TEP with $N-1$ criterion requires more investment compared to TEP with probabilistic approaches as it should supply the demand under all single contingencies. Loss of load cost (LOLC) as a reliability index is calculated for the selected plan for both cases, and LOLC for TEP with $N-1$ criterion is much less than LOLC for TEP with the probabilistic approach, showing the impact of extra investment on improving system reliability. By considering $N-1$ criterion, the system quality and reliability indexes will be less sensitive to load variations and components’ rate of outage compared to probabilistic approaches.

O’Neill et al proposed a comprehensive mathematical formulation for dynamic optimal power system planning and investment by integrating unit commitment, transmission switching, and $N-1$ contingency analysis into a power system operation cost formulation in (O’Neill et al. 2011). But as the authors mentioned in their paper, it is a very complex and computationally expensive model even for a very small case study, so it is not practical for large-scale networks at this time. More practical formulations for TEP optimization with $N-1$ contingency analysis are formulated in (Rudkevich 2012; Khodaei et al. 2010; Moreno et al. 2013; Zhang et al. 2012; Majidi-Qadikolai and Baldick 2016a, b, 2018). Rudkevich (2012) proposed a nodal capacity market framework for generation and transmission expansion planning. He used the flow cancelation technique to represent a fixed list of contingencies in a reliability dispatch formulation, in which all resources are dispatched at zero costs and load shedding will be penalized at value of lost load (VOLL) price. Khodaei et al. (2010) proposed a three-stage transmission and generation expansion planning optimization formulation with Benders decomposition technique and considered contingency analysis for all existing and candidate lines and integrated transmission switching to alleviate violations in line flows. In Carrion et al. (2007), transmission expansion and reinforcement are formulated as a stochastic optimization problem to reduce vulnerability of the system in case of deliberate attacks.

2.3 Uncertainties

Fast technology changes, new policies, increasing penetration of mobile/flexible demand along with intermittent nature of renewable resources make it hard to accurately predict future generation mix/location and demand as inputs for TEP studies; therefore, these uncertainties should be explicitly modeled/evaluated in TEP process by system planners. It should be emphasized that developing a single expansion plan using methods that heavily depend on engineering judgment can result in a plan that is costly and inefficient when the implications of uncertainties are considered. Munoz et al. (2014), Munoz and Watson (2015) and Cedeño and Arora (2011) evaluated the impact of ignoring uncertainties on transmission planning by comparing the results of deterministic, heuristic, and stochastic TEP for different case studies. Their result shows that stochastic TEP may select some lines that will not be selected by either deterministic or heuristic methods.

The TEP optimization problem can be formulated as a two-stage stochastic resource allocation problem (a class of mixed-integer stochastic programming) to explicitly model uncertainties using a finite set of scenarios (Kall and Woodruff 1994). In this formulation, in the first stage, a decision about building a new transmission line is made, and the impact of this decision on power system operation under different scenarios is evaluated in the second stage. To capture all macro- and micro-uncertainties, usually a large number of scenarios are generated in the early stages of planning (there are different methods to generate scenarios to represent uncertainties such as Monte Carlo method (used by (Akbari et al. 2011)) and using historical data with statistical modeling (used by (Park and Baldick 2013)), and different clustering techniques are developed to reduce the number of scenarios (Munoz and Watson 2015; Park and Baldick 2013). There are also some commercial packages such as (SCENRED GAMS 2002) that can be used for this purpose. Akbari et al. (2011) integrated Available Transmission Capacity (ATC) constraints into a multi-stage stochastic TEP problem. They used GAMS/SCENRED as a tool to reduce a very large number of randomly generated scenarios and solved TEP with all contingencies for the IEEE-24 bus system. The impact of adding ATC constraints to TEP is evaluated; however, the performance of the model for large-scale systems is not discussed. Alvarez Lopez et al. (2007) integrated uncertainties and risks in load, availability of generation and transmission lines into a stochastic generation and transmission capacity expansion planning problem and formulated it as a nonlinear mixed-integer optimization problem. A probabilistic method for capturing uncertainties in TEP is proposed in (Buygi et al. 2004). They developed probabilistic locational marginal pricing (LMP) index and suggested value-based criteria, i.e., decreasing congestion cost and reducing weighted deviation of mean of LMPs for selecting new transmission lines. In  Zhang et al. (2015), Benders decomposition with aggregated multi-cuts is used to solve TEP under uncertainties. Pringles et al. (2015) used least-square Monte Carlo dynamic programming to solve stochastic TEP. They deployed sensitivity analysis to determine decision regions to execute, postpone, or reject transmission investment candidates.

Although formulating TEP as a two-stage stochastic optimization problem provides a strong modeling capability (Park and Baldick 2013; Guo Chen et al. 2012; Munoz and Watson 2015; Majidi-Qadikolai and Baldick 2016a), solving the extensive form (EF) of this problem is not tractable even for medium size problems especially when $N-1$ contingency analysis is added to the problem. Therefore, decomposition and heuristic techniques should be used for solving TEP for medium to large-scale systems.

Robust optimization is another method to integrate uncertainties into the TEP formulation. In robust optimization, uncertainties are represented using a range for each uncertain parameter or a budget of uncertainty for collections of uncertain parameters instead of developing scenarios (as used by stochastic optimization), and it finds a plan that is robust for the worst-case scenario. In this case, the final result is usually too conservative, which motivates an adaptive robust optimization (Bertsimas et al. 2011) formulation with budget limit constraints to mitigate the level of robustness (conservativeness of results). Ruiz and Conejo (2015), Garcia-Bertrand and Minguez (2016), Minguez and Garcia-Bertrand (2016) formulated the TEP problem as an adaptive robust optimization.

3 Transmission Expansion Planning Formulation and Decomposition Techniques

As stated in Sect. 2, the transmission expansion problem can be formulated as static (single-stage) or dynamic (multi-stage), deterministic or probabilistic, stochastic or robust. In this section, we investigate static TEP with stochastic/robust optimization techniques to address uncertainties. For mathematical formulations, variable/parameter definitions are provided in the beginning of this chapter.

3.1 Two-Stage Stochastic TEP Formulation

As discussed in Sect. 2.3, stochastic programming is one of the widely used methods to model uncertainties (by developing different scenarios) in the decision-making process for resource allocation problems. To capture uncertainties, different scenario generation/reduction methods might be used to finalize the input scenario set. The quality of scenarios is critical and can significantly affect the selected expansion plan. For example, in ERCOT, historical data along with workshops with stakeholders are used to develop scenarios for long-term TEP (ERCOT System Planning 2014). It should be mentioned that minimizing the expected value is a better criterion for micro-uncertainties in cases where probability distributions can be estimated from empirical data. The two-stage stochastic TEP is formulated as follows:

$$\begin{aligned} Z^{*}{=}&\min \limits _{x} \{ \zeta ^\intercal \textit{\textbf{x}} {+} \mathbb {E}_{} \min \limits _{y \in \Xi } Q(\textit{\textbf{x}},\tilde{\xi },y) \} \end{aligned}$$

(1)

$$\begin{aligned} \mathrm {st.} \quad&\textit{\textbf{x}}\in \{0,1\}^{|N_l|} \end{aligned}$$

(2)

where $\textit{\textbf{x}}$ is the first stage binary decision variable, $\tilde{\xi }$ is a random variable vector for second stage uncertainties, y is the second stage continuous decision variables vector, and $\Xi$ defines the feasible region for variable y. $\mathbb {E}_{}{\min \limits _{y} Q(\textit{\textbf{x}},\tilde{\xi },y)}$ represents the expected value of operation costs including load shedding and wind curtailment penalty and generation costs for TEP problem formulation with the expectation taken over the random variable $\tilde{\xi }$. This expected value is approximated with a weighted sum of a limited number of scenarios as follows (Ermoliev and Wets 1988):

$$\begin{aligned} \mathbb {E}_{}{\min \limits _{y}Q(\textit{\textbf{x}},\tilde{\xi },y)}\approx \sum \limits _{\omega \in \Omega }P^{\omega } \min \limits _{y^\omega } Q(\textit{\textbf{x}},{\xi }^\omega ,y^\omega ) \end{aligned}$$

(3)

where $\min \limits _{y^\omega } Q(\textit{\textbf{x}},{\xi ^\omega },y^\omega )$ is the optimal value of power system operation over choices of second stage variables for a given scenario $\omega$, and $\Omega$ is a discrete approximation to the distribution of $\tilde{\xi }$ (Majidi-Qadikolai and Baldick 2016a). The extensive form of the two-stage stochastic TEP can be written as follows:

$$\begin{aligned} Z^{*} {=}&\min \limits _{x,y^{\omega }} \left\{ \zeta ^\intercal \textit{\textbf{x}} + \sum \limits _{\Omega } P^\omega \left[ \sum \limits _{N_s}(\sum \limits _{N_b} q_k r^\omega _{k,c}){+} \sum \limits _{N_{wg}} \gamma _g CW^\omega _{g} {+}\sum \limits _{N_{g}} Co_g p^\omega _g \right] \right\} \end{aligned}$$

(4)

$$\begin{aligned} \mathrm {st.} \quad&{-}\sum \limits _{L_k} f^\omega _{l,c}{+} \sum \limits _{G_k}p^\omega _{g}{+}r^\omega _{k,c}{=}d^\omega _k \end{aligned}$$

(5)

$$\begin{aligned}&{-}{M}_l(1-C_{l,c} x_l)\le f^\omega _{l,c}{-}B_{l,l} \Delta \theta ^\omega _{l,c} \end{aligned}$$

(6)

$$\begin{aligned}&{M}_l(1-C_{l,c} x_l)\ge f^\omega _{l,c}{-}B_{l,l} \Delta \theta ^\omega _{l,c} \end{aligned}$$

(7)

$$\begin{aligned}&CW^\omega _{g}\ge (P_g^{max,\omega }-p^\omega _{g}) \end{aligned}$$

(8)

$$\begin{aligned}&(C_{l,c} x_l)f_{l}^{min} \le f^\omega _{l,c} \le f_{l}^{max}(C_{l,c} x_l)\end{aligned}$$

(9)

$$\begin{aligned}&P_g^{min}\le p^\omega _{g}\le P_g^{max} \end{aligned}$$

(10)

$$\begin{aligned}&0\le r^\omega _{k,c} \le d_k \end{aligned}$$

(11)

$$\begin{aligned}&{-}\frac{\pi }{2} \le \theta ^\omega _{k,c} \le \frac{\pi }{2} \end{aligned}$$

(12)

$$\begin{aligned}&CW^\omega _{g}\ge 0 \end{aligned}$$

(13)

$$\begin{aligned}&x_l{=}1, \quad \forall l \in N_o \end{aligned}$$

(14)

$$\begin{aligned}&x_l \in \{0,1\}, \quad \forall l\in N_l \end{aligned}$$

(15)

In  (4), $y^\omega$ is the second stage decision variables vector that includes power generation ($p^\omega _g$), load shedding ($r^\omega _{k,c}$), wind curtailments ($CW^\omega _g$), branch flows ($f^\omega _{l,c}$), and voltage angles ($\theta ^\omega _{k,c}$) for all scenarios and all operation states. The formulation minimizes the objective function over all first stage (x) and second stage ($y^\omega$) decision variables and is constrained by (5)–(15). Equation (5) enforces power balance at each bus. Equations (6) and (7) represent power flow in transmission lines using the big-M technique. Equation (8) measures wind curtailment at each bus. Equation (9) shows flow ($f^\omega _{l,c}$) in branches should be between maximum and minimum capacity limits. Equations (10)–(12) enforce power plants’ dispatch $p^\omega _g$, load shedding $r^\omega _{k,c}$, and voltage angles $\theta ^\omega _{k,c}$, respectively, to be between their minimum and maximum limits. Equation (13) enforces nonnegativity of wind curtailment. Equation (14) sets decision variables for existing lines to 1. Equation (15) enforces that $x_l$ is a binary decision variable for transmission lines ($x_l=1$ when line l is built and $x_l=0$ when line l is not built).

Depending on the size of the network and the number of scenarios, solving the extensive form of problem (1) can be extremely computationally expensive. Therefore, decomposition techniques are used to find a near-optimal answer for large-scale problems.

3.2 Robust Optimization TEP Formulation

Robust optimization is a technique for modeling uncertainties and finding reliable solutions for the worst-case scenario. As discussed in Sect. 2.3, adaptive robust optimization can be used to adjust the level of robustness. Jabr (2013) and Ruiz and Conejo (2015) used this technique for TEP studies. Robust TEP can be formulated as three-level optimization problem as follows:

$$\begin{aligned} Z^{*}{=}&\min \limits _{x} \{ \zeta ^\intercal \textit{\textbf{x}} {+} \max \limits _{\mathscr {\xi } \in \mathscr {D}} \; [\min \limits _{y \in \Xi } {Q(\textit{\textbf{x}},{\xi }, y)}] \} \end{aligned}$$

(16)

$$\begin{aligned} \mathrm {st.} \quad&\textit{\textbf{x}}\in \{0,1\}^{|N_l|} \end{aligned}$$

(17)

In objective function (16), in the first level, the best transmission expansion plan ($\textit{\textbf{x}}$) is selected by minimizing the total system cost. In the second level, a realization of uncertain variables $\varvec{\xi }$ is selected from uncertainty set $\mathscr {D}$ that maximizes system operation costs ($Q(\textit{\textbf{x}},{\xi }, y)$) to represent the worst-case scenario. In the third level, based on selected x and $\varvec{\xi }$ from the first and the second levels, system operator tries to find the best values for third-level decision variables y (from its feasible set $\Xi$) to minimize system operation cost.

The result of robust optimization-based TEP is sensitive to uncertainty set definition; therefore, as stated in (Ruiz and Conejo 2015), having a careful definition of uncertainty set $\mathscr {D}$ is critical for an effective representation of uncertainties. A polyhedral uncertainty set is common to represent load and generation uncertainties. It can be described using the following constraints:

$$\begin{aligned} \xi _k \in [\xi _k^{min}, \xi _k^{max}] \end{aligned}$$

(18)

$$\begin{aligned} \frac{\sum |\xi _k^{ref}-\xi _k|}{\sum |\xi _k^{max}-\xi _k^{min}|} \le UB_a \end{aligned}$$

(19)

Equation (18) shows each uncertain parameter (load or generation here) may change between a minimum and a maximum value. Equation (19) is added to the robust optimization formulation to control the level of robustness (adaptive robust optimization). It is usually defined at regional/area level to mitigate the worst-case scenario. For example, it is less likely that outputs of all wind farms located at the same region face $100\%$ deviation from their reference value at the same time. In this equation, $\xi _k^{ref}$ is a reference point to measure divisions ($\xi _k^{min}\le \xi _k^{ref} \le \xi _k^{max}$), and $UB_a$ is uncertainty budget limit that can have a value between 0 and 1 ($0\le UB_a\le 1$).

The extended form of robust optimization formulation formulation can be written as follows (Ruiz and Conejo 2015):

$$\begin{aligned} Z^{*}{=}&\min \limits _x \left\{ \zeta ^\intercal \textit{\textbf{x}} {+} \max \limits _\xi \; \left[ \min \limits _y \sum \limits _{N_s}(\sum \limits _{N_b} q_k r_{k,c}){+} \sum \limits _{N_{wg}} \gamma _g CW_{g} {+}\sum \limits _{N_{g}} Co_g p_g \right] \right\} \end{aligned}$$

(20)

$$\begin{aligned} \mathrm {st.} \quad&{-}\sum \limits _{L_k} f_{l,c}{+} \sum \limits _{G_k}p_{g}{+}r_{k,c}{=}d_k \end{aligned}$$

(21)

$$\begin{aligned}&{-}{M}_l(1-C_{l,c} x_l)\le f_{l,c}{-}B_{l,l} \Delta \theta _{l,c} \end{aligned}$$

(22)

$$\begin{aligned}&{M}_l(1-C_{l,c} x_l)\ge f_{l,c}{-}B_{l,l} \Delta \theta _{l,c} \end{aligned}$$

(23)

$$\begin{aligned}&CW_{g}\ge (P_g^{max}-p_{g}) \end{aligned}$$

(24)

$$\begin{aligned}&(C_{l,c} x_l)f_{l}^{min} \le f_{l,c} \le f_{l}^{max}(C_{l,c} x_l)\end{aligned}$$

(25)

$$\begin{aligned}&P_g^{min}\le p_{g}\le P_g^{max} \end{aligned}$$

(26)

$$\begin{aligned}&0\le r_{k,c} \le d_k \end{aligned}$$

(27)

$$\begin{aligned}&{-}\frac{\pi }{2} \le \theta _{k,c} \le \frac{\pi }{2} \end{aligned}$$

(28)

$$\begin{aligned}&CW_{g}\ge 0 \end{aligned}$$

(29)

$$\begin{aligned} \mathrm {st.} \quad&0 \le P_g^{max} \le \hat{P_g^{max}} \end{aligned}$$

(30)

$$\begin{aligned}&d_k^{min} \le d_k \le d_k^{max} \end{aligned}$$

(31)

$$\begin{aligned}&\frac{\sum (\hat{P_g^{max}}-P_g^{max})}{\sum \hat{P_g^{max}}} \le UB^G_a \end{aligned}$$

(32)

$$\begin{aligned}&\frac{\sum ({d_k}-d_k^{min})}{\sum {d_k^{max}-d_k^{min}}} \le UB^D_a \end{aligned}$$

(33)

$$\begin{aligned} \mathrm {st.} \quad&x_l{=}1, \quad \forall l \in N_o \end{aligned}$$

(34)

$$\begin{aligned}&x_l \in \{0,1\}, \quad \forall l\in N_l \end{aligned}$$

(35)

In the objective function (20), y is the third-level decision variables vector that includes power generation ($p_g$), load shedding ($r_{k,c}$), wind curtailments ($CW_g$), branch flows ($f_{l,c}$), and voltage angles ($\theta _{k,c}$) for all scenarios and all operation states. Constraints (21)–(29) form the feasible region $\Xi$. The second level decision variable $\xi$ and includes the worst realization of demand ($d_k^{max}$) and generation ($\hat{P_g^{max}}$). Constraints (30)–(33) form the feasible region for uncertain variables ($\mathscr {D}$). Equations (30) and (31) limit minimum and maximum generation and load deviation at each bus, respectively. Equations (32) and (33) are uncertainty budget limits for generation and load at area a. Equations (34) and (35) limit values of the first level decision variable (x) to be 0 or 1 and set their value equal to 1 for all existing branches.

Decomposition-based formulations for the robust optimization TEP are developed in (Jabr 2013; Ruiz and Conejo 2015).

3.3 Constraint Filtering and Optimization Problem Size Reduction

Constraints define the feasible region of an optimization problem. In many cases, only a small subset of modeled constraints contribute in forming the final feasible region, and others can be removed from optimization problem without affecting the final optimal result. The key issue is finding which constraints can be removed. The following simple linear programming example with two variables is used for illustration purpose.

$$\begin{aligned} Z=&\min \limits _{y_1,y_2} \; 2y_1+5 y_2 \end{aligned}$$

(36)

$$\begin{aligned} \mathrm {st.} \quad&y_1 +2 y_2 \le 6 \end{aligned}$$

(37)

$$\begin{aligned}&y_1-y_2 \le 0 \end{aligned}$$

(38)

$$\begin{aligned}&y_1 \le 3 \end{aligned}$$

(39)

$$\begin{aligned}&y_2 \le 5 \end{aligned}$$

(40)

$$\begin{aligned}&y_1 \ge 0 \end{aligned}$$

(41)

$$\begin{aligned}&y_2 \ge 0 \end{aligned}$$

(42)

Constraints (37)–(42) limit the choice of $y_1$ and $y_2$ values by defining the feasible region for these two variables. These constraints and the formed feasible region are shown in Fig. 1. Lines C1 to C6 represent constraints (37)–(42), respectively, and the yellow triangle demonstrate the feasible region. The optimal solution is shown as the bullet. For this optimization problem, C3 and C4 do not contribute in forming the feasible region; therefore, removing them will reduce the problem size without affecting the optimal solution.

Fig. 1

Constraints and the feasible region

In power system operation, most of the constraints are not necessary for forming the feasible region. For example in ERCOT, there were only about 400 contingency constraints (out of tens of millions of possible constraints) that were binding at some time during 2013 (Potomac Economics 2014). Usually during very low load/low wind periods, a single outage of any line will not cause overload on other lines in most power systems. In other words, constraints related to those contingencies will be dominated by other constraints in the optimization problem and will not affect the feasible region and the optimal answer. Therefore, for this particular case we can ignore contingencies and solve OPF instead of SCOPF. As constraints related to contingencies are dominated, results of OPF will be feasible for SCOPF as well. Although eliminating passive constraints can significantly reduce problem size, finding all active constraints forming the feasible region is challenging.

Ardakani and Bouffard (2013) developed a technique called umbrella constraint identification to find all necessary and sufficient constraints for DC-SCOPF formulation. Abiri-Jahromi and Bouffard (2017a, b) developed loadability set to find necessary constraints for minimal representation of the feasible region for SCOPF by projecting demand-generation-network spaces onto the demand space only. Madani et al. (2017) have found a minimal subset of security constraints for a general SCUC formulation that guarantees the satisfaction of all security constraints. This formulation does not depend on commitment decision for generators and can handle load and generation forecast errors. Majidi-Qadikolai and Baldick (2016a, b) developed heuristic algorithms for SCOPF contingency constraint reduction. This method does not guarantee to find the minimal subset, but it can significantly decrease the problem size and it is computationally very cheap, and it can be used for both deterministic and stochastic formulations.

3.4 Decomposition Techniques

Solving the extensive form of a two-stage stochastic TEP optimization problem for large-scale networks is not practically feasible; therefore, Horizontal or Vertical decomposition techniques or both can be used to decompose the original problem for large systems. These techniques are discussed in this section.

3.4.1 Vertical Decomposition

Benders decomposition (BD) is one of the widely used vertical decomposition technique for solving two-stage stochastic TEP (Benders 1962). It divides the original problem into two parts, i.e., master and subproblem and uses “cuts” from dual of the subproblem to model its constraints in the master problem (Granville et al. 1988). References Granville et al. (1988), Park and Baldick (2013), Guo Chen et al. (2012), Zhang et al. (2015), Akbari et al. (2011), Munoz et al. (2014), Khodaei et al. (2010) applied BD to solve TEP optimization problem.

Although in several papers it is claimed that BD is easily scalable (for TEP) and can be used for real-size problems, Munoz et al. (2014) showed that even for medium size networks when the number of scenarios is large (50 or more), an optimality gap between $3\%$ to $6\%$ would need to be accepted in the BD algorithm to get the result in a reasonable time. For large-scale problems, the subproblem itself will be hard to solve, and a large number of iterations between master and subproblem is required to meet optimality gap requirements. This drawback worsens when reliability constraints are added to the TEP problem, in which subproblems should be solved for normal and under contingency operation states for all scenarios.

The column-and-constraint generation method (also called cutting-plane method) is another vertical decomposition technique that can be used to decompose a two-stage problem. In this method, primal “cuts” are used to represent the subproblem constraints in the master problem instead of dual cuts used by BD. Convergence guarantees and other properties of this method are explained in (Jiang et al. 2013; Zeng and Zhao 2013). Jabr (2013) and Ruiz and Conejo (2015) used BD and cutting-plane decomposition techniques, respectively, for solving robust TEP.

The following generic two-stage stochastic linear program is used to explain mathematical formulation for BD algorithm.

$$\begin{aligned} SLP{=}&\min \limits _{x,y} \; cx + \sum \limits _{\omega \in \Omega } p^{\omega } f^{\omega } y^{\omega } \end{aligned}$$

(43)

$$\begin{aligned} \mathrm {st.} \quad&Ax \quad =b \end{aligned}$$

(44)

$$\begin{aligned}&-B^{\omega }x+D^{\omega }y^{\omega }=d^{\omega }, \quad \forall \omega \in \Omega \end{aligned}$$

(45)

$$\begin{aligned}&x\in \mathscr {X}, \; y^{\omega } \ge 0, \quad \forall \omega \in \Omega \end{aligned}$$

(46)

Fig. 2

Standard Benders decomposition algorithm

The BD algorithm decomposes the SLP into two problems, i.e., master problem and subproblem, and solves them iteratively. The master problem includes first stage decision variable/constraints and a relaxed version of the second stage constraints.

$$\begin{aligned} Master=&\min \limits _{x,\theta } \; cx +\phi \end{aligned}$$

(47)

$$\begin{aligned} \mathrm {st.} \quad&Ax \quad =b \end{aligned}$$

(48)

$$\begin{aligned}&-G^ix+\phi \ge g^i, \quad i=1,\dots , l \end{aligned}$$

(49)

$$\begin{aligned}&x\in \mathscr {X} \end{aligned}$$

(50)

In the subproblem, at iteration i the first stage decision variable (x) is fixed, and the problem is solved for the second stage decision variable ($y^{\omega }$).

$$\begin{aligned} Subproblem =&\min \sum \limits _{\omega \in \Omega } p^{\omega } f^{\omega } y^{\omega } \end{aligned}$$

(51)

$$\begin{aligned} \mathrm {st.} \quad&D^{\omega }y^{\omega }=d^{\omega }+B^{\omega }x \;: \; \pi ^{\omega }, \quad \forall \omega \in \Omega \end{aligned}$$

(52)

$$\begin{aligned}&y^{\omega } \ge 0, \quad \forall \omega \in \Omega \end{aligned}$$

(53)

After solving the subproblem and assuming it is feasible, coefficients in (54) and (55) are calculated. These coefficients are used to form optimality cuts (equation (49)) that will be sent to the master problem for the next iteration. The standard BD algorithm is summarized in Fig. 2.

$$\begin{aligned} G^i=\sum \limits _{\omega \in \Omega } p^{\omega } \pi ^{\omega } B^{\omega } \end{aligned}$$

(54)

$$\begin{aligned} g^i=\sum \limits _{\omega \in \Omega } p^{\omega } \pi ^{\omega } d^{\omega } \end{aligned}$$

(55)

For more details and other forms of BD algorithms, please see reference Conejo et al. (2006).

3.4.2 Horizontal Decomposition

Progressive Hedging (PH) is aimed at decomposing a two-stage stochastic resource allocation problem horizontally by solving the problem for each scenario separately and adding non-anticipativity constraints to couple the first stage decision variables (standard PH) (Rockafellar and Wets 1991). The PH method for mixed-integer problems is a heuristic method that finds an upper bound answer for the non-convex optimization problem; however, Gade et al. (2016) developed a method to also calculate a lower bound for results of the PH algorithm in order to quantify the quality of results. One drawback of standard PH algorithm is that for problems with a large number of scenarios and integer variables, it may need a large number of iterations to satisfy non-anticipativity constraints (and sometimes it may never converge if no heuristic action is taken inside the algorithm).

For the standard PH algorithm, the TEP problem (1) can be rewritten as the following so-called scenario formulation:

$$\begin{aligned} Standard \, \textit{PH}{=}&\min \limits _{x,y} \; \sum \limits _{\omega \in \Omega } p^{\omega } (c\textit{\textbf{x}}^{\omega }+ f^{\omega } y^{\omega }) \end{aligned}$$

(56)

$$\begin{aligned} \mathrm {st.} \quad&A\textit{\textbf{x}}^{\omega } \quad =b, \quad \forall \omega \in \Omega \end{aligned}$$

(57)

$$\begin{aligned}&-B^{\omega }\textit{\textbf{x}}^{\omega }+D^{\omega }y^{\omega }=d^{\omega }, \quad \forall \omega \in \Omega \end{aligned}$$

(58)

$$\begin{aligned}&\textit{\textbf{x}}^{\omega }\ge 0, \; y^{\omega } \ge 0, \quad \forall \omega \in \Omega \end{aligned}$$

(59)

$$\begin{aligned}&\textit{\textbf{x}}^1=\dots =\textit{\textbf{x}}^s \end{aligned}$$

(60)

A copy of decision variable vector $\textit{\textbf{x}}^\omega$ is created for each scenario $\omega$ in $\Omega$ that allows solution of the TEP problem for each scenario independently, and non-anticipativity constraints (60) are added to couple first stage solutions and guarantee that the final expansion plan does not depend on scenarios.

Instead of decomposing the problem for each individual scenario, it is possible to use bundles of scenarios ($\mathscr {B}=\{\mathscr {B}_1, \dots , \mathscr {B}_b \}$) for decomposition. Equations (56)–(60) can be rewritten for bundled PH as follows:

$$\begin{aligned} Bundled \, PH{=}&\min \limits _{x,y} \sum \limits _{\mathscr {B}}[p^{\mathscr {B}_i} (c \textit{\textbf{x}}^{\mathscr {B}_i}) {+} \sum \limits _{\mathscr {B}_i} Pu^{\omega } f^{\omega } y^{\omega }] \end{aligned}$$

(61)

$$\begin{aligned} \mathrm {st.} \quad&A\textit{\textbf{x}}^{\mathscr {B}_i} \quad =b, \quad \forall {\mathscr {B}_i}\in {\mathscr {B}} \end{aligned}$$

(62)

$$\begin{aligned}&-B^{\omega }\textit{\textbf{x}}^{\mathscr {B}_i}+D^{\omega }y^{\omega }=d^{\omega }, \quad \forall {\mathscr {B}_i}\in {\mathscr {B}},\forall \omega \in \Omega \end{aligned}$$

(63)

$$\begin{aligned}&\textit{\textbf{x}}^{\mathscr {B}_i}\ge 0, \; y^{\omega } \ge 0, \quad \forall {\mathscr {B}_i}\in {\mathscr {B}},\forall \omega \in \Omega \end{aligned}$$

(64)

$$\begin{aligned}&\textit{\textbf{x}}^{\mathscr {B}_1}=\dots =\textit{\textbf{x}}^{\mathscr {B}_b} \end{aligned}$$

(65)

In this case, a copy of decision variable vector $\textit{\textbf{x}}^{\mathscr {B}_i}$ is created for all ${\mathscr {B}_i}$s in ${\mathscr {B}}$. Non-anticipativity constraints (65) are explicitly modeled for scenario bundles, and they are implicitly modeled for scenarios within each bundle ($\kappa$ scenarios in each bundle already have the same first stage decision variable $\textit{\textbf{x}}^{\mathscr {B}_i}$). Therefore, a bundled PH will have fewer non-anticipativity constraints compared to a standard PH ($|\mathscr {B}|<|\Omega |$), which usually reduces the number of iterations for convergence.

Fig. 3

Progressive hedging algorithm with bundled scenarios

Through an iterative process, PH will converge to a unique answer for the first stage decision variables by appropriately penalizing deviations of non-anticipative variables from their mean values. The PH algorithm with bundled scenarios is shown in Fig. 3. In the first line, the initial value of the iteration counter ($\upsilon$) and multiplier vector ($\textit{\textbf{W}}^\upsilon _{\mathscr {B}_i}$) is set. From line 2–4, the TEP optimization problem for each bundle is solved separately (and can be parallelized). In line 5, the weighted sum of individual expansion plans ($\textit{\textbf{x}}^{{\mathscr {B}_i, \upsilon }}$s) is calculated. Line 6 calculates the deviation (Err) from averaged expansion plan ($\varvec{\hat{x}}^\upsilon$). Lines 7–15 cover the main iterative part of the bundled PH algorithm. In line 8, the value of counter is updated. Line 9 updates the value of multiplier vector by using penalty vector $\varvec{\rho }$. Lines 10–12 solve an updated TEP formulation with multiplier and penalizing deviation from average value of first stage decision variables. This optimization problem is solved for each bundle independently, so they can be solved in parallel. Lines 13 and 14 update the calculated average value for $\textit{\textbf{x}}$ and Err, respectively.

Stochastic unit commitment (Ryan et al. 2013), and transmission planning (Munoz and Watson 2015; Majidi-Qadikolai and Baldick 2018) are examples of PH algorithm application in power system. Crainic et al. (2014) used PH for commodity network design, and in (Escudero et al. 2012), PH algorithm is used for solving multi-stage stochastic mixed-integer problems.

3.4.3 Hybrid Decomposition

Hybrid decomposition uses both horizontal and vertical decomposition techniques to solve a large-scale stochastic optimization problem (Majidi-Qadikolai and Baldick 2018). It applies PH decomposition to horizontally decompose the original problem first, and then BD is used to vertically decompose each subproblem.

In PH algorithm (Fig. 3), extensive form of the problem is solved in lines 3 and 11. However, for very large-scale problems, solving the extensive form of these subproblems can also be computationally expensive. In the hybrid method, optimization subproblems in lines 3 and 11 of Fig. 3 will be solved using the BD algorithm. It divides the original problem into smaller subproblems to keep the original problem computationally tractable, and both PH and BD simulations can be distributed between multiple machines and be solved in parallel (see Sects. 4.3.5 and 4.3.6 for more discussion).

4 A Generalized Framework for Stochastic TEP Studies

A generalized decomposition framework for solving stochastic TEP studies for networks with different sizes, proposed in (Majidi-Qadikolai and Baldick 2018), is reviewed in this section. This framework is scalable, configurable, and easily maintainable.

4.1 Framework Overview

The framework is designed to be flexible and configurable for different problem sizes on different machines. It can be configured to solve a problem in extensive form (EF), or using PH, BD, and hybrid techniques (by setting its parameters) that provides more flexibility from the modeling perspective. The proposed framework can be summarized as follows:  

Phase 0::

Data preparation  

Step 1::

Input data and setting parameters

Input data includes the base network, scenarios, and candidate lines list. In this step, the planner configures the framework by setting its parameters; i.e. the number of scenarios in each bundle ($\kappa$) and the type of decomposition technique that should be used (PH, BD or Hybrid) for phases I and II. Settings for phase II can be modified later in step 4 if it is necessary.

 

Phase I::

TEP without contingency analysis  

Step 2::

Scenario bundling

In this step, OPF for the base (existing) network is solved and calculated load shedding and wind curtailment will be used to develop an attribute for scenario bundling. After developing appropriate criteria, bundles of scenarios are formed (see subsection 4.2).

Step 3::

Solving TEP

In this step, based on inputs from step 1 and bundles from step 2, TEP for normal operation states is solved. This step can be parallelized.

 

Phase II::

TEP with contingency analysis

This phase is run if contingency analysis should be integrated in the TEP process.  

Step 4::

Scenario Bundling

Based on parameter settings, the scenario bundling method can be used to bundle scenarios.

Step 5::

Solving TEP with contingency analysis

In this step, TEP with contingency analysis is solved. Either PH, BD, or hybrid may be used for solving this large-scale optimization problem. This step can be parallelized if PH and/or BD are selected as the solving algorithm. The contingency constraint reduction technique developed in (Majidi-Qadikolai and Baldick 2016a, b), can be used for solving TEP for each subproblem in this step.

 

Phase III::

Quantifying the quality of results

If PH or hybrid is selected for phase I and/or II, then it will be necessary to find optimality gap to quantify the quality of results.  

Step 6::

Calculating a lower bound answer

In this step, the proposed lower bound formulation for PH in (Gade et al. 2016) is used to calculate a lower bound.

Step 7::

Calculate optimality gap The optimality gap ($\varepsilon$) can be calculated using the upper bound from step 5 (or step 3 in case of TEP without contingency analysis) and the lower bound from step 6. The selected plan is $\varepsilon -suboptimal$.

 

  The framework is summarized in the flowchart in Fig. 4.

Fig. 4

Flowchart of the generalized framework

4.2 Scenario Bundling

The main purpose of scenario bundling is to create heterogeneous groups of scenarios with minimum dissimilarity between the groups collectively (based on selected attributes/criteria) and with relatively the same computational burden. Having similar bundles will improve the performance of PH algorithm by facilitating convergence of non-anticipativity constraints, as for a set of identical groups of scenarios, PH only needs one iteration to converge (although the choice of bundling does not necessarily reduce computational time). In contrast to clustering in which the objective is to minimize dissimilarity within groups (by forming homogeneous groups), scenario bundling tries to minimize dissimilarity between groups (see Majidi-Qadikolai and Baldick (2018) for mathematical formulation). As finding such a grouping can be computationally expensive, Majidi-Qadikolai and Baldick (2018) developed a heuristic method to solve this problem faster. This method bundles scenarios through three steps, i.e., classification, clustering, and grouping into bundles. The clustering step divides scenarios into multiple classes based on defined criteria (it is computationally very cheap). As scenarios in each class are clustered separately, the computational time for the clustering step is reduced. The grouping step allows integration of group level bundling criteria while forming heterogeneous bundles. These steps are explained in more detail in the following subsections. It should be noted that scenario bundling is required only if $1< \kappa < |\Omega |$, where $\kappa$ is the size of each bundle, $\Omega$ is the set of all scenarios, and $|\Omega |$ represents the size of this set.

4.2.1 Classification

In classification, a model or classifier is constructed to predict class labels such as, for example, “safe” or “risky” for bank loan application, or “light” and “heavy” loading conditions for electric networks. There are different classification methods such as decision tree induction, Bayes classification methods, and rule-based classification (Han and Kamber 2011). The rule-based method is used here, because its structure allows us to easily integrate expert knowledge into the bundling process. It has the following structure:

$$\begin{aligned} \mathbf{IF} \quad Condition \quad \mathbf{THEN} \quad Conclusion \end{aligned}$$

(66)

For our banking example, it can be written as

$$\begin{aligned} \mathbf{IF} \quad age\le 25 \;AND\; student \quad \mathbf{THEN} \quad Safe \end{aligned}$$

For electric network example, we can have

$$\begin{aligned} \mathbf{IF} \; average\,line\,loading \ge 50\% \; \mathbf{THEN} \; Heavily \, loaded \, network \end{aligned}$$

Rule-based classification will partition the original scenario set $\Omega$ into a finite number of non-empty classes $\mathscr {I}=\{\mathscr {I}_1,\dots ,\mathscr {I}_q\}$.

Different classification rules can be defined depending on the purpose of a study. For numerical analysis in Sect. 5, the number of important lines for contingency analysis (ICLs) can be used as a classifier in step 4. It might be necessary to adjust the number of scenarios in classes (those that are close to boundaries) for feasibility of the clustering step. Classification is an optional part of the bundling process, and if there is no classifier, then there will be only one class that includes all scenarios ($\mathscr {I}=\{\mathscr {I}_1\}$).

4.2.2 Clustering

Clustering is the process of grouping a set of objects in a way that objects within a cluster have the highest similarity. In this step, scenarios in each class ($\mathscr {I}_i$) are clustered based on selected attribute/developed criteria, and form the set $\mathscr {S}^i=\{\mathscr {S}^i_1, \dots , \mathscr {S}^i_c \}$. Without loss of generality, scenarios are clustered in clusters with the same size, and the size of each cluster ($\mathscr {C}_s$) can be calculated from the following equation.

$$\begin{aligned} \mathscr {C}_s=\frac{|\Omega |}{\kappa } \end{aligned}$$

(67)

where we assume that $|\Omega |$ is divisible by $\kappa$.

It is important to choose an attribute/criteria that is appropriate for the purpose of the study and provides insight for grouping phase. For example, for TEP without contingency analysis (step 3 of the framework), load shedding and wind curtailment penalties are major factors driving transmission expansion plans as they will be curtailed only if there is not enough transmission capacity to transfer their output (for wind) and/or supply them (for demand). Therefore, a weighted sum of load shedding and wind curtailment (LW) can be defined as a clustering attribute for this step. For phase II of the framework, TEP with contingency analysis is solved in step 5. As contingencies can have huge impact on selected transmission expansion plan (Majidi-Qadikolai and Baldick 2016b), important contingency list can be used to form an attribute for scenario clustering in this step.

Partitioning method is used to create clusters based on defined attributes. The objective of this clustering optimization problem is to minimize the distance between different attributes of objects (scenarios here) in a cluster. For step 2, scenarios with closest LW values are clustered together, and for step 4, scenarios with highest similarity in their important contingency lists will be clustered together (see Majidi-Qadikolai and Baldick (2018) for mathematical formulation).

4.2.3 Grouping into Bundles

In the last step, members of each cluster are distributed between groups (bundles) with the objective of minimizing dissimilarity between groups (by forming heterogeneous bundles). For the scenario set $\Omega$, a bundle set $\mathscr {B}=\{\mathscr {B}_1, \dots , \mathscr {B}_b \}$ of non-empty and mutually exclusive subsets ($\forall i\ne j, \; \mathscr {B}_i \bigcap \mathscr {B}_j=\emptyset$ and $\bigcup _j \mathscr {B}_j=\Omega$) is formed.

Scenarios in each cluster share similar characteristics (attributes used for classification and clustering). Therefore, one can form bundles of heterogeneous scenarios by randomly distributing members of each cluster between bundles. It is also possible to define new criteria for grouping in this step.

For example, for phase I of the framework, scenarios can be distributed between groups with the objective of minimizing the distance between aggregated LW values ($LW_{\mathscr {B}_i}$) between groups (bundles) because this attribute has a major impact on the TEP in step 3. For step 4 of the framework, total number of operational states ($N_s$) in each bundle can be used as a grouping attribute because it has a huge impact on computational time requirement for each bundle, and forming bundles with relatively the same computational burden will improve the performance of parallelizing in PH algorithm (see Sect. 5 for numerical results).

As a separate stochastic TEP is solved for each bundle in PH algorithm, the probability of each scenario should be updated based on Equations (68) and (69):

$$\begin{aligned}&P_{\mathscr {B}_i}=\sum _{\omega \in \mathscr {B}_i} P^\omega \quad \forall \mathscr {B}_i \in \mathscr {B} \end{aligned}$$

(68)

$$\begin{aligned}&Pu^\omega = \frac{P^{\omega }}{P_{\mathscr {B}_i}}\quad \forall \omega \in \mathscr {B}_i, \forall \mathscr {B}_i \in \mathscr {B} \end{aligned}$$

(69)

$$\begin{aligned}&|\Omega |=\sum _{\mathscr {B}_i \in \mathscr {B}}|\mathscr {B}_i| \end{aligned}$$

(70)

$$\begin{aligned}&\sum _{\mathscr {B}_i \in \mathscr {B}} P_{\mathscr {B}_i} =1 \end{aligned}$$

(71)

where $P^\omega$ is the original probability of scenario $\omega$, $P_{\mathscr {B}_i}$ is probability of bundle $\mathscr {B}_i$ in set of bundles $\mathscr {B}$, and $Pu^\omega$ is updated probability of scenario $\omega$ as a member of bundle $\mathscr {B}_i$. Equations (70) and (71) enforce scenario bundling to be mutually exclusive.

4.3 Model Performance Discussion

In this section, different factors affecting the performance of the framework are investigated.

4.3.1 Parameter Settings for the Framework

The size of each bundle ($\kappa$) and the choice of a decomposition method are set in step 1 in the framework (see Sect. 4.1). Table 1 shows different possible combinations for setting these two parameters. For the PH algorithm, by setting $\kappa =1$ a standard PH is solved, $1<\kappa <|\Omega |$ will result in a bundled PH, and $\kappa =|\Omega |$ is equivalent to solving the extensive form (EF) of the optimization problem. If BD is selected as the solving method, then for $1\le \kappa <|\Omega |$, the problem is solved separately for each bundle, and a heuristic method should be used to select a unique first stage answer. For $\kappa =|\Omega |$, a standard BD is solved. When hybrid method is selected, for $1\le \kappa <|\Omega |$, both PH and BD are used for solving the problem in steps 3 and/or 5 in the framework. For $\kappa =|\Omega |$, hybrid method will be the same as BD method. These parameters can be set independently for phases I and II providing more flexibility, potentially improving the effectiveness of the framework.

Table 1 Different parameter settings for the framework

4.3.2 Factors Affecting the Choice of Parameters

The size of the problem, the design of decomposition algorithms, existing hardware infrastructure, and solvers are critical for making a decision about setting parameters for the framework. These factors are briefly overviewed in the following.

• The size of the problem (d)

  The number of structural constraints (SC), Equations (5)–(8), continuous (CV) and binary (BV) decision variables are main factors for the size of the TEP optimization problem. For the extensive form of this TEP formulation from Sect. 3.1 (depending on the choice and design of decomposition algorithms, new variables and constraints may be added), these values can be calculated from the following equations:

  $$\begin{aligned}&d=\{SC, CV, BV\}\end{aligned}$$

  (72)
  $$\begin{aligned}&SC= (2\times (|N_b|+|N_l|)\times |N_s^\omega |+|N_{wg}|)\times |\Omega | \end{aligned}$$

  (73)
  $$\begin{aligned}&CV= ((2\times |N_b|+|N_l|)\times |N_s^\omega |+|N_g|+|N_{wg}|) \times |\Omega | \end{aligned}$$

  (74)
  $$\begin{aligned}&BV= |N_n| \end{aligned}$$

  (75)

  If no contingency constraint reduction technique is used, then $|N_s^\omega |=|N_l|+1$ to model outage of each branch.

• Design of decomposition algorithms

  PH and BD are not black-box software packages with input and output vectors. These algorithms are designed based on specific needs and conditions. For BD, there are several different designs such as standard BD (Benders 1962), multi-cuts BD (Birge and Louveaux 1988), and nested BD (Roger Glassey 1973), and each design can be configured differently. For PH, either the standard form (Rockafellar and Wets 1991) or the bundled form (Wets 1989) might be used. Similar to BD, there are several internal settings for PH that can affect the performance of this algorithm.

• Existing hardware infrastructure

  The machine that is used to solve the TEP problem has an undeniable impact on the choice of a decomposition algorithm and the size of each bundle ($\kappa$). Machines with high computing power are usually capable of solving larger problems that make it possible to choose bundled PH with a large bundle size ($\kappa$). In the case of using multiple machines (or virtual machines for Cloud-based workstations), implemented parallel computation structure will be another key factor.

• Solvers

  The main feature of a solver that affects the choice of parameters for the framework is its capability to distribute computation burden over multiple cores of a CPU and use all computing power of the machine. GUROBI and CPLEX are examples of commercial solvers with this capability.

As discussed above, there are several factors that can affect hardware and software design of this framework. For a designed framework, running a few individual simulations can provide a relatively good insight about the performance of each module, and help on setting parameters for the framework.

4.3.3 PH Performance Improvement

Several heuristics such as finding appropriate values for $\varvec{\rho }$, variable freezing, cyclic behavior detection, and terminating PH when the number of remaining unconverged variables is small can be used to improve the performance of the PH algorithm (Watson and Woodruff 2011). In the following, some of these heuristic methods are reviewed in detail.

• Choice of $\varvec{\rho }$: A good approximation for $\varvec{\rho }$ is important for the PH algorithm to perform well. As shown in Fig. 3, the value of multiplier vector ($\textit{\textbf{W}}^\upsilon _{\mathscr {B}_i}$) is updated using penalty vector $\varvec{\rho }$, and an appropriate multiplier vector can affect the number of required iterations for PH convergence, and the quality of the lower bound answer (Gade et al. 2016). In Watson and Woodruff (2011), different heuristic methods for calculating effective values for $\varvec{\rho }$ are proposed. Our experience with those methods shows that for the TEP problem using the following equation from Watson and Woodruff (2011) results in a better convergence rate.

  $$\begin{aligned} \rho _l=\frac{\zeta _l}{x_l^{max}-x_l^{min}+1} \end{aligned}$$

  (76)

  where $\rho _l$ is the $l^{th}$ element of vector $\varvec{\rho }$, and

  $$\begin{aligned} x_l^{max}=\max _{\mathscr {B}_i \in \mathscr {B}} x_l^{\mathscr {B}_i}\end{aligned}$$

  (77)
  $$\begin{aligned} x_l^{min}=\min _{\mathscr {B}_i \in \mathscr {B}} x_l^{\mathscr {B}_i} \end{aligned}$$

  (78)

  For values of ${\rho _l}$ close to the unit cost of its associated variable, the PH algorithm should have a better performance both from convergence speed and quality of results. Selecting higher values for ${\rho _l}$ will increase convergence rate but may negatively affect the quality of results. On the other hand, very small values for ${\rho _l}$ can improve the quality of results (by decreasing optimality gap), but can significantly increase the number of iterations and simulation time.

• Variable Freezing: To improve the convergence of PH algorithm, the variable freezing technique can be used. Based on this technique, first stage decision variables with values that did not change over the past $\vartheta$ iterations are frozen for future iterations. For example, for a case with 5 bundles and $\vartheta =4$, the value of the decision variable $x_l$ is frozen if for all 5 bundles during all 4 successive iterations $\upsilon +1, \upsilon +2, \upsilon +3, \upsilon +\vartheta =\upsilon +4$, its value did not change and was the same across all bundles ($x_l^{\upsilon +1,1}=\dots =x_l^{\upsilon +4,5}$). The impact of freezing variables can be investigated from two perspectives, namely simulation time and the selected plan.

  • Impact on simulation time

    By freezing binary variables, total number of binary decision variables is decreased as frozen variables have fixed values. It improves the performance of the algorithm by decreasing computational time for each iteration (as a TEP optimization problem with fewer binary variables will typically be solved faster) and reducing the number of iterations (as a PH problem with fewer non-anticipativity constraints will typically converge faster).

  • Impact on the selected plan

    When a decision variable is frozen, the implicit assumption is that its value will not change during subsequent iterations, but this assumption may not always be valid. Therefore, the selected plan might be negatively affected when variable freezing technique is used, especially for small values of $\vartheta$ like 1 or 2. By using more conservative values for $\vartheta$, this effect can be mitigated.

  The selected plan will be more sensitive to a small value for $\vartheta$ when there are several relatively similar candidate lines (in terms of cost and/or electric parameters) in a geographically limited area. For a large-scale network in which candidate lines are widely spread, a smaller value for $\vartheta$ can be selected. Using the variable freezing technique may result in situations with only a very few unfrozen decision variables. Then PH can be terminated (to decrease the number of iterations), and the TEP with remaining binary variables solved in the extensive form or using a BD algorithm.

• Identical Parallel Candidate Lines: We have also noticed that having two (or more) identical parallel candidate lines can result in an unnecessary nonzero values of Err on lines 6 and/or 14 in PH algorithm (Fig. 3) when only one of those lines is selected as a part of expansion plan. We recommend to slightly modify the investment cost for otherwise identical lines to break the symmetry.

4.3.4 Optimality Gap

The optimality gap is used as a measure for quantifying the quality of results in an optimization-based TEP. Based on Table 1, the TEP problem is solved using one of these five methods, i.e., heuristic, extensive form (EF), PH, BD, and hybrid. For parameter settings that will result in a heuristic method, the optimality gap cannot be calculated to quantify the quality of results. For the EF method, the optimality gap of the final result will be less than or equal to the solver’s setting for maximum optimality gap. For BD, achieving the optimality gap is set as the stopping criterion; therefore, for EF and BD methods, it is possible to guarantee a pre-defined optimality gap (assuming that the algorithm successfully terminates). On the other hand, for PH and hybrid methods, the optimality gap is calculated after the algorithm is terminated to quantify the quality of final results, and there is no guarantee that the final optimality gap will be less than or equal to a pre-defined threshold. As discussed in Sect. 4.3.3, using appropriate values for $\varvec{\rho }$ and setting a conservative value for $\vartheta$ can improve the optimality gap of the PH algorithm.

4.3.5 Scalability and Maintainability

Scalability is one of the main features of this framework. Figure 5a shows the size of the EF of a stochastic TEP problem with security constraints. In this Fig., $d^\omega$ represents the size of the TEP problem for scenario $\omega$ ($d^\omega =\{SC^\omega ,CV^\omega ,BV^\omega \}$).

$$\begin{aligned}&SC^\omega = 2\times (|N_b|+|N_l|)\times |N_s^\omega |+|N_{wg}| \end{aligned}$$

(79)

$$\begin{aligned}&CV^\omega = (2\times |N_b|+|N_l|)\times |N_s^\omega |+|N_g|+|N_{wg}| \end{aligned}$$

(80)

$$\begin{aligned}&BV^\omega = |N_n| \end{aligned}$$

(81)

For a sample case with 6000 buses, 8000 existing branches, 500 conventional power plants, 100 wind farms, 100 candidate lines, and 10 scenarios, the size of the problem is $d^\omega =\{228.5M,162.8M, 100\}$ when $|N_s^\omega |=8101$ and $s=10$ (M stands for million). Total size of the problem in Fig. 5a will be $d=\{2285M,1628M, 100\}$. This problem is practically impossible to solve in the EF. There are constraint reduction techniques (Madani et al. 2017; Majidi-Qadikolai and Baldick 2016a; Ardakani and Bouffard 2013) that can be used to decrease the size of this problem. Let us assume using the VCL algorithm (Majidi-Qadikolai and Baldick 2016b) reduces the size of $N_s^\omega$ form 8101 to 50. The size of the EF of this problem will be $d=\{14M,10M,100\}$. Even after a massive problem size reduction, solving the EF of the problem still remains computationally extremely expensive.

The BD algorithm (shown in Fig. 5b) moves binary decision variables to the master problem and keeps all continuous variables in the subproblem. As the subproblem is a linear program, it is expected to be solved very fast; however, for the network in this example, the size of the subproblem will be $\{14M,10M,0\}$ which is not easy to solve especially when it should be solved at every BD algorithm iteration.

Figure 5c shows how bundled PH algorithm will decompose the problem. By creating bundles of two scenarios, the size of each subproblem for bundled PH will be $\{2.8M,2.0M,100\}$ (or $\{1.4M,1.0M,100\}$ for standard PH). Solving the extensive form of these subproblems might still be hard because of the large number of binary variables. In Fig. 5d, the hybrid method is used to decompose the problem both vertically and horizontally. By using this method, the size of each problem that needs to be solved in EF can be decreased up to $\{1.4M,1.0M,0\}$, which is a significant size reduction compared to $\{14M,10M,100\}$ for Fig. 5a.

Fig. 5

Impact of different decomposition techniques, $d^\omega$: size of the problem for scenario $\omega$, s: the number of scenarios (6 for this example)

The size of this case study may increase either by increasing the number of candidate lines or the number of scenarios. The BD feature of the hybrid method will keep us away from exponentially increasing computational time as a result of adding new binary variables, and the bundled PH feature will keep the size of each subproblem relatively unchanged even if the total number of scenarios is increased significantly (by increasing the number of bundles instead of increasing the size of each bundle). Therefore, the problem remains tractable, demonstrating the scalability of the proposed framework.

Another important feature of this framework (from practicality perspective) is its maintainability. Because it is module-based (BD algorithm, PH algorithm, bundling algorithm), each module can easily and (relatively) independently be upgraded as technology improves.

4.3.6 Parallelizing

With proper hardware, parallelizing decreases computational time for solving a series of independent simulations and improves scalability. Simulations in steps 3 and 5 in the framework can be parallelized, if PH, BD (with special configurations), or hybrid is selected to reduce elapsed time for solving TEP optimization problem by starting all simulations at the same time.

• PH algorithm: Based on PH algorithm for bundled scenarios shown in Fig. 3, lines 3 and 11 are run for each bundle (or each scenario in case of standard PH) independently. Therefore, we can parallelize both for loops (lines 2–4 and 10–12) in this algorithm and start all simulations in each loop at the same time to decrease computational time. It should be noted that lines 10–12 should be solved for each iteration of the PH algorithm, and decreasing computational time here can be rewarding from the performance improvement perspective. As shown in lines 5 and 13 in Fig. 3, the algorithm can proceed to the next step when all parallelized simulations are completed. In the bundling process, bundles should be developed that need relatively similar computational time, so that the framework can benefit the most from parallelizing.

• BD algorithm: For standard BD, in which one cut is sent to the master problem in each iteration, the subproblem is usually solved in extensive form. For multi-cuts BD (Birge and Louveaux 1988) and nested BD (Roger Glassey 1973; Akbari et al. 2011; Khodaei et al. 2010), it is possible to solve subproblems in parallel that will decrease computational time.

• Hybrid method: As hybrid algorithm uses both PH and BD to solve a problem, it can benefit from both vertical and horizontal decompositions and parallelize the problem-solving with both algorithms (if applicable). For example, by using bundled PH, the problem will be horizontally parallelized for each bundle ${\mathscr {B}_i}$. A nested BD can be used to solve each bundle, in which feasibility cuts for under contingency operation states can be created in parallel.

5 Case Study and Numerical Results

In this section, numerical analysis for three case studies from (Majidi-Qadikolai and Baldick 2016a, b, 2018; Majidi-Qadikolai et al. 2018) are presented. All simulations are done with a personal computer with 2.0-GHz CPU and 32 GB of RAM. MATLAB R2014a, YALMIP R20150626 package (Lofberg 2004), and GUROBI 5.6 (Gurobi Optimization, Inc 2014) are used as programming language, modeling tool and a solver respectively. To calculate the elapsed “Simulation Time,” MATLAB built-in function tic toc is used. Steps 3 and 5 are parallelized using MATLAB built-in function parfor where PH is selected as a solving algorithm.

5.1 13-Bus Test System

This case study contains 13 buses, 33 existing lines, 16 power plants, 9 load centers, and 36 candidate lines with 100 scenarios to capture uncertainties in wind and load (Majidi-Qadikolai and Baldick 2016b) (shown in Fig. 6—See Appendix for details). A new line investment cost is assumed $\$ 1M/\text{mile }$, and load shedding is penalized at $\$9000/\text{MWh }$ and $\$500/\text{MWh }$ penalty for wind curtailment. This small case study with a large number of scenarios is used to demonstrate different steps of the framework. Table 2 shows developed case studies. The proposed method in (Majidi-Qadikolai and Baldick 2016a) is used for contingency constraint reduction for cases B–E.

Fig. 6

13-bus system

Table 2 Case study definition

In case A, the extensive form (EF) of two-stage stochastic TEP is solved without any constraint reduction. For case B, the proposed method in (Majidi-Qadikolai and Baldick 2016a) is used to reduce contingency constraints, and the EF of the reduced model is solved. Case C is a standard PH in which the size of bundles is set to 1. For case D, scenarios are bundled randomly with 5 bundles with size $\kappa =20$ using MATLAB built-in function randperm. For case E, scenarios are bundled using the bundling method from Sect. 4.2.

5.1.1 PH Algorithm Settings

Values for $\varvec{\rho }$ are calculated based on (76). Variables that are consistent across bundles and do not change over the most recent 4 iterations will be frozen at their values ($\vartheta =4$). Moreover, if the number of remaining binary variables is less than or equal to 3, the PH algorithm is terminated, and the extensive form of the problem is solved for remaining decision variables. These settings are applied to cases C–E.

5.1.2 Model Performance Discussion

The simulation result for these five cases is summarized in Table 3. For case A, we were unable to get any results after 12 days. It shows that solving the EF of TEP with all constraints is not practical even for this small case study. For case B, the TEP optimization problem is solved in 25 min with $2.7\%$ optimality gap. Standard PH in case C needs more than 2 hours to solve this problem, and the final result is $29.5\%$-suboptimal. It shows that the standard PH may not have a good performance when the number of scenarios is large. For Case D, bundling reduced computational time by $50\%$ and optimality gap is dropped to $1.65\%$. For case E, computational time is reduced to 15 minutes, and the quality of results is significantly improved by decreasing optimality gap to $0.24\%$. The selected settings for framework for case E solves this problem more than 8 times faster than standard PH (case C) and 5 times faster than randomly bundled PH (case D). It also finds results with higher quality (optimality gap of $0.24\%$ compared to $1.65\%$ and $29.4 \%$ for randomly bundled PH and standard PH, respectively). From a computation time perspective, cases B and E are relatively similar, but the quantified quality of results is significantly different, and case E provides a better optimality gap in somewhat less time.

Table 3 Summary of results for 13-bus system

To further investigate the impact of parallelizing and variable freezing on computational time, we compared the performance of cases C–E under the following three alternatives:

• Alternative 1: With variable freezing and without parallelizing

• Alternative 2: Without variable freezing and with parallelizing

• Alternative 3: With variable freezing and with parallelizing

Table 4 summarizes the impact of these two factors on optimality gap and computational time for cases C-E under these three alternatives.

The result from the second row shows that variable freezing may negatively affect the quality of results and increases the optimality gap (Alternative 2, in which variable freezing is ignored, has the lowest optimality gap). As expected, parallelizing will not affect the quality of results (similar optimality gaps for Alternative 1 and Alternative 3). The third row in Table 4 shows the computational time for three alternatives. For Alternative 1, standard PH (Case C) is affected the most (compared to cases D and E) when parallelizing is not used because each iteration includes running TEP for all individual scenarios (simulation time increased from 2.05 to 93.92 hours). For bundled PH, both cases D and E could solve the problem in approximately the same time showing that when simulations are run sequentially (instead of in parallel), the impact of balancing computational burden between bundles (that will result in an earlier termination for a parallelized for loop) will be less effective. Variable freezing has a significant impact on computational time as it will decrease both the number of iterations and computational time for each iteration. Comparing the computational time and optimality gap for Alternative 2 and Alternative 3 shows the trade-off between quality of results and computational time. For example, for case E, the optimality gap is slightly increased from $0.12\%$ to $0.24\%$; however, the computational time is decreased from 82.7 hours to 0.25 hours demonstrating the effectiveness of the heuristic methods used for PH performance improvements.

Table 4 Impact of parallelizing and variable freezing on computational performance

5.2 Reduced ERCOT System

A reduced ERCOT network is developed with 3179 buses, 474 generation units, 3598 load centers, 123 wind farms, and 4458 branches. All non-radial 138kV and 345kV lines in the ERCOT network are explicitly modeled. Generators and loads that were connected to lower voltage levels or radial network are moved to nearby modeled buses. Ten different scenarios are developed to model load and wind uncertainties (using historical data) with 46 new lines as candidates for transmission expansion (Majidi-Qadikolai and Baldick 2018). Similar to the 13-bus system, five cases A–E are simulated to compare the results. As total number of scenarios is 10 for this case, $\kappa$ is set to 10 for cases A and B. For phase I in case E, $\kappa =5$ and for case D and phase II in case E, $\kappa =2$. The proposed method in (Majidi-Qadikolai and Baldick 2016a) is used to solve TEP in lines 3 and 11 of the bundled PH algorithm (Fig. 3). The parameter $\vartheta$ is set to 3. Other parameters are set the same as the 13-bus system.

Numerical results are given in Table 5. We could not get a feasible solution for cases A and B after 15 days, demonstrating the need for decomposition-based methods for large-scale problems. As the number of scenarios is not large for this system, standard PH (case C) has a reasonable performance; however, the elapsed time of over a week may not be acceptable. For case D (randomly bundled scenarios), simulation is terminated manually after 14.9 days and a lower bound is calculated. The fifth column (case E) demonstrates the impact of the proper framework design/setting on improving quality of results (decreasing optimality gap from $6.24\%$ to $0.97\%$) and reducing computational time (by more than 5.3 times) for solving this large-scale problem.

Results for this case demonstrates that bundling by itself may not necessarily improve the performance of PH without careful consideration of choice of bundles, because as explained in Sect. 3.4.2, each iteration for the PH algorithm is finished only when TEP for all bundles are completely solved. Because of this, randomly grouping scenarios may result in forming TEP subproblems with significantly different sizes (based on (73) and (74)) although the size of bundles ($\kappa$) is similar. This comparison also highlights the importance of the grouping step in the scenario bundling.

Table 5 Summary of results for reduced ERCOT system

5.3 Full ERCOT System—High Load Growth Area Project

For this case study, a full ERCOT network model is used, and an area of the transmission system with high load growth at existing load centers is evaluated (Majidi-Qadikolai et al. 2018). In this project, a fast-growing load pocket in Central Texas is studied with the assumption that load growth will increase current load forecasts by 50% in the near-term horizon. Transmission planners may look at varying load assumptions as a sensitivity scenario to their base case studies. The sensitivity case studies are useful to anticipate what transmission infrastructure may be required if certain less anticipated conditions unfold. In this case study, the on-peak condition is evaluated. An initial list of candidate upgrades (including 23 lines and transformers) are identified by the transmission planning team. The area of study for this project and candidate options (doted lines) are shown in Fig. 7.

Fig. 7

Area of study for high area load growth project. Solid lines show existing branches (Red: 345 kV, Blue: 138 kV, Green: 69 kV). Dotted lines show candidate branches for expansion

We have evaluated this project under the following conditions:

• Without low-cost option

• With low-cost option

“Low-cost option” refers to minor transmission upgrades that are not known/included in the initial candidate lines list, but during practical TEP studies might be captured by planning experts and are added to their analysis (Majidi-Qadikolai et al. 2018).

5.3.1 Case A: Ignoring Low-Cost Options

If we ignore the possibility of “low-cost options” and use standard TEP optimization formulation (from Sect. 3.1) with preliminary candidate options, the TEP optimization tool selects two new lines and two transformers as the optimal expansion plan. The summary of results is shown in Table 6, and selected branches are highlighted (solid brown lines) in Fig. 8.

Table 6 Summary of results for high load growth area project without low-cost options

Fig. 8

Selected branches for case A (without low-cost options). New branches are highlighted with solid brown lines

5.3.2 Case B: Integrating Low-Cost Options

For the second case, “low-cost options” feature is integrated into TEP formulation to capture potential upgrades in local area that might not be part of the initial candidate options. The summary of results for this case is provided in Table 7, and selected branches are highlighted in Fig. 9.

Table 7 Summary of results for high load growth area project with low-cost options

Fig. 9

Selected branches for case B (with low-cost options). New branches are highlighted with solid brown lines, and low-cost upgrades are highlighted with dotted brown lines

As shown in this Table, compared to case A, the TEP tool in this case suggested upgrading two existing lines as low-cost options instead of building one new transmission line and one new transformer. The planning engineers confirmed that these upgrades are practically possible, and they have added two new upgrades to the primary candidate list. Those upgrades are selected by TEP optimization tool when we rerun step 2. The selected plan in case B is around $35 million less expensive than the selected plan in case A (it is a fair comparison because in practice it is not possible to define all potential upgrades at the beginning of each study). Although the result of case A is optimal from mathematical perspective, the results of case B for this project demonstrates that a combination of advantages of optimization-based approaches and expertise of transmission engineers can lead to a better expansion plan.

Appendix

In this appendix, input date for 13-bus system is provided (Tables 8, 9, and 10).

Table 8 Load and generation data in (MW)

Table 9 Existing transmission network data

Table 10 Candidate lines