Coordination of power and natural gas markets via financial instruments

Abstract

Current electricity and natural gas markets operate with deterministic description of uncertain supply, and in a temporally and sectorally decoupled way. This practice in energy systems is being challenged by the increasing integration of stochastic renewable energy sources. There is a growing need for exchanging operational flexibility among energy sectors, which requires to improve the sectoral coordination between electricity and natural gas markets. In addition, the dispatch of flexible units in both sectors needs to be made in a more uncertainty-aware manner, requiring to strengthen the temporal coordination between day-ahead and real-time energy markets. We explore the use of existing financial instruments in the form of virtual bidding (VB) as a market-based solution to enhance both sectoral and temporal coordination in energy markets. It is established in the literature that VB by purely financial players is able to enhance the temporal coordination between deterministic day-ahead and real-time markets. By developing various stochastic equilibrium and optimization models, we show that VB by physical players, i.e., gas-fired power plants, at the interface of power and natural gas systems is of great potential to improve not only the temporal coordination between deterministic day-ahead and real-time markets, but also the sectoral coordination between deterministic electricity and natural gas markets. We exploit a fully stochastic co-optimization model as an ideal benchmark, and numerically illustrate the benefits of VB for increasing the overall market efficiency in terms of reduced expected operational cost of the entire energy system. We eventually show that flexible resources in both electricity and natural gas markets are dispatched more efficiently in the day-ahead stage when VB exists.

1 Introduction

The growing share of power production from stochastic renewable energy sources, e.g., wind and solar power units, increases the need for operational flexibility¹ to deal with their variability and uncertainty (NERC 2010). Natural gas-fired power plants are one of the main sources of flexibility in power systems, and are able to compensate for the production variability and uncertainty caused by stochastic renewable sources (Meibom et al. 2013; Gil et al. 2014). These gas-fired power plants operate at the interface of electricity and natural gas systems, yielding both physical and economic interactions (Fleten and Nasakkala 2010). The natural gas system is crucial for ensuring fuel availability and technical feasibility, while it is also able to provide power systems with flexibility through stored gas in the pipelines (Correa-Posada and Sánchez-Martín 2015; Yang et al. 2018; Ordoudis et al. 2019). The volatile and uncertainty-driven dispatch of gas-fired power plants to offset wind and solar intermittency introduces demand fluctuations into the natural gas market and propagates uncertainty from the power system to the gas sector (Heinen et al. 2017; Dall’Anese et al. 2017; Nicholson and Quinn 2019; Ratha et al. 2020).

1.1 On the need for sectoral coordination between electricity and natural gas markets

Despite increasing interdependencies between energy sectors, in practice electricity and natural gas markets are still cleared sequentially and separately. In various countries and regions, the electricity market participants including gas-fired power plants estimate the gas price and submit their offer to the electricity market accordingly. Given the gas demand determined from the clearing outcomes of the electricity market, the natural gas market operator clears the market and disseminates the actual gas prices, which are not necessarily identical to estimated gas prices in the electricity market (Byeon and Van Hentenryck 2020; Ordoudis et al. 2019). In several countries these two energy markets are usually asynchronous, implying that the timing of the gas nomination cycles is not necessarily well aligned with the needs of the electricity sector (Tabors et al. 2012). In addition, power and natural gas markets generally use different trading mechanisms. For example, European gas markets decouple the trading and transport of natural gas by using an entry-exit system (Hallack and Vazquez 2013; Schewe et al. 2020), which is not fully harmonized with the current zonal market design of European electricity markets. Despite all these differences, sectoral coordination between electricity and natural gas markets is crucial for renewable-based energy systems. This need has been recognized by several market regulators, including Federal Energy Regulatory Commission (FERC) in the U.S. that issued Order 809 in April 2015. This order makes changes to the gas nomination cycles to improve day-ahead and intra-day coordination of power and natural gas systems. The details and implications of this order are described in Carter et al. (2016), Orvis and Aggarwal (2018) and Craig et al. (2020). The sectoral coordination of power and natural gas systems in their short-term operations has been extensively addressed in the recent literature. While several papers suggest an extreme regulatory solution by merging the operational problems of power and natural gas systems (Correa-Posada and Sánchez-Martín 2015; Zlotnik et al. 2016; Chen et al. 2019; Manshadi and Khodayar 2019; Ordoudis et al. 2019; Schwele et al. 2019; Roald et al. 2020; Ratha et al. 2020), some other works develop more practical solutions where the clearing sequence of power and natural gas markets is preserved. For example, Zhao et al. (2019) propose a coordination mechanism that does not require the exchange of proprietary information between power and natural gas system operators—the only information to be exchanged between the two markets are fuel price, supply and demand. An operational equilibrium model for sequential but interrelated power and natural gas systems is developed in Chen et al. (2020). As a practical solution, Byeon and Van Hentenryck (2020) propose a hierarchical tri-level optimization methodology that makes the unit commitment problem aware of gas networks, while recognizes and eliminates the invalid bids of gas-fired power plants. Finally, Ordoudis et al. (2020) propose various methods built upon bi-level programming, where coordination between power and natural gas systems is obtained by optimally adjusting the gas volume availability as well as the estimated gas price in the day-ahead electricity market.

1.2 On the need for temporal coordination between trading floors

In addition to the lack of proper coordination between various energy sectors, another challenge for renewable-based energy markets is the potentially loose temporal coordination between electricity markets that are cleared sequentially in different points of time² (Morales et al. 2014). The majority of current electricity markets throughout the world clear several sequential markets in short run, including day-ahead (DA), intra-day and real-time (RT) markets (Daraeepour et al. 2019). The DA market is cleared based on a deterministic description of uncertain supply. Given updated but still single-point deterministic forecasts of uncertain supply, intra-day and RT markets adjust the power system imbalances. Similar balancing stages, which are not necessarily market-oriented, exist in the natural gas sector. For example, there is a single balancing stage in the Danish gas system, where the day-ahead gas nominations can be modified (Energinet 2021). This practice is likely different in various countries. For the sake of a stylized model, we consider a single balancing stage, the so-called RT market, in the power sector as the representative of all balancing stages to be cleared after the DA electricity market. Similarly, we consider a single RT market for the natural gas sector to be cleared after the DA gas market. The key point is that despite all advances in forecasting tools, the deterministic forecast of stochastic renewable energy sources used at the DA stage can still be erroneous, which may cause wrong unit commitment and dispatch decisions. This eventually results in market inefficiency, i.e., a comparatively high operational cost for the whole system (Jonsson et al. 2010). To resolve such an inefficiency, temporal coordination between DA and RT electricity markets and also between DA and RT natural gas markets is required (Morales and Pineda 2017). Through temporal coordination, the DA electricity and the DA gas markets become uncertainty-aware. In contrast, via sectoral coordination, the electricity market becomes aware of the gas sector. Both temporal and sectoral coordination are desired for future energy systems with high penetration of renewables.

1.3 Financial instruments as coordination mechanisms

The market-based mechanisms for improving both sectoral and temporal coordination of power and natural gas systems range from an extremely disruptive choice of designing a fully stochastic integrated energy market to less-disruptive solutions that preserve the current regulatory framework with separate, sequential and deterministic clearing of the markets. The latter, i.e., less disruptive (or “soft”) market mechanisms, is the focus of this paper, while the former, i.e., the fully stochastic integrated energy market, is used as an ideal benchmark to assess the performance of the proposed mechanisms. By soft market mechanisms, we refer to any mechanism or process that increases the overall system efficiency while respecting the current operational and economic regulations. These non-disruptive coordination mechanisms can be of financial, operational or communicative nature. These mechanisms aim at enhancing the information flow, either directly or indirectly, between the systems and creating incentives for each sector to dispatch resources in a way that benefits the overall system. Among others, the soft market-based mechanisms for coordination of energy sectors can be achieved by direct or indirect information exchange among the markets (Zhao et al. 2019; Byeon and Van Hentenryck 2020), defining new market products (Warrington et al. 2013; Wang and Hobbs 2016; Chen et al. 2017; O’Malley et al. 2019), prescribing new bidding formats (Liu et al. 2015; O’Connell et al. 2016; Savelli et al. 2018; Bobo et al. 2021), and introducing new market players which act as coordinators at the interface of different sectors.

In this paper, we focus on the use of financial instruments to enhance the coordination of power and natural gas markets. Specifically, we explore the effect of virtual bidding (Hogan 2016), also known as “convergence bidding” (Li et al. 2015), as a soft market-based mechanism for improving both temporal and sectoral coordination of power and natural gas systems under uncertainty. Virtual bidding (VB) refers to financial arbitrage between two trading floors in an energy market, e.g., between DA and RT electricity markets. A virtual bidder may earn profit due to price difference in DA and RT markets by performing arbitrage. This virtual bidder can be a purely financial player who has no physical asset, the so-called explicit virtual bidder, or she can be one of the existing physical market players, the so-called implicit virtual bidder (Isemonger 2006; Mather et al. 2017). An example of an implicit virtual bidder is a generator, who performs arbitrage between DA and RT markets by selling electricity in DA more than her installed capacity. Further details about VB and in particular explicit and implicit VB will be provided later in Sect. 2.2.

Virtual bidding exists today in various electricity markets throughout the world. Most of U.S. electricity markets allow both forms of virtual bidding, i.e., explicit and implicit, such as California Independent System Operator, CAISO (Li et al. 2015), Midcontinent Independent System Operator, MISO (Birge et al. 2018) and Pennsylvania-New Jersey-Maryland Interconnection, PJM (Hogan 2016). Although explicit VB is not allowed in European energy markets, implicit VB may occur to some extent (Papavasiliou et al. 2021). The reason for this is that every power producer in European energy markets is allowed to submit an offer in the DA market based on her entire production portfolio. This leaves room for producers to do arbitrage to some extent between DA and RT markets.

Note that by taking VB into account in this paper as a coordination tool, we do not intend to focus on a market in a specific country. In contrast, we consider a stylized market model, where both electricity and gas markets allow VB, and they are cleared sequentially in both DA and RT stages. This stylized model offers a general market framework, but it may not respect all regulatory and operational details of markets in the U.S., or in Europe, or anywhere else.

1.4 State of the art, contributions, and paper organization

The implications of VB on converging DA and RT prices and on strengthening the temporal coordination among various trading floors, e.g., between deterministic DA and RT electricity markets, have been extensively studied in the literature, see for example Parsons et al. (2015), Ito and Reguant (2016), Morales and Pineda (2017), Birge et al. (2018), Kazempour and Hobbs (2018) and Kohansal et al. (2020). The reason for such an improvement is that VB increases market liquidity and brings additional uncertainty-related information to the deterministic DA market through virtual bids. The underlying assumption is that virtual bidders possess more adequate information about uncertainty compared to the market operator who clears the DA market in a deterministic manner. Therefore, it is implicitly assumed that virtual bidders have used a proper stochastic model to make uncertainty-aware bidding decisions. These bids indirectly make the deterministic DA market uncertainty-aware. In order to quantify the maximum potential of VB to improve the coordination, we assume each virtual bidder has perfect foresight of DA and probability distribution of RT prices. The affiliated assumption is that this virtual bidder is risk-neutral, and is not going to use such information in a strategic manner. All these assumptions together imply that we consider “perfect” virtual bidding³. Compared to the stochastic market-clearing model as an ideal benchmark, it is worth noting that the efficiency of the deterministic DA market may not be “fully” improved by VB under some circumstances (Parsons et al. 2015), or VB might not be able to do so alone (Morales and Pineda 2017), or may have some limits (Ito and Reguant 2016; Birge et al. 2018). An example of such conditions is markets where virtual bidders behave strategically (Lo Prete et al. 2019a, b).

In the context of the existing literature, the main contribution of this paper is to show that VB is able to improve not only the temporal coordination among deterministic DA and RT markets, but also the sectoral coordination among deterministic electricity and natural gas markets. To the best of our knowledge, the effect of VB on sectoral coordination improvement among energy sectors has not been addressed in the literature. We illustrate that virtual bidders (here, gas-fired power plants) are of great potential to behave as coordinators at the interface of power and natural gas systems, and enhance the overall efficiency by indirect information exchange among the two sectors through making informed unit commitment and dispatch decisions. In particular, we aim at quantifying the maximum potential of VB for improving both sectoral and temporal coordination of electricity and natural gas markets under supply uncertainty. To this purpose, we first integrate explicit VB to electricity and natural gas markets, which achieves temporal coordination between deterministic DA and RT markets in each energy sector. Then, we investigate the possibility of natural gas-fired power plants, who are at the interface of power and natural gas markets, to behave as implicit virtual bidders. We illustrate that such implicit virtual bidders have the capability to achieve both temporal and sectoral coordination in deterministic electricity and natural gas markets.

From a methodological perspective, we model renewable generation uncertainty via a finite set of scenarios, and develop several stochastic generalized Nash equilibrium (GNE) problems (Facchinei and Kanzow 2007), whose solution existence can be mathematically ensured under some assumptions. These stochastic equilibrium models serve as simulation tools for deriving policy implications to explore how much VB can improve the sectoral and temporal coordination in renewable-based electricity and natural gas markets. We also provide analytical insights by comparing the GNE problems and the ideal benchmark, i.e., the two-stage stochastic co-optimization problem (Pritchard et al. 2010; Zavala et al. 2017; Zakeri et al. 2019). It is important to emphasize that all stochastic equilibrium models in this paper are developed based on a stylized market framework. While this framework may not accurately reflect the way actual energy markets are cleared in specific countries, we develop our models in a general and compact manner to gain overall insights into the coordination of power and natural gas markets. These stylized models should be seen as policy tools, since they are not intended to be used for market clearing in practice. The manuscript is organized as follows. In Sect. 2 we provide more details about temporal and sectoral market coordination, the concept of VB and our modeling assumptions. Sects 3 and 4 contain the mathematical formulations of GNE models with explicit and implicit VB, respectively. The formulation of the ideal benchmark model is included in Sect. 5. In Sect. 6, we show the numerical results for a case study, and finally Sect. 7 concludes the paper. For clarity purposes, we maintain the general representation of optimization problems throughout the paper, and include their detailed representations in the online appendix (Schwele et al. 2021).

2 Preliminaries

This section first highlights the temporal and sectoral coordination of power and natural gas markets under uncertainty. Then, it further describes both types of VB (explicit and implicit). Finally, it summarizes the modeling assumptions made in this paper.

2.1 Two-dimensional coordination: temporal and sectoral

Fig. 1

Sequential setup of electricity and natural gas markets, including four market-clearing sequences I to IV

The independent market operators clear each trading stage (DA and RT) separately and sequentially for electricity and natural gas markets. The current market-clearing framework for electricity and natural gas systems is illustrated in Fig. 1, including four market-clearing sequences. First, the electricity market is cleared in a DA auction 12–36 h before actual energy delivery using a deterministic forecast of uncertain parameters, e.g., renewable power generation and natural gas prices. Note that future natural gas prices directly impact the marginal production cost of natural gas-fired power plants and consequently the merit order⁴ in the electricity market. Second, the natural gas DA market is cleared for given natural gas demand of gas-fired power plants determined by their dispatch in the electricity market. Third, once the uncertainty is realized (e.g., scenario $\omega$ occurs), the RT electricity market is cleared to adjust imbalances under fixed DA unit commitment and dispatch decisions. Fourth, the natural gas market is cleared in RT, while the dispatch of gas suppliers in DA and the demand of natural gas-fired power plants in RT are given.

The sequential setup in Fig. 1 is totally uncoordinated in both temporal and sectoral dimensions. This setup is temporally uncoordinated since both electricity and gas markets in DA are cleared based on the available deterministic forecast in that stage, without foresight into the potential deviations that may realize in RT. It is also sectorally uncoordinated because the electricity market is cleared based on an estimation of natural gas price, and the gas market is cleared afterwards. As it is common in practice, the integration of operating reserve as an extra market product is able to potentially enhance the temporal coordination between DA and RT markets. However it may bring extra inefficiencies if the value assigned for the minimum reserve requirement in the DA market is not properly selected (Doherty and O’Malley 2005; Zugno and Conejo 2015). This can be an even more challenging issue in European markets, where energy and reserve markets are cleared sequentially (Dominguez et al. 2019). Aligned with such a sequential energy and reserve market-clearing framework, Dvorkin et al. (2019) propose a stochastic bi-level program that determines the optimal value for the minimum reserve requirement. Note that we exclude the reserve market as it is not the focus of this study.

While the share of stochastic renewable energy sources is growing, the lack of temporal and sectoral coordination in electricity and natural gas markets may cause market inefficiency. In other words, the overall operational cost of electricity and natural gas systems in DA and RT might be comparatively higher than that cost in the ideal co-optimization benchmark. The reason for such an inefficiency is suboptimal DA dispatch decisions made due to uncoordinated DA market clearing. If flexible resources are dispatched in the DA stage inefficiently, they will not be available in RT to cope with imbalances. As a consequence, more expensive actions, e.g., load curtailment, might be required. Therefore, it is desirable to dispatch the flexible sources in DA in an efficient manner while preserving the current sequential market-clearing framework. This requires soft market-based mechanisms for enhancing the temporal and sectoral coordination of power and natural gas markets, which is the focus of this paper.

Fig. 2

An example for the hourly trading profile of an explicit virtual bidder in the DA market (left), in the RT market (middle), and her final position, i.e., the sum of her trades in DA and RT (right). The amount of trade in DA and RT could be negative (as an energy buyer) or positive (as an energy seller), but the final position is always zero

Fig. 3

Explicit virtual bidding by arbitraging electricity between the DA and RT electricity markets (or by arbitraging gas between DA and RT natural gas markets). This type of virtual bidding has potential to enhance temporal coordination between DA and RT markets

2.2 Virtual bidding

Virtual bidding is a purely financial instrument for market players including suppliers, consumers, and financial traders to do arbitrage based on price differences between trading floors (Hogan 2016; Li et al. 2015; Birge et al. 2018). We explain below both forms of VB, i.e., explicit and implicit VB (Isemonger 2006; Mather et al. 2017).

2.2.1 Explicit virtual bidding

An explicit virtual bidder is a purely financial player who does not own any physical assets. Therefore, her positions in DA and RT need to even out to zero. For example, an explicit virtual bidder may buy 10 MWh in the DA electricity market in a specific hour at the DA market price in that hour, and then sells the same 10 MWh back in the RT electricity market at the same hour but at the price of the RT market. Therefore, her payoff is equal to the difference between the DA and RT prices times the amount of virtually traded power. Fig. 2 illustrates another example for the hourly trading profile of an explicit virtual bidder. Assuming that the explicit virtual bidder is a price-taker with perfect foresight into the distribution of DA and RT prices, she is supposed to enhance informational and productive efficiency of the two-settlement market by bringing more competitiveness, liquidity and transparency to wholesale energy markets.

Figure 3 illustrates how such an explicit VB is integrated into the two-settlement market-clearing setup. While DA and RT energy markets are cleared deterministically and sequentially, the explicit virtual bidder solves a stochastic program maximizing her own expected profit. The outcomes of the stochastic program of virtual bidders, i.e., virtual trades, are exogenous in DA and RT markets. In other words, these virtual bidders act as self-scheduling market players. This means that they make their DA dispatch decisions internally, rather than submitting price-quantity bids to the DA market. However, these self-scheduling market players can equivalently be viewed as financial players who submit price-quantity bids to the DA market at sufficiently low (high) selling (purchasing) prices to ensure such bids would be cleared. In our proposed model, the amount of virtual trade is exogenous in DA and RT market-clearing problems, while each virtual bidder is still paid or pays based on the corresponding market-clearing prices. Therefore, the DA and RT market-clearing prices are exogenous in the stochastic program of the virtual bidder. It is obvious from Fig. 3 that a set of interrelated optimization problems (one for DA market clearing, one for RT market clearing per scenario, and one for explicit virtual bidder) is required to explore the performance of explicit VB. This clarifies the need for developing a stochastic equilibrium model. It is demonstrated in Kazempour and Hobbs (2018) that this setup can bring temporal coordination between deterministic DA and RT electricity markets. This is an interesting insight for market operators since they can keep the market clearing deterministic, while leaving the correction of market inefficiency to virtual bidders. However, VB may not always work in such a desirable way, as discussed in Parsons et al. (2015), Morales and Pineda (2017) and Birge et al. (2018).

Fig. 4

An example for the hourly trading profile of an implicit virtual bidder, e.g., a gas-fired power plant, in the DA market (left), in the RT market (middle), and her final position, i.e., the sum of her trades in DA and RT (right). The amount of trades in DA and RT could be negative (as an energy buyer) or positive exceeding the installed capacity (as energy seller), but the final position should lie within zero and the installed capacity

2.2.2 Implicit virtual bidding

Unlike the explicit VB, the implicit virtual bidder is a physical market player who blends virtual bids with physical bids. Figure 4 shows an example for the hourly trading profile of an implicit virtual bidder, who is able to do arbitrage between DA and RT markets as long as her final position, i.e., the sum of her trade in the DA and RT markets, lies within her actual operational limits.

Fig. 5

Implicit virtual bidding by a natural gas-fired power plant, who is on the interface of electricity and natural gas systems. This power plant self-schedules her power productions and gas consumptions in DA and RT electricity and natural gas markets. This type of virtual bidding has potential to enhance temporal and sectoral coordination between DA and RT electricity and natural gas markets

An example of such a player is a natural gas-fired power plant who is at the interface of power and natural gas systems, as illustrated in Fig. 5. This power plant has potential to enhance both temporal and sectoral coordination in electricity and natural gas markets. Although the presence of explicit VB may eliminate the motivation for physical players to perform arbitrage, physical players may still find self-scheduling profitable to forgo the market and dispatch their production/consumption themselves outside the market. For example, assume a natural gas-fired power plant that has perfect foresight into future DA and RT power and gas prices, and realizes that her profit is not maximized when she participates in deterministic electricity and natural gas markets. In other words, she has the opportunity to gain a higher profit in expectation by self-scheduling outside the market (Guo et al. 2016; Sioshansi et al. 2010). Note that the power production and gas consumption of this power plant are exogenous in the market-clearing problems, while she still pays/is paid based on the market-clearing prices (Jha and Wolak 2015; Papavasiliou et al. 2015). An implicit virtual bidder may benefit from self-scheduling by solving her own stochastic program with better representation of uncertainty and technical constraints for a longer time horizon. However, these self-schedulers take on the full risk of RT price uncertainty. The influence of risk aversion and price volatility on the decision of generators to do self-scheduling is discussed in Papavasiliou et al. (2015) and Conejo et al. (2004).

2.3 Modeling framework and assumptions

In general, stochastic equilibrium models are computationally challenging, and therefore simplifying assumptions might be required. In addition, these assumptions enable us to quantify the maximum potential of VB in enhancing temporal and sectoral coordination in the proposed simulation tool. In the following we explain our assumptions.

As pointed out in Sect. 1, we consider two trading floors (DA and RT) only, and other potential floors, e.g., intra-day adjustment markets, are excluded. We also consider simple price-quantity bids only, discarding any other types of bids, e.g., bid curves and block bids, which are prevalent in European electricity markets. Wind power production is assumed as the only source of uncertainty. Note that the wind power forecast in DA is a single point (deterministic), while different scenarios may occur in RT, i.e., we are not sure about the actual outcome of the uncertain parameter. Wind power uncertainty is represented using a finite set of scenarios. The wind power production cost is zero, and can be spilled at zero cost.

Both electricity and natural gas demands are assumed to be inelastic to price. This implies that demand-side flexible resources are discarded. All demand and supply in both energy sectors are assumed to be located at a single node, neglecting the transmission systems. By discarding the natural gas network, we exclude the potential flexibility that can be provided by the stored gas in the pipelines, which is known as linepack.

A multi-period unit commitment scheduling model is used in the power sector. We relax the binary nature of commitment status of conventional generators to lie within zero and one, but in a tight manner (Hua and Baldick 2017). This relaxation ensures convexity, which is required to solve the stochastic equilibrium model as a mixed complementarity problem, while providing more accurate cost estimates than pure dispatch models. The production cost of generators is assumed to be a linear function. We assume all market players including virtual bidders (either explicit or implicit) to act competitively, non-strategically, and in a risk-neutral manner when participating in the markets, so they offer at prices identical to their marginal costs. We assume virtual bidders can always zero out their position in RT.

Notation We denote by ${\mathbb {R}}$ and ${\mathbb {R}}_+$ free and non-negative real numbers, respectively. We use upper case letters for matrices and lower case letters for vectors. Bold lower case letters denote vectors of variables. Note that e is the vector of ones and $(.)^\top$ is the transpose operator. We use functions h(.) and g(.) to show equality and inequality constraints in every optimization problem, but note that these constraints for different optimization problems are not necessarily identical.

3 Temporal coordination

In this section, we explore the temporal coordination between deterministic DA and RT markets via explicit VB. The sectoral coordination between deterministic electricity and natural gas markets via implicit VB will be discussed later in Sect. 4.

3.1 Temporal coordination between DA and RT markets

We present below optimization problems in the power sector, and then in the gas sector. These optimization problems are interrelated and construct a stochastic equilibrium problem.

3.1.1 Explicit electricity virtual bidder

The expected profit-maximization problem of each explicit electricity virtual bidder $r \in {\mathcal {R}}$ over the time horizon ${\mathcal {T}}$ writes as

$$\begin{aligned}&\Bigg \{\mathop {\text {max}}\limits _\mathbf{v ^{\mathrm{E}}_{r},\varvec{\Delta }{} \mathbf{v} ^{\mathrm{E}}_{r}} \ {\varvec{\lambda }^{\mathrm{DA,E}}}^\top \ \mathbf{v} ^{\mathrm{E}}_{r} + \left( \sum _{\omega } \pi _{\omega } \ \varvec{\lambda }^{\mathrm{RT,E}}_{\varvec{\omega }}\right) ^\top \varvec{\Delta }{} \mathbf{v} ^{\mathrm{E}}_{r} \end{aligned}$$

(1a)

$$\begin{aligned}&\!\!\text {subject to}\ \mathbf{v} ^{\mathrm{E}}_{r} + \varvec{\Delta }{} \mathbf{v} ^{\mathrm{E}}_{r} = 0 \Bigg \} \ \forall {r}. \end{aligned}$$

(1b)

Note that (1) is a two-stage stochastic linear program. The virtual bidder determines her DA position $\mathbf{v} ^{\mathrm{E}}_{r} \in {\mathbb {R}}^{{\mathcal {T}}}, \ r \in {\mathcal {R}}$ given the DA electricity prices $\varvec{\lambda }^{\mathrm{DA,E}} \in {\mathbb {R}}^{{\mathcal {T}}}$ as well as the distribution of RT electricity prices $\varvec{\lambda }^{\mathrm{RT,E}}_{\omega } \in {\mathbb {R}}^{{\mathcal {T}}}, \ \omega \in \Omega$ weighted by probability $\pi _{\omega }$ over the set of scenarios $\omega \in \Omega$. This virtual bidder is a purely financial player without physical assets, and therefore is obliged to offset her DA position by her RT position $\varvec{\Delta }{} \mathbf{v} ^{\mathrm{E}}_{r} \in {\mathbb {R}}^{{\mathcal {T}}}, \ r \in {\mathcal {R}}$ in each scenario⁵. Objective function (1a) maximizes the expected profit of explicit virtual bidder who arbitrages between the DA and RT electricity markets. Constraint (1b) ensures that the virtual bidder sells (buys) the same amount back in the RT market that was bought (sold) in the DA market. One important observation about this explicit virtual bidder is that she enforces the convergence of DA and expected RT electricity prices (Kazempour and Hobbs 2018). Derived from Karush–Kuhn–Tucker (KKT) optimality conditions associated with (1), the virtual bidder enforces the DA and the expected RT electricity prices to be equal, i.e., $\varvec{\lambda }^{\mathrm{DA,E}}=\sum _{\omega } \pi _{\omega }\varvec{\lambda }^{\mathrm{RT,E}}_{\omega }$. See online appendix (Schwele et al. 2021) for further details. Note that market operators treat the dispatch decision of virtual bidders as fixed input into the market-clearing problem presented in the following section.

3.1.2 DA electricity market

Consider ${\mathcal {G}}$ number of gas-fired generators and ${\mathcal {C}}$ number of non gas-fired generators, such that ${\mathcal {G}} \ \cup \ {\mathcal {C}} = {\mathcal {I}}$. Besides, consider ${\mathcal {J}}$ number of wind power units. For given production cost of non gas-fired generators $\mathbf{C} ^{\mathrm{E}} \in {\mathbb {R}}^{{\mathcal {C}}}_{+}$, estimation of natural gas prices $\tilde{\varvec{\lambda }}^{\mathrm{G}} \in {\mathbb {R}}^{{\mathcal {T}}}$ to compute the production cost $C(\tilde{\varvec{\lambda }}^{\mathrm{G}}) \in {\mathbb {R}}^{{\mathcal {G}} \times {\mathcal {T}}}$ for gas-fired generators, and fixed dispatch of virtual bidders $\mathbf{v} ^{\mathrm{E}}_{r}$ obtained from (1), the electricity market operator clears the market in DA to minimize the total operational cost of the power system as

$$\begin{aligned}&\quad \mathop {\text {min}}\limits _\mathbf{p ,\mathbf{u} ,\mathbf{s} ,\mathbf{w} } \ e^\top \mathbf{p} ^{\mathrm{C}} \ \mathbf{C} ^{\mathrm{E}} + e^\top \mathbf{p} ^{\mathrm{G}} \ C(\tilde{\varvec{\lambda }}^{\mathrm{G}}) \ e + e^\top \mathbf{s} \ e \end{aligned}$$

(2a)

$$\begin{aligned}&\!\!\text {subject to}\ h(\mathbf{p} ,\mathbf{w} ,\mathbf{v} ^{\mathrm{E}}_{r}) = 0:\varvec{\lambda }^{\mathrm{DA,E}}, \end{aligned}$$

(2b)

$$\begin{aligned}&\qquad \qquad \quad g(\mathbf{p} ,\mathbf{w} ,\mathbf{u} ,\mathbf{s} ) \le 0. \end{aligned}$$

(2c)

Note that (2) is a deterministic linear program. Variables $\mathbf{p} ,\mathbf{u} ,\mathbf{s} \in {\mathbb {R}}^{{\mathcal {T}}\times {\mathcal {I}}}_{+}$ are the dispatch, commitment status, and start-up cost of conventional generators in DA, respectively. In particular, $\mathbf{p} ^{\mathrm{C}} \in {\mathbb {R}}^{{\mathcal {T}}\times {\mathcal {C}}}_{+}$ and $\mathbf{p} ^{\mathrm{G}} \in {\mathbb {R}}^{{\mathcal {T}}\times {\mathcal {G}}}_{+}$ are the DA dispatch of non gas- and gas-fired generators, respectively. The commitment status $\mathbf{u}$ is relaxed to lie within zero and one. Besides, $\mathbf{w} \in {\mathbb {R}}^{{\mathcal {T}}\times {\mathcal {J}}}_{+}$ refers to the DA dispatch of wind power units, limited by their deterministic forecast in DA.

Objective function (2a) minimizes the total system cost in DA, including the operational and start-up costs of conventional generators. Equality constraint (2b) enforces the balance between power production and consumption in DA with inelastic demand. The virtual DA positions $\mathbf{v} ^{\mathrm{E}}_{r}$ are treated as given inputs. The dual variable associated with power balance (2b), i.e., $\varvec{\lambda }^{\mathrm{DA,E}} \in {\mathbb {R}}^{{\mathcal {T}}}$, provides the DA electricity price. Recall that this vector of dual variables was treated as exogenous values in the problem of virtual bidders (1). Inequality constraints (2c) enforce lower and upper bounds on the DA dispatch of wind and conventional generation, impose ramping limits of conventional generators, represent the tight relaxation of unit commitment, and compute the start-up cost of each conventional generator. The detailed representation of all equality and inequality constraints is given in the online appendix (Schwele et al. 2021).

3.1.3 RT electricity market

The actual wind power production is realized in RT, which might not be necessarily identical to the deterministic wind power forecast in DA. Therefore, the electricity market operator clears the RT market to make the necessary adjustments in order to keep the system balanced. The balancing actions are the power adjustment of generators and the two extreme actions, i.e., wind spillage and load shedding. The (relaxed) commitment status of fast-starting conventional generators ${\mathcal {F}}\subset {\mathcal {I}}$ and therefore their start-up cost can be updated in RT, while that is not the case for the slow-starting generators ${\mathcal {S}}\subset {\mathcal {I}}$. Note that ${\mathcal {F}} \ \cup \ {\mathcal {S}} = {\mathcal {I}}$. For given production costs of non gas-fired and gas-fired generators $\mathbf{C} ^{\mathrm{E}} \in {\mathbb {R}}^{{\mathcal {C}}}_{+}$ and $C(\tilde{\varvec{\lambda }}^{\mathrm{G}}) \in {\mathbb {R}}^{ {\mathcal {G}} \times {\mathcal {T}}}$, load shedding cost $\mathbf{C} ^{\mathrm{sh}, \mathrm{E}} \in {\mathbb {R}}^{{\mathcal {T}}}_{+}$, fixed dispatch of explicit virtual bidders $\varvec{\Delta }{} \mathbf{v} ^{\mathrm{E}}_{r}$ achieved from (1) and fixed DA electricity market-clearing outcomes $\mathbf{p}$ and $\mathbf{u}$ obtained from (2), the RT electricity market clearing under scenario $\omega \in \Omega$ writes as

$$\begin{aligned}&\Bigg \{\mathop {\text {min}}\limits _{\begin{array}{c} \varvec{\Delta } \mathbf{p} _{\omega },\varvec{\Delta } \mathbf{u} _{\omega },\varvec{\Delta } \mathbf{s} _{\omega },\\ \varvec{\Delta } \mathbf{w} _{\omega },\varvec{\Delta } \mathbf{d} ^{\mathrm{E}}_{\omega } \end{array}} \,\,e^{\top } \varvec{\Delta } \mathbf{p} ^{\mathrm{C}}_{\omega } \ \mathbf{C} ^{\mathrm{E}} + e^{\top } \varvec{\Delta } \mathbf{p} ^{\mathrm{G}}_{\omega } \ C(\tilde{\varvec{\lambda }}^{\mathrm{G}}) \ e + e^{\top } \varvec{\Delta } \mathbf{s} _{\omega } \ e + \mathbf{C ^{\mathrm{sh}, \mathrm{E}}}^{\top } \ \varvec{\Delta } \mathbf{d} ^{\mathrm{E}}_{\omega } \end{aligned}$$

(3a)

$$\begin{aligned}&\qquad \! \quad \text {subject to}\ h(\varvec{\Delta } \mathbf{p} _{\omega },\varvec{\Delta } \mathbf{w} _{\omega },\varvec{\Delta } \mathbf{d} ^{\mathrm{E}}_{\omega },\varvec{\Delta } \mathbf{v} ^{\mathrm{E}}_{r}) = 0:\varvec{\lambda }^{\mathrm{RT,E}}_{\varvec{\omega }}, \end{aligned}$$

(3b)

$$\begin{aligned}&\qquad \qquad \qquad \qquad g(\varvec{\Delta } \mathbf{p} _{\omega },\varvec{\Delta } \mathbf{w} _{\omega }, \varvec{\Delta } \mathbf{d} ^{\mathrm{E}}_{\omega },\varvec{\Delta } \mathbf{u} _{\omega }, \varvec{\Delta } \mathbf{s} _{\omega },\mathbf{p} ,\mathbf{u} ) \le 0, \Bigg \} \ \forall \omega . \end{aligned}$$

(3c)

Note that (3), one per scenario, is a deterministic linear program. We denote by $\varvec{\Delta } \mathbf{p} _{\omega } \in {\mathbb {R}}^{{\mathcal {T}} \times {\mathcal {I}}}$ the power adjustment of conventional generators. In addition, $\varvec{\Delta } \mathbf{u} _{\omega } \in {\mathbb {R}}^{{\mathcal {T}} \times {\mathcal {F}}}$ and $\varvec{\Delta } \mathbf{s} _{\omega } \in {\mathbb {R}}^{{\mathcal {T}} \times {\mathcal {F}}}$ refer to the adjusted relaxed commitment decision and the adjusted start-up cost of fast-starting units, respectively. Wind spillage and load shedding actions are denoted by $\varvec{\Delta } \mathbf{w} _{\omega } \in {\mathbb {R}}^{{\mathcal {T}} \times {\mathcal {J}}}_{+}$ and $\varvec{\Delta } \mathbf{d} ^{\mathrm{E}}_{\omega } \in {\mathbb {R}}^{{\mathcal {T}}}_{+}$, respectively.

Objective function (3a) minimizes the total balancing cost for underlying scenario $\omega$. Equality constraint (3b) balances the wind power deviations in RT from the DA schedule with the position of virtual bidders $\varvec{\Delta }{} \mathbf{v} ^{\mathrm{E}}_{r}$ as fixed input. The dual variable vector $\varvec{\lambda }^{\mathrm{RT,E}}_{\varvec{\omega }} \in {\mathbb {R}}^{{\mathcal {T}}}$ represents the RT electricity prices under scenario $\omega$. Recall that this vector was exogenous in the problem of virtual bidders (1). Inequality constraints (3c) enforce lower and upper bounds on the load shedding and power adjustment of wind power units, conventional slow- and fast-starting generators, restrict the ramp-rate limits of conventional generators, enforce the adjusted unit commitment, and calculate the start-up cost for fast-starting units. The detailed representation of constraints is provided in the online appendix (Schwele et al. 2021).

3.1.4 Explicit natural gas virtual bidder

Similarly to the electricity VB, the profit-maximization problem of each explicit natural gas virtual bidder $q \in {\mathcal {Q}}$ participating in the natural gas DA and RT markets is given by the following two-stage stochastic linear program:

$$\begin{aligned}&\Bigg \{\mathop {\text {max}}\limits _\mathbf{v ^{\mathrm{G}}_{q},\varvec{\Delta }{} \mathbf{v} ^{\mathrm{G}}_{q}} \ {\varvec{\lambda }^{\mathrm{DA,G}}}^\top \ \mathbf{v} ^{\mathrm{G}}_{q} + \left( \sum _{\omega } \pi _{\omega }\ \varvec{\lambda }^{\mathrm{RT,G}}_{\varvec{\omega }}\right) ^\top \varvec{\Delta } \mathbf{v} ^{\mathrm{G}}_{q} \end{aligned}$$

(4a)

$$\begin{aligned}&\!\!\text {subject to}\ \mathbf{v} ^{\mathrm{G}}_{q} + \varvec{\Delta }{} \mathbf{v} ^{\mathrm{G}}_{q} = 0 \Bigg \} \ \forall {q}. \end{aligned}$$

(4b)

For given DA and RT natural gas market prices $\varvec{\lambda }^{\mathrm{DA,G}} \in {\mathbb {R}}^{{\mathcal {T}}}$ and $\varvec{\lambda }^{\mathrm{RT,G}}_{\varvec{\omega }} \in {\mathbb {R}}^{{\mathcal {T}}}, \ \omega \in \Omega$, the virtual bidder solves (4) to maximize her expected profit stemming from the price differences in DA and RT natural gas markets. Her decision variables are DA positions, i.e., $\mathbf{v} ^{\mathrm{G}}_{q} \in {\mathbb {R}}^{{\mathcal {T}}}$ and RT positions, i.e., $\varvec{\Delta }{} \mathbf{v} ^{\mathrm{G}}_{q} \in {\mathbb {R}}^{{\mathcal {T}}}$. Recall that we assume that the virtual bidder has a perfect foresight into future DA and distribution of RT prices over scenarios. Equality constraint (4b) zeros out the DA and RT trades of the explicit virtual bidder. As an important observation, this explicit virtual bidder enforces the DA and the expected RT natural gas prices to be equal, i.e., $\varvec{\lambda }^{\mathrm{DA,G}}=\sum _{\omega } \varvec{\lambda }^{\mathrm{RT,G}}_{\varvec{\omega }}$. This observation can be derived by the KKT optimality conditions associated with (4).

3.1.5 DA natural gas market

For given scheduled natural gas consumption of gas-fired generators as a function of $\mathbf{p} ^{\mathrm{G}}$ obtained from the DA electricity market (2) and the DA trade of virtual bidders $\mathbf{v} ^{\mathrm{G}}_{q}$ determined in (4), the natural gas market operator clears the DA market with ${\mathcal {K}}$ number of gas suppliers as

$$\begin{aligned}&\quad \mathop {\text {min}}\limits _\mathbf{g } \ \,\,e^\top \mathbf{C} ^{\mathrm{G}} \mathbf{g} \ e \end{aligned}$$

(5a)

$$\begin{aligned}&\!\!\!\!\!\!\text {subject to}\ h(\mathbf{g} ,\mathbf{p} ^{\mathrm{G}},\mathbf{v} ^{\mathrm{G}}_{q}) = 0: \varvec{\lambda }^{\mathrm{DA,G}} \end{aligned}$$

(5b)

$$\begin{aligned}&\qquad \qquad g(\mathbf{g} ) \le 0, \end{aligned}$$

(5c)

where (5) is a deterministic linear program. Parameters in the vector $\mathbf{C} ^{\mathrm{G}} \in {\mathbb {R}}^{{\mathcal {K}}}_{+}$ represent the supply cost of gas suppliers, and variables in the matrix $\mathbf{g} \in {\mathbb {R}}^{{\mathcal {T}} \times {\mathcal {K}}}_{+}$ are the DA schedule of those suppliers. Objective function (5a) minimizes the total gas supply cost. Equality constraint (5b) represents the DA natural gas supply balance with inelastic demand including given gas demand for power production and virtual trade $\mathbf{v} ^{\mathrm{G}}_{q}$. The “actual” natural gas prices are derived through dual variables $\varvec{\lambda }^{\mathrm{DA,G}} \in {\mathbb {R}}^{{\mathcal {T}}}$, which are not necessarily identical to the estimated prices $\tilde{\varvec{\lambda }}^{\mathrm{G}}$ used in the electricity market-clearing problems (2) and (3). Constraint (5c) enforces the lower and upper bounds on the gas supply. The detailed representation of constraints is provided in the online appendix (Schwele et al. 2021).

3.1.6 RT natural gas market

The natural gas operator clears the RT natural gas market to offset the change in fuel consumption of gas-fired generators $\varvec{\Delta } \mathbf{p} ^{\mathrm{G}}_{\omega }$ occurred under scenario $\omega$. This deterministic linear problem writes as

$$\begin{aligned}&\Bigg \{\mathop {\text {min}}\limits _{\varvec{\Delta } \mathbf{g} _{\omega },\varvec{\Delta } \mathbf{d} ^{\mathrm{G}}_{\omega }} \ \! e^\top \mathbf{C} ^{\mathrm{G}}\varvec{\Delta } \mathbf{g} _{\omega } \ e + \mathbf{C ^{\mathrm{sh,G}}}^\top \ \varvec{\Delta } \mathbf{d} ^{\mathrm{G}}_{\omega } \end{aligned}$$

(6a)

$$\begin{aligned}&\text {subject to}\ h(\varvec{\Delta } \mathbf{g} _{\omega },\varvec{\Delta } \mathbf{p} ^{\mathrm{G}}_{\omega },\varvec{\Delta } \mathbf{d} ^{\mathrm{G}}_{\omega }, \varvec{\Delta }{} \mathbf{v} ^{\mathrm{G}}_{q}) = 0:\varvec{\lambda }^{\mathrm{RT,G}}_{\varvec{\omega }} \end{aligned}$$

(6b)

$$\begin{aligned}&\qquad \qquad \quad g(\varvec{\Delta } \mathbf{g} _{\omega },\varvec{\Delta } \mathbf{d} ^{\mathrm{G}}_{\omega }, \mathbf{g} ) \le 0 \Bigg \} \ \forall \omega , \end{aligned}$$

(6c)

where objective function (6a) minimizes the total balancing cost. The first balancing action is gas supply adjustment $\varvec{\Delta } \mathbf{g} _{\omega } \in {\mathbb {R}}^{{\mathcal {T}} \times {\mathcal {K}}}$ whose cost is $\mathbf{C} ^{\mathrm{G}} \in {\mathbb {R}}^{{\mathcal {K}} \times {\mathcal {T}}}_{+}$. The second but extreme balancing action is the natural gas load shedding $\varvec{\Delta } \mathbf{d} ^{\mathrm{G}}_{\omega } \in {\mathbb {R}}^{{\mathcal {T}}}_{+}$ at the comparatively high cost of $\mathbf{C} ^{\mathrm{sh,G}} \in {\mathbb {R}}^{ {\mathcal {T}}}_{+}$. Equality constraint (6b) balances the gas supply adjustments in RT. The actual natural gas RT prices under scenario $\omega$ are the vector of dual variables $\varvec{\lambda }^{\mathrm{RT,G}}_{\varvec{\omega }} \in {\mathbb {R}}^{{\mathcal {T}}}$. Constraints (6c) enforce the lower and upper bounds on gas supply, gas adjustments and gas load shedding. The detailed representation of constraints is given in the online appendix (Schwele et al. 2021).

3.2 Analysis of stochastic equilibrium problems

In order to achieve temporal coordination, the profit-maximization problem of explicit virtual bidders as well as the DA and RT market-clearing optimization problems need to be solved simultaneously. Note that the explicit virtual bidders do not link the electricity and natural gas markets, but they will be linked later in Sect. 4 with implicit VB. For now, we can identify two stochastic equilibrium problems, one per energy sector. The first stochastic equilibrium problem related to the electricity sector includes optimization problems (1) $\forall {r}$, (2) and (3) $\forall {\omega }$. The second stochastic equilibrium problem corresponding to the natural gas sector consists of (4) $\forall {q}$, (5) and (6) $\forall {\omega }$. Note that these two stochastic equilibrium problems should be solved sequentially, i.e., one should first solve (1–3), and then for given natural gas demands, (4–6) can be solved.

Remark 1

Each linear optimization problem (2), (3), (5) and (6) related to DA and RT market-clearing problems can be equivalently reformulated as a pure Nash equilibrium problem, wherein price-taking agents maximize their profit in a perfectly competitive market.

The KKT optimality conditions of each optimization problem (2), (3), (5) and (6) and its corresponding pure Nash equilibrium problem are identical. As explained in Remark 1, each optimization problem (2), (3), (5) and (6) can be replaced by a set of optimization problems that constitute the corresponding Nash equilibrium problem. However, solving these problems simultaneously as the equilibrium problems (1–3) and (4–6) leads to coupled strategy sets and jeopardizes integrability of the equilibrium problem (Facchinei and Pang 2007).

Remark 2

Both stochastic equilibrium problems (1–3) and (4–6) are GNE problems.

In both stochastic equilibrium problems, the feasible set of some players depends on the decision of other players. We focus on equilibrium problem (1–3). The same discussion is also valid for the equilibrium problem (4–6). The trading decisions of electricity virtual bidders in (1), i.e., $\mathbf{v} ^{\mathrm{E}}_{r}$ and $\varvec{\Delta }{} \mathbf{v} ^{\mathrm{E}}_{r}$, appear within the power balance constraints in (2) and (3). Replacing (2) and (3) with their equivalent Nash equilibrium problems (as mentioned in Remark 1) will not change the GNE nature of the overall problem, as the DA power schedule of generators affects the feasible set of those generators in their RT problem. Note that (1–3) is a special GNE problem, since variables of (1) affect the feasible set of optimization problems (2) and (3), but not the other way around. In other words, the feasible set of optimization problem (1) is independent of DA and RT market outcomes. Similarly, the DA market outcomes in (2) impact the RT constraints in (3), but again not the other way around. One can interpret this linkage among (1), (2) and (3) in this way that there is no feedback among the feasible set of players. However, this specific linkage of optimization problems in (1)-(3) does not change the fact that it is a GNE problem.

The resulting challenge is that a GNE problem in general is formulated as a quasi-variational inequality (Pang and Fukushima 2005), which is generally hard to solve and admits multiple or even infinite solutions (Facchinei and Kanzow 2007). Note that Facchinei and Kanzow (2007), Harker and Pang (1990), Harker (1991), Schiro et al. (2013), Krawczyk (2007), Fukushima (2011) and Kulkarni and Shanbhag (2012) explore a specific class of GNE problems with shared constraints. However, the coupling constraints in our proposed stochastic equilibrium problems, i.e., (1–3), and (4–6), are not shared constraints.

Remark 3

Existence of a solution to the proposed stochastic GNE problems can be mathematically proven under some circumstances.

The basis of this proof relies upon Harker (1991, Theorem 1) and Harker (1991, Theorem 2), provided that the feasible set of every agent in the GNE problems is non-empty, convex and compact. In our case, this condition will be fulfilled only if we assume bounds on market prices, i.e., by imposing price floors and caps, and bounds on virtual trades, e.g., by imposing a budget constraint for each virtual bidder. The investigation of solution uniqueness for these GNE problems is not straightforward (Harker 1991; Fukushima 2011).

4 Sectoral and temporal coordination

In order to enhance the sectoral coordination between electricity and natural gas markets, this section extends the model in Sect. 3 and allows natural gas-fired generators to act as implicit virtual bidders. In other words, they are allowed to self-schedule outside the markets to optimally allocate their operational flexibility in the electricity market and their fuel consumption in the natural gas market. Each self-scheduler, i.e., implicit virtual bidder⁶, maximizes her own expected profit. Similar to the explicit virtual bidders, we assume that each self-scheduler has a perfect foresight into DA and distribution of RT prices over scenarios in both electricity and natural gas markets. Note that including these self-schedulers in the model links the power and natural gas markets, so that a single stochastic equilibrium problem is achieved.

We consider both slow- and fast-starting types of gas-fired generators as potential self-schedulers. The difference between these two types of generators is that the slow-starting gas-fired units fix their unit commitment status in DA and cannot change it in the RT, while the fast-start units can. The expected profit maximization problem of each self-scheduling slow-starting gas-fired unit ${\mathcal {G}}\cap {\mathcal {S}}$ participating in both electricity and natural gas markets is

$$\begin{aligned}&\quad \mathop {\text {max}}\limits _\mathbf{p ,\mathbf{u} ,\mathbf{s} ,\varvec{\Delta } \mathbf{p} _{\omega }} \ \left( \varvec{\lambda }^{\mathrm{DA,E}} - C(\varvec{\lambda }^{\mathrm{DA,G}})\right) ^\top \mathbf{p} - e^\top \mathbf{s} \nonumber \\&\qquad \qquad \qquad \quad \!\!\!+ \sum _{\omega } \pi _{\omega } \Big [\left( \varvec{\lambda }^{\mathrm{RT,E}}_{\varvec{\omega }} -C(\varvec{\lambda }^{\mathrm{RT,G}}_{\varvec{\omega }}) \right) \Big ]^\top \varvec{\Delta } \mathbf{p} _{\omega } \end{aligned}$$

(7a)

$$\begin{aligned}&\qquad \!\!\text {subject to} \ g(\mathbf{p} ,\mathbf{u} ,\mathbf{s} ) \le 0:\varvec{\mu }, \end{aligned}$$

(7b)

$$\begin{aligned}&\qquad \qquad \qquad \quad \! g(\varvec{\Delta } \mathbf{p} _{\omega },\mathbf{p} ,\mathbf{u} ) \le 0: \varvec{\nu }_{\omega }, \ \forall \omega , \end{aligned}$$

(7c)

where (7) is a two-stage stochastic linear program, whose objective function (7a) maximizes the expected profit of the underlying self-scheduling gas-fired generator. Note that this objective function includes the actual DA and RT gas prices $\varvec{\lambda }^{\mathrm{DA,G}}$ and $\varvec{\lambda }^{\mathrm{RT,G}}_{\varvec{\omega }}$ from models (5) and (6), and not the estimated gas price $\tilde{\varvec{\lambda }}^{\mathrm{G}}$. This problem is subject to the DA (7b) and RT operational constraints (7c), so that the final production of gas-fired units in RT have to lie within their feasible operational limits.

Similarly, each fast-start self-scheduling gas-fired unit ${\mathcal {G}} \cap {\mathcal {F}}$ solves a two-stage stochastic linear program to maximize her expected profit as

$$\begin{aligned}&\quad \mathop {\text {max}}\limits _{\begin{array}{c} \mathbf{p} ,\mathbf{u} ,\mathbf{s} ,\varvec{\Delta } \mathbf{p} _{\omega },\\ \varvec{\Delta } \mathbf{u} _{\omega },\varvec{\Delta } \mathbf{s} _{\omega } \end{array}} \left( \varvec{\lambda }^{\mathrm{DA,E}} - C(\varvec{\lambda }^{\mathrm{DA,G}})\right) ^\top \mathbf{p} - e^\top \mathbf{s} \nonumber \\&\qquad \qquad \qquad \quad \!\!\! + \sum _{\omega } \pi _{\omega }\Big [ \left( \varvec{\lambda }^{\mathrm{RT,E}}_{\varvec{\omega }} - C(\varvec{\lambda }^{\mathrm{RT,G}}_{\varvec{\omega }}) \right) ^\top \varvec{\Delta } \mathbf{p} _{\omega } + e^\top \varvec{\Delta } \mathbf{s} _{\omega }\Big ] \end{aligned}$$

(8a)

$$\begin{aligned}&\qquad \!\!\text {subject to} \ g(\mathbf{p} ,\mathbf{u} ,\mathbf{s} ) \le 0:\varvec{\mu }, \end{aligned}$$

(8b)

$$\begin{aligned}&\qquad \qquad \qquad \quad \! g(\varvec{\Delta } \mathbf{p} _{\omega },\varvec{\Delta } \mathbf{u} _{\omega }, \varvec{\Delta } \mathbf{s} _{\omega },\mathbf{p} ,\mathbf{u} ) \le 0:\varvec{\nu }_{\omega }, \ \forall \omega . \end{aligned}$$

(8c)

The resulting stochastic GNE problem includes optimization problems (2), (3) $\forall {\omega }$, (5), (6) $\forall {\omega }$, (7) and (8). Note that in this stochastic equilibrium problem, the decisions of self-schedulers $\mathbf{p}$, and $\varvec{\Delta } \mathbf{p} _{\omega }$ in (7) and (8) are exogenous values within the market-clearing problems (2), (3), (5) and (6).

Remark 4

Let us consider a case with both implicit and explicit VB. If the dispatch of self-schedulers in DA is restricted by either (7b) or (8b), the stochastic equilibrium problem will be feasible if and only if such DA constraints are inactive. Any non-zero dual variable corresponding to the DA constraints of self-schedulers will make the stochastic equilibrium problem infeasible.

Including explicit and implicit VB requires solving (1–8) as a GNE problem by neglecting the operational bounds of self-schedulers in DA, i.e., (7b) and (8b). Self-schedulers can submit physical and virtual bids as long as their positions in RT adhere to their feasible operational limits, thus acting as implicit virtual bidders.

5 Ideal benchmark

We compare the proposed “soft” market-based mechanism for power and natural gas coordination to the ideal benchmark of a fully stochastic integrated energy market clearing. This ideal benchmark is indeed a disruptive solution to achieve a full temporal and sectoral coordination, which ignores the current market sequences. Assuming that the given set of scenarios is a good representation of the probability distribution of uncertainty, the stochastic market clearing efficiently makes informed DA decisions by anticipating the potential recourse actions in RT (Pritchard et al. 2010; Morales et al. 2012; Zakeri et al. 2019; Zavala et al. 2017). In this benchmark, the fully integrated power and natural gas system is co-optimized under complete exchange of operational information. The resulting two-stage stochastic linear program aims at minimizing the total expected operational cost of both sectors in DA and RT, and writes as

$$\begin{aligned}&\mathop {\text {min}}\limits _{\begin{array}{c} \mathbf{p} ,\mathbf{u} ,\mathbf{s} ,\mathbf{w} ,\mathbf{g} , \varvec{\Delta } \mathbf{p} _{\omega },\\ \varvec{\Delta } \mathbf{u} _{\omega },\varvec{\Delta } \mathbf{s} _{\omega }, \varvec{\Delta } \mathbf{w} _{\omega },\\ \varvec{\Delta } \mathbf{d} ^{\mathrm{E}}_{\omega },\varvec{\Delta } \mathbf{g} _{\omega },\varvec{\Delta } \mathbf{d} ^{\mathrm{G}}_{\omega } \end{array}} e^\top \mathbf{p} ^{\mathrm{C}} \mathbf{C} ^{\mathrm{E}} + e^\top \mathbf{s} \ e + e^\top \mathbf{C} ^{\mathrm{G}} \mathbf{g} \ e + \sum _{\omega } \pi _{\omega } \Big ( e^\top \varvec{\Delta } \mathbf{p} _{\omega } \mathbf{C} ^{\mathrm{E}} + e^\top \varvec{\Delta } \mathbf{s} _{\omega } \ e \nonumber \\&\qquad \qquad \qquad \qquad \!\!\!\!\! + \mathbf{C ^{\mathrm{sh,E}}}^\top \ \varvec{\Delta } \mathbf{d} ^{\mathrm{E}}_{\omega } +e^\top \mathbf{C} ^{\mathrm{G}} \varvec{\Delta } \mathbf{g} _{\omega } \ e +\mathbf{C ^{\mathrm{sh,G}}}^\top \ \varvec{\Delta } \mathbf{d} ^{\mathrm{G}}_{\omega } \Big ) \Big ] \end{aligned}$$

(9a)

$$\begin{aligned}&\text {subject to} \ (2{b}), (2{c}), (5{ b}), (5{c}), \end{aligned}$$

(9b)

$$\begin{aligned}&\qquad \qquad \quad (3{b}), (3{c}), (6{b}),(6{c}), \ \forall \omega . \end{aligned}$$

(9c)

Objective function (9a) minimizes the total DA system cost for power production and gas supply as well as the expected RT balancing costs in both sectors, while respecting the operational constraints in DA (9b) and in RT (9c) for each scenario. The stochastic optimization problem (9) can be equivalently reformulated as a pure Nash equilibrium problem, wherein each market player is a stochastic decision-maker, who maximizes her expected profit with respect to DA and RT operational constraints with perfect information regarding uncertainty and prices in both sectors.

Remark 5

The GNE problem (1–8) defined in Sect. 4 including explicit and implicit VB is not necessarily equal to the ideal benchmark (9), since their KKTs are different.

Recall that the GNE problem enforces convergence of DA and expected RT prices in both power and natural gas sectors through the optimality conditions of explicit virtual bidders. On the contrary, in the stochastic market clearing problem (9), the DA and RT prices converge in expectation only if all DA operational inequalities are non-binding, i.e., every market player acts as an unrestrained arbitrager between DA and RT markets. This can be easily explored by checking the KKT optimality conditions associated with (9). Note that this observation is valid under this circumstance that an operational constraint with the corresponding dual variable of equal to zero at the optimal point is necessarily non-binding. The co-optimization of power and natural gas system correctly accounts for the impact of natural gas prices on the merit order of the electricity supply curve. Allowing all gas-fired units to self-schedule in the sequential setup with perfect knowledge over both natural gas and electricity prices approximates system integration. This is further explored in the following proposition.

Proposition 1

If DA operational bounds on $\mathbf{p} ,\mathbf{u} ,\mathbf{w} ,\mathbf{g}$ in the stochastic optimization problem (9) are non-binding, the DA and the RT prices converge in expectation (i.e., $\varvec{\lambda }^{\mathrm{E,DA}} =\sum _{\omega }\pi _{\omega }\varvec{\lambda }^{\mathrm{E,RT}}_{\varvec{\omega }}$ and $\varvec{\lambda }^{\mathrm{G,DA}} =\sum _{\omega }\pi _{\omega } \varvec{\lambda }^{\mathrm{G,RT}}_{\varvec{\omega }}$) and the outcomes of (9) are equal to the GNE problem (1–8) when all gas-fired units are implicit virtual bidders.

Proof

This is proven by demonstrating that the KKT optimality conditions of the two problems above under the conditions mentioned are identical—See online appendix (Schwele et al. 2021) for more details. $\square$

Table 1 summarizes all models introduced. While sequential and ideal benchmark can be solved as linear programs (LP), all other models are recast as mixed complementarity problems (MCP) by concatenating all KKT conditions from the respective optimization models.

Table 1 Summary of market setup models

6 Numerical results

This section provides a case study to analyze and compare the proposed market setups presented in Sects. 3, 4 and 5, which are summarized in Table 1. We solve all models using an Intel CoreTM i7-7820HQ with four processors clocking at 2.70 GHz and 16 GB of RAM in GAMS using PATH and CPLEX solver for MCP and LP models, respectively. The CPU time for LP models is below 1 second, while that time for different MCPs varies between 1 and 800 seconds. See online appendix (Schwele et al. 2021) for further details.

6.1 Input data

This case study contains a power system with 6 non gas-fired generators (namely, ${\mathcal {C}}^1$ to ${\mathcal {C}}^6$) and 4 gas-fired generators (namely, ${\mathcal {G}}^1$ to ${\mathcal {G}}^4$). These gas-fired generators connect the power system to a natural gas system with four gas suppliers, namely ${\mathcal {K}}_1$ to ${\mathcal {K}}_4$. We consider a 24-hour time horizon. All technical details of generators and natural gas suppliers as well as the total hourly demand in both power and natural gas sectors are provided in the online appendix (Schwele et al. 2021). Note that the demand in both sectors is certain, and the only source of uncertainty is assumed to be the wind power. Wind forecast and scenarios are also given in the online appendix. The natural gas supply curve is shown in Fig. 6, which is the same throughout all 24 h. Figure 7 illustrates the shifting of the electricity merit order curve due to a potential change in the natural gas price. The reason for this shift is that the gas price affects the marginal production cost of the gas-fired generators. Since in both DA and RT stages, the electricity market is cleared before the natural gas market, the electricity market operator needs an estimation of the gas price. In the following, we assume that the electricity market operator uses the average gas supply cost, i.e., $2.5/kcf, as a deterministic and static estimation of the natural gas prices in both DA and RT. The value of lost load in the electricity and natural gas sectors are set to $600/MWh and $300/kcf, respectively. The wind power penetration, i.e., total wind power capacity installed divided by the total electricity demand, is 34%. The next subsections provide the market outcomes obtained from different setups.

Fig. 6

Natural gas supply function

Fig. 7

Electricity merit order depending on natural gas price. The plots on the left-hand, middle, and right-hand sides show the merit order corresponding to the low, average and high prices for natural gas (as illustrated in Fig. 6), respectively

6.2 Main results: total expected system cost

The total expected cost of electricity and natural gas systems achieved under different market setups is shown in Fig. 8. As expected, the highest system cost corresponds to the sequential setup Seq (first bar in Fig. 8), which is a fully uncoordinated model. On the other hand, the fully coordinated ideal model (i.e., last bar in Fig. 8) yields the lowest cost. In this case study, the full temporal and sectoral coordination results in a 7.06% cost reduction. The three proposed setups Seq+eVB, Seq+iVB and Seq+VB provide partial coordination, and therefore, the system cost achieved in those setups is between the upper and lower bounds. Among these three market setups, Seq+VB with both implicit and explicit VB yields the highest cost saving, which is 6.94% (fourth bar in Fig. 8). Out-of-sample simulation relaxes our assumptions of perfect knowledge of virtual bidders. An analysis of out-of-sample performance can be found in the online appendix (Schwele et al. 2021). In the following three subsections, we discuss in details how each market setup impacts the DA schedules. For clarity, we focus on DA dispatch of one of the slow-start gas-fired generators, i.e., ${\mathcal {G}}^4$, and analyze how each market setup affects her dispatch, and therefore her individual expected profit.

Fig. 8

Total expected cost of the electricity and natural gas systems calculated by (9a) under different market setups. The percentages show the reduction in the total expected system cost compared to that cost in the fully uncoordinated sequential setup (first bar)

Fig. 9

Hourly DA schedule of slow-start gas-fired generator ${\mathcal {G}}^4$ as well as DA and expected RT market-clearing prices obtained from fully uncoordinated sequential market setup Seq. The left- and right-hand side plots correspond to the electricity and natural gas market outcomes, respectively

6.3 Upper bound: Sequential market setup (Seq)

The corresponding market-clearing outcomes of the fully uncoordinated sequential market setup Seq are given in Fig. 9. The DA schedules in this setup have no foresight into uncertainty in the RT operation and sectoral interactions between the two systems. Thus, the DA and expected RT prices can significantly differ. An example of such case is the electricity market price during hours 14 to 22 in the left-hand side plot and the natural gas market price during hours 9 to 13 and 18 to 20 in the right-hand side plot of Fig. 9. The slow-start gas-fired generator ${\mathcal {G}}^4$ is dispatched in the DA electricity market myopically, without considering the volatility of the actual hourly natural gas price and the need for flexibility provided by ${\mathcal {G}}^4$ in RT. This generator is scheduled in hours 10 to 13 relying on the comparatively low estimated gas price, while her real production cost is higher due to comparatively high natural gas market prices. When power system flexibility is required, which is evident from the high expected RT electricity prices in hours 14 and 20, generator ${\mathcal {G}}^4$ is unable to provide upward adjustment since she is already dispatched at full capacity in DA. Apart from the high expected system cost, this inefficient DA dispatch results in a negative expected profit (-$529,059) for ${\mathcal {G}}^4$, as given in Table 2. The faulty estimation of natural gas prices when clearing the electricity market leads to underestimating power generation costs and overestimating the profits of ${\mathcal {G}}^4$ in RT, such that ${\mathcal {G}}^4$ actually operates at negative profits in RT, see Fig. 10. This illustrates the need for market coordination, and specifically the potential of scheduling power generators in DA more efficiently.

Table 2 Expected profit of each generator under different market setups

6.4 Lower bound: ideal benchmark (Ideal)

In this ideal stochastic co-optimization model, the DA decisions are made while perfectly foreseeing uncertainty in RT as well as the sectoral interdependencies. As given in Fig. 11, the DA and expected RT prices converge in both power and natural gas sectors. The fully efficient DA dispatch in this ideal market setup ends up to a non-negative expected profit for all generators (see Table 2), including ${\mathcal {G}}^4$ whose expected profit is $8833.

Fig. 10

Hourly profit in DA and in expectation in RT of slow-start gas-fired generator ${\mathcal {G}}^4$ obtained from fully uncoordinated sequential market setup Seq. The left-hand plot shows the estimated profits using natural gas price estimations while the actual profits for realized natural gas prices are depicted on the right-hand side

Fig. 11

Hourly DA schedule of slow-start gas-fired generator ${\mathcal {G}}^4$ as well as DA and expected RT market-clearing prices obtained from fully coordinated market setup Ideal. The left- and right-hand side plots correspond to the electricity and natural gas market outcomes, respectively

6.5 Temporal coordination: Seq+eVB

Fig. 12

Hourly DA schedule of explicit virtual bidder (i.e., the purely financial player) and slow-start gas-fired generator ${\mathcal {G}}^4$ as well as DA and expected RT market-clearing prices obtained from market setup Seq+eVB. The left- and right-hand side plots correspond to the electricity and natural gas market outcomes, respectively

Recall that the market setup Seq+eVB provides the DA-RT temporal (but not sectoral) coordination by allowing explicit VB in both electricity and natural gas markets. Note that it is sufficient to consider a single explicit virtual bidder only in each sector since the transmission network is not considered. The hourly amount of DA virtual bids in both sectors is shown in Fig. 12. The virtual bidders act as either buyers or sellers over the 24 hours in the DA market. For example, the virtual bidder in DA electricity market acts as a seller in hours 3–6, 10, 11, 20, and 24, while as a buyer in the rest of hours as illustrated in the left-hand plot of Fig. 12. The DA positions of this player are going to be zeroed out by her RT actions. Practically, this means that every MWh the virtual bidder sells in DA in hours 3–6, 10, 11, 20, and 24 will be bought back in the same hours in RT. The right-hand plot of Fig. 12 shows that in the DA natural gas market, the virtual bidder acts as a supplier in most hours. She behaves as a natural gas consumer only in hours 5, 10, 11 and 24. Note that allowing explicit VB achieves full convergence of DA and expected RT prices in both power and gas markets. Explicit VB also impacts the DA dispatch of generators. For example, the slow-start gas-fired generator ${\mathcal {G}}^4$ is no longer dispatched between hours 2 and 11, while she is fully dispatched in hours 13 to 22. Explicit VB alone decreases the total expected system cost, but to the disadvantage of several individual generators. For example, the expected profit most generators decreases compared to the fully coordinated sequential model and only gas-fired generators ${\mathcal {G}}^1$ and ${\mathcal {G}}^4$ are better off.

6.6 Temporal and sectoral coordination: Seq+iVB and Seq+VB

The efficient dispatch of market players operating on the interface of electricity and natural gas sectors can enhance the sectoral coordination. A foresighted schedule of gas-fired generators in the DA electricity market may improve not only the temporal coordination with the RT electricity market, but also the sectoral coordination with the DA natural gas market. We analyze below the two market setups Seq+iVB and Seq+VB separately.

Fig. 13

Hourly DA schedule of slow-start gas-fired generator ${\mathcal {G}}^4$ as well as DA and expected RT market-clearing prices obtained from market setup Seq+iVB. Generator ${\mathcal {G}}^4$ does self-scheduling. The left- and right-hand side plots correspond to the electricity and natural gas market outcomes, respectively

6.6.1 Self-scheduling gas-fired generators: Seq+iVB

As realized in the previous subsections, the DA dispatch of gas-fired generator ${\mathcal {G}}^4$ in setup Seq is inefficient, such that she ends up to a negative expected profit. This shows the significant potential for this generator to do self-schedule, rather than participating in the markets relied upon a deterministic sequential clearing procedure. Figure 13 shows the DA dispatch and market outcomes when generator ${\mathcal {G}}^4$ acts as an implicit virtual bidder. Note that in this setup, the implicit virtual bidder has to still respect her operational constraints in both DA and RT stages. This restriction will be relaxed later in setup Seq+VB. According to Fig. 13, generator ${\mathcal {G}}^4$ increases her production during hours 1 to 13 when the actual natural gas price is comparatively low, whereas she reduces her power production and consequently natural gas consumption when the gas price is comparatively high in hours 14 to 24. As presented in Fig. 14, allowing this gas-fired generator to self-schedule alone increases her expected profit to $11,415. Moreover, the total social welfare is improved in terms of reducing the non-negative expected profits for other generators and reducing the total expected system cost by 6.37% (third bar in Fig. 8). Another important observation is that the self-scheduling by ${\mathcal {G}}^4$ causes shrinking the price spread between DA and expected RT prices in both power and gas sectors.

Fig. 14

Hourly profit in DA and in expectation in RT of slow-start gas-fired generator ${\mathcal {G}}^4$ self-scheduling in the sequential market setup Seq+iVB

6.6.2 Explicit and implicit virtual bidding: Seq+VB

This setup allows explicit VB by purely financial players and implicit VB by gas-fired generator ${\mathcal {G}}^4$. Figure 15 shows that the explicit and implicit VBs together achieve full price convergence in expectation in both power and natural gas markets. When generator ${\mathcal {G}}^4$ is allowed to submit virtual bids in the electricity and natural gas markets, the amount of explicit virtual trade decreases significantly in the electricity market and almost disappears in the natural gas market compared to Fig. 12. Note that ${\mathcal {G}}^4$ extends her bidding behaviour in the DA electricity and natural gas markets beyond her operational constraints acting as an implicit virtual bidder. For example, virtual bids are submitted to act as an electricity consumer and natural gas producer in the DA markets, e.g., in hours 3, 4 and 9. More specifically, she bids in DA below her operational capacity in hours 3, 4 and 9 and above her capacity in hours 12, 13, and 19-21. The convergence of DA and expected RT prices indicates full temporal coordination. Moreover, the additional system cost reduction compared to the case with explicit VB only (see second and fourth bars in Fig. 8) suggests improved sectoral coordination. All generators can expect a non-negative expected profit in this market setup with both implicit and explicit VB. The implicit virtual bidder ${\mathcal {G}}^4$ expects to earn $8,319. Although this generator can extend her bidding activity beyond her operational constraints in DA, her expected profit is lower than that in a case when ${\mathcal {G}}^4$ is the only self-scheduler in the market setup without explicit VB (Seq+iVB). However, when explicit VB is allowed (Seq+iVB and Seq+VB), generator ${\mathcal {G}}^4$ is better off by submitting virtual bids, see Table 2.

Fig. 15

Hourly DA schedule of explicit (i.e., purely financial player) and implicit virtual bidder (i.e., generator ${\mathcal {G}}^4$) as well as DA and expected RT market-clearing prices obtained from market setup Seq+VB. The left- and right-hand side plots correspond to the electricity and natural gas market outcomes, respectively

6.7 Main observations

Based on the above results, allowing market players to arbitrage seems to enhance the coordination of sectors and trading floors. The inclusion of explicit VB results in generating better price signals that reflect the uncertainties inherent in the RT stages. These price signals improve DA schedules so that the existing flexibility is allocated and utilized more efficiently. The VB improves the temporal coordination of the sequential DA and RT markets in the electricity and natural gas sectors. The self-scheduling gas-fired generator strengthens the temporal coordination of DA and RT markets by decreasing the price spread and improves the sectoral coordination by making use of her superior information of natural gas prices. In the same manner, the implicit VB by gas-fired generators helps sectoral coordination between the electricity and natural gas markets and improves the temporal coordination between DA and RT markets. Such a gas-fired generator is able to arbitrage both between the trading floors and between the sectors by submitting virtual bids in the electricity and natural gas markets. That way the coordination between the sectors flourishes via better information exchange. More specifically, better price signals and improved DA schedules help allocate and utilize the existing flexibility more efficiently. The DA schedules are improved through bidding activities that better reflect the uncertainties and that take into account the interactions of power and gas sectors.

7 Conclusion

This work explores the capability of financial instruments via VB either by purely financial players (explicit VB) or by physical players like gas-fired generators (implicit VB) in improving the temporal and sectoral coordination in two-stage (DA and RT) electricity and natural gas markets under uncertainty. We use two models as benchmarks: a fully uncoordinated sequential model which achieves an upper bound for the total expected system cost, and a stochastic ideal co-optimization which provides full temporal and sectoral coordination and yields a lower bound for the total expected system cost. The resulting models with VB are equilibrium problems, including the deterministic market-clearing problems in DA and RT in both power and gas sectors, and the two-stage stochastic optimization problems of virtual bidders, who maximize their expected profit.

Our results reveal that competitive virtual bidders who have prefect insight into the probability distribution of RT prices in power and natural gas markets increase the efficiency of deterministic sequential markets, such that the resulting total expected system cost is between the lower and upper bounds. In our case study, it is illustrated that the inclusion of virtual bidding can result in an expected system cost that is very close to the lower bound. In particular, the explicit VB provides a temporal coordination of the DA and RT stages in power and natural gas markets. Moreover, implicit VB by gas-fired generators brings both temporal and sectoral coordination. This implies that the sequential market with VB may approximate the stochastic ideal integrated energy system, and help reveal and exploit the existing flexibility in the systems more efficiently.

The main policy implication is that a disruptive market re-design to a stochastic and integrated energy market might not be necessarily crucial for unlocking the existing flexibility. Instead, this can be done to some extent via financial instruments by allowing VB, while preserving the current sequential market-clearing setup.

As potential future works, it is of interest to consider virtual bidders with heterogeneous information and risk attitudes. It is also of interest to relax the assumption that explicit and implicit virtual bidders have perfect knowledge of the probability distribution of real-time prices. This requires modeling the potential information asymmetry in the equilibrium model (Lo Prete et al. 2019b; Dvorkin et al. 2019). In addition, it is important to analyze the cases where virtual bidders behave as strategic players (Kohansal et al. 2020). In particular, financial implications of risk attitudes and how strategic behaviour affects market outcomes should be studied. Additional market participants, e.g., demand-side flexibility providers, should be considered in future works, who have the potential to perform arbitrage by adapting their consumption levels. As another potential extension, the reserve market should be included to investigate how the procurement of operating reserve products (Cleland et al. 2015) affects the market outcome for virtual bidders and flexibility providers, and thus the temporal and sectoral coordination. One can also explore how the existence of diverse bidding formats, e.g., in the form of block bids, can impact the performance of virtual bidders. Additional sources of uncertainty and their potential correlation can also be considered. Another potential extension is to include network constraints, especially in the natural gas sector as it allows modeling linepack (stored gas in the pipelines). However, it will need either approximation (Correa-Posada and Sánchez-Martín 2015; Ordoudis et al. 2019) or relaxation (Borraz-Sánchez et al. 2016; Schwele et al. 2019) methods to convexify the linepack model. In particular, it is of interest to explore whether the existence of financial instruments impacts the way the renewable supply uncertainty is being propagated from the power sector to the natural gas network. In this line, a systemic risk analysis for integrated energy systems will be very relevant.

The proposed stochastic equilibrium model may become computationally hard to solve if more players and scenarios are considered, and thus more efficient solution techniques might be required. One potential solution can be distributed optimization by solving the problem as an iterative Walrasian auction, e.g., similar to the methods used in Höschle et al. (2018) and Mays et al. (2019). However, the potential challenge is that the underlying GNE problem may have multiple solutions, and this may affect the convergence of such an iterative approach.