Electricity Markets and Regulation

Abstract

The reliability and cost of electricity is critical to the success of modern economies. Infrastructure needed to provide the desired services requires large amounts of capital expenditure and governments are keen to ensure prices are kept to a minimum while maintaining the required reliability. To this end, electricity markets have been introduced to allow electricity generating and retailing companies to compete. At the same time regulations have been put in place to minimize anticompetitive activities and ensure that any monopoly services are charged at fair and reasonable rates.

This chapter explores the design and operation of electricity markets and regulation in relation to power systems. It briefly considers the evolution of the electricity industry and provides a general description of the range of market models used and the associated regulation applied to assist their efficient operation. Practical examples from various countries around the world are also provided.

The design and application of the various market models continues to evolve and it is recommended that the reader access current information to stay aware of the latest developments. A useful source is work by Study Committee C5, which can be found at the e-cigre web site (https://e-cigre.org). Other sites include those that have a regulatory or government sponsored oversight of the operations of the parties. Examples include Ofgem in the UK (https://www.ofgem.gov.uk/about-us/who-we-are/gas-and-electricity-markets-authority), ENTSOE in Europe (https://www.entsoe.eu), AEMC in Australia (https://www.aemc.gov.au) and FERC in the US (https://www.ferc.gov/market-oversight/market-oversight.asp).

One area that is in its infancy at the time of writing relates to developments such as micro-grids with distributed generation, battery storage and smart load control. These are now becoming viable and are likely to operate both as local micro-markets and as participants in the larger markets.

This chapter addresses the design and operation of markets and regulation in relation to power systems. The discussion will cover the topic at a reasonably high level and provide references where appropriate to a more detailed examination.

There are many texts that can provide a detailed description of markets, their structure and their benefits and failings. It is however useful to give a brief overview in order to set the context for a discussion of electricity markets.

Any mechanism that brings buyers and sellers together to trade by establishing prices for a quantity of product or service is a market. The concept of trading has been around since very early civilization. Trading can occur by direct negotiation between a seller and a buyer or an organized market where buyers and sellers compete to achieve a trade that produces the best commercial return.

Markets are beneficial when the competitive pressures on the players lead to higher efficiency than the alternative (e. g., a central utility) but only if the costs of running the market are less than the benefits gained. Markets need to be workably competitive to deliver value. When there is a risk of inefficient outcomes because of insufficient competition leading to the exercise of market power, regulators or governments often apply rules or place limits on commercial behaviors. It is always a difficult trade-off between rules to limit market behaviors and the reduced efficiency that can result. However, potentially distorted outcomes are better than allowing excessive market power. One way to achieve this balance is to limit the periods or conditions where one or more players can dominate but allow sufficient time to signal an opportunity for new investment. An often forgotten aspect of market power is that it can either raise or lower prices. For example, a strong player can force prices down until its competitors are forced out of the market. This enhances the market power of the strong player who is then free to raise prices.

Throughout history there have been examples of market failure. In recent times, the sub prime mortgage disaster that led to the global financial crisis and the collapse of Lehmann Brothers in the US are good examples [18.1]. An example related to electricity is the California market in the US where, amongst other things, market failure led to widespread interruptions to supply, bankruptcy of Enron and near bankruptcy of other utilities. This is discussed in more detail in the section on North America.

In some cases, a product or service can be a natural monopoly. For example, electricity network infrastructure can be viewed as a monopoly as it is normally difficult to duplicate. This is because of economic and environmental barriers for construction of a second transmission line in competition with an existing one. In these situations, prices, behaviors and service levels are often regulated, or otherwise managed. An industry regulator is often charged with overseeing the activities of monopolies but other solutions are also used such as by the auctioning of a license to build and operate the network infrastructure, as is done in Brazil.

Regulation often appears in conjunction with competitive arrangements in order to ensure some level of protection for the participants in the market. Electricity has been subjected to some form of regulation for many years, particularly in relation to transmission and distribution. This has included regulation of customer tariffs, monitoring and approval of expenditure and, various aspects of technical and safety performance.

In order to minimize conflicts of interest, transmission and distribution systems are often separated from generation and retailing. However, many arrangements have also evolved where distribution and retailing are combined, or remain combined when a previously vertically integrated utility is disaggregated. In these systems regulators often require the commercial arrangements to be ring-fenced from each other.

Third party access to transmission and distribution is central to competitive electricity markets. Access involves a set of rights and obligations that allow new entrants to connect to the network in order to compete. Over the last 30 years, access and market or competition-oriented reforms have been introduced to the electricity sector in many countries.

This chapter examines the foundations of the electricity industry and how competitive markets can enhance its economic efficiency and therefore assist in minimizing costs to end users. The chapter introduces a number of the different forms of electricity trading that have emerged as the industry has matured technically and economically. The description starts by very briefly tracing the evolution of the industry from a small disaggregated service through to a large central commodity and most recently back towards a more disaggregated, market-oriented industry driven by dramatic changes in technology and cost. Current electricity markets have evolved around larger power systems and the discussion that follows will mainly focus on these. Developments such as microgrids with distributed generation, battery storage and smart load control are now becoming viable and it is likely that these systems will participate in the larger markets and internal micro-markets will develop within the microgrids. A number of examples of markets that have been implemented across the world are briefly described, together with some particular challenges that have been faced as they have evolved. The second half of the chapter focuses on key aspects of regulation of electricity businesses, together with a high level description of the regulation implemented in a number of countries.

18.1 Electricity Industry Structure

Electricity has two challenging characteristics. Firstly there must be equilibrium between the quantity dispatched and that consumed by loads and storage mediums at any instant (net of losses incurred in transport across networks). Until recently the only loads that were material were those of end use customers, including relatively inflexible hydro pumped storage, and this required flexible generation facilities to track variations in customer load. More recently, large flexible chemical storage technology has emerged. As the following sections will explain, the development of flexible storage is reducing the need for flexible generation and also counteracting the introduction of intermittent generation technology. Secondly the laws of physics govern the physical operation of the power system and these determine the path electricity will flow, rather than any commercial or regulated mechanism.

Power systems are the interconnection of a number of generators and end users located over a wide area. The earliest power systems from the late 1800s and into the first half of the twentieth century were township systems with local generation. Over time the systems of separate townships were linked together to share costs with generation distributed across the townships and eventually connecting very large areas. In the second half of the twentieth century as the size of the consumer demand grew, larger and larger generators were built. Eventually most of the township generation was retired leading to the era of central generation. This development led to industry structures based on specific functions for generation, transmission, distribution and the interface with end consumers (retailing).

From the late twentieth century the cost of small and distributed generation fell dramatically reducing the benefits of scale and leading to a shift in the mix of generation back towards a combination of central and distributed generation. Much of the distributed generation of today utilizes wind and solar-photovoltaic (PV) technologies, which also bring benefits of low emissions of carbon dioxide. This shift is blurring the distinction between generation, transmission, distribution and retailing. For example, individual consumers may now host solar-PV on their household rooftop and export to their neighbors via the distribution network. Retailers may facilitate this transfer or simply be bypassed.

The newer technologies have different technical characteristics. Wind, solar-PV, wave and tidal generation technologies are generally non-synchronous in that they are usually connected via power electronics in contrast with spinning AC machines that are directly connected. They exhibit low (or no) inertia, which is important as this affects the rate at which power system frequency can deviate from the level needed for the power system to remain stable. Where the percentage of this form of generation is significant, it can change the dynamic performance of power systems. In some power systems, this shift is already requiring a major change in their operation. For example, where previously high fault currents due to short circuits may have restricted operation, the newer technologies create fault currents that may be too low to be detected by existing protection equipment.

Sitting between pure supply and pure consumption, large scale battery storage technology is reshaping the operation of power systems. Hydro pumped storage along with limited numbers of flywheels and compressed air energy storage have been in use for many years but limited by cost or suitable location. Recent technological developments and lower costs of chemical battery storage are allowing deployment of storage devices across the entire supply chain. Batteries are being installed in conjunction with intermittent technologies connected to major transmission facilities and behind the meter in households to better match the requirements of end customer demand. Storage devices such as batteries or flywheels can also counteract the effects of low inertia from other new technologies. While the cost and performance of batteries are constantly improving, they are currently more suitable for short term back up which allows time for longer-term support such as gas turbines or hydro storage to be brought into service.

Rooftop solar-PV, micro-generators and some small wind farms are typical examples of small generators, embedded in the distribution system. As the development of these technologies grows quite rapidly around the world, their effect is becoming more pronounced. For example, large-scale application of rooftop solar-PV may reduce the power flows from higher voltage levels of the system at a particular substation to zero. It could even result in power flows from lower to higher voltage levels of the system at the substation. Power flows can also swing dramatically as intermittent clouds appear and screen the PV cells. While, in parts of the world the majority of current rooftop solar-PVs are not yet metered, recent technological innovations in measurement, communications and trading, including through the application of block-chain concepts, are resulting in these widely distributed resources being more closely monitored and controlled. This also makes possible peer to peer trading arrangements with potentially strong effects on the physical and commercial operation of the broader system and any market operating on it.

An extreme example of the variability of solar can be seen from the impact of the solar eclipse that occurred in Europe in March 2015.

At the time of the eclipse the installed solar-PV capacity in the Continental European synchronous system was approximately 89 GW, and the potential solar-PV reduction was expected to be as high as 34 GW. The effect in Germany is shown in Fig. 18.1. The gradient of the steep ramp that shows the sudden loss of solar-PV generation was estimated to be 2–4 times higher than normal daily ramping, while the short morning peak caused a significant increase in demand for flexible power plants. While the actual reduction in solar-PV output was lower on the day (19.2 GW), due to more than expected cloud cover, there was still a substantial change. The event was anticipated, however, and adequate reserves were scheduled, ensuring there was no risk to power system security. The key points are that there needs to be adequate back up to cover the significant loss of generation that can occur and the solar event may cause a much more rapid ramp rate than would normally occur.

Fig. 18.1

Impact of solar eclipse on PV generation in Germany, 20.03.2015. (reprinted by permission, © Amprion)

Utilities in the US also undertook contingency planning and scheduled additional generation for the solar eclipse in the US in August 2017. However, it had a fairly minimal impact, partly due to the lower than expected temperatures and some cloud cover [18.2].

Large-scale wind and solar-PV farms have also been constructed to take advantage of economies of scale and the benefits of being close to strong wind and solar resources. While it is possible to predict the timing and size of the outputs of these intermittent sources over the short term with reasonable accuracy, wind strength does not always align with the demand of customers. In addition, local cloud cover can affect the output of large-scale solar facilities (which is less of an issue with distributed small-scale solar-PV). Fast acting and flexible generation, storage systems and transmission interconnectors can be used to counter the effect of variations (and any inaccuracies). Renewable energy technologies with substantial inherent storage, in particular solar-thermal, have a dispatchability advantage over technologies without storage. Solar-thermal also brings significant inertia, accentuating its advantage, although at significantly higher cost for now.

18.2 Industry Organization

While the physics of the electricity supply process is the same everywhere, there are many ways the industry can be organized commercially and functionally.

Historically, the dominant organizational model was a utility that owned all elements of the supply chain or contracted specific activities such as individual generators. The internal operations of these utilities were based on a management hierarchy. The entities typically had a franchise to supply end-consumers in defined geographic areas and government-appointed bodies regulated prices charged to end consumers. A variation of this model was where local supply boards took bulk supply from central generators.

In the early years, there was massive growth of the power systems driven by strong social objectives. Governments took a lead role; either directly through government owned entities or regulated private companies. As the systems grew, the efficient size of generating units also increased and the number of generating companies within a particular power system remained relatively small. In addition, the IT and communications systems were not yet suitable to host real time competition. These circumstances tended to make monopoly ownership and control of generation the natural path.

As the power systems evolved, major utilities interconnected their networks forming large power systems across different government boundaries and between countries, continuing the earlier pattern of interconnection between townships. Interconnected utilities established agreements and sometimes, common control facilities to coordinate the operation of their power systems.

Since the early 1990s, many utility-based arrangements have been amended to introduce third party access to the networks and competitive market mechanisms with the aim of reducing the cost to the end customer by:

• Providing end consumers with a choice of supplier

• Introducing short term competition into day-to-day operation of power systems and, in particular

• Devolving many decisions including investment decisions to individual generators, retailers and customers, i. e., away from a command and control based central management by system controllers.

The short-term competitive day-to-day operations of power systems have been enabled by advances in communication, supervisory control and data acquisition technologies that have allowed the vast amount of data needed to manage a power system this way. The devolution of decisions required a dramatic move away from the centralized hierarchy model of managing a power system. In particular, the power system operators became managers of system security and service providers to competing generation and retailing businesses, which all depend on the neutrality of the network businesses to reach their customers.

18.3 Electricity Markets

Specific technical characteristics of electricity are that it is invisible and, when supplied via a shared alternating current network, it is also indivisible. In addition, supply and demand must be balanced on a second-by-second basis. Together, these factors restrict the form a market can take for trading of the physical product. Forward sales for electricity, however, can be transacted much like any other commodity. For example, this could be in a financial exchange or by direct negotiation in over the counter sales. The need to balance real time supply and demand will generally mean delivery does not perfectly match forward sale volumes and some form of real time balancing or spot market is needed in parallel with the forward market.

In order for disaggregated entities to collectively make economically efficient decisions that also maintain security and reliability, they require timely information about supply, demand and cost. A number of market models have evolved to provide a platform for a central market and system operators have been allocated the task of receiving and publishing this information. The various models employ a different mix of central control and disaggregated decision-making, where the separate generators and retailers are responsible for their own commercial wellbeing.

18.3.1 Supply and Demand

To run a power system effectively there needs to be adequate capacity to meet total demand and an ability to match variations in demand across each day. The total customer demand varies across the day and across the year in most systems. The variation is generally more pronounced in countries located away from the equator and varies with the season and use of heating and cooling. Figure 18.2 presents a typical winter demand curve for a day in a cool climate and shows overnight load is low and there is a peak that occurs for a short period in the early evening, generally driven by domestic cooking, lighting and in some cases, heating.

Fig. 18.2

A cool climate winter daily demand curve

Traditionally, generation plant has been characterized by the role it plays. Generation plant that runs most hours of the day is termed base load. Plant that is used only occasionally, or for limited hours per day to follow variations in demand, is termed peaking plant, with intermediate load plant running for part of the time. The most cost effective mix of technologies will depend on the capital and operating cost of available technologies and fuels and can only be assessed over the long term. Decisions about which technology is best to run within a day depend on the controllable cost of operation, the variable operating cost of plant that has been built (i. e., excluding sunk capital cost) and the shape of the daily demand curve. The demand in Fig. 18.2 would be met using the lowest cost plant across the entire day (making this plant base load) and higher cost plant for less time during the peaks.

Historically, in large power systems, coal, nuclear, hydro or combined cycle gas fired generation generally operated near to 24 h a day (base load). These generators typically have high capital costs and low operating costs and, with the possible exception of hydro, are not designed to be turned on and off on a regular basis. On the other hand, peaking plant such as open cycle gas turbines have a much lower capital cost but higher operating costs and are much easier to turn on and off. In a market system where generators offer a price to the system operator for dispatch you would expect the lowest cost generators to offer low prices at low load times to ensure that they run all the time.

More recently there has been a substantial increase in wind and solar generation in many countries. The operating costs of these technologies are low as there is no fuel cost. So, they will want to run as often as the wind blows or the sun shines. In bid based market arrangements they will also tend to bid lower prices to ensure they run when they can. This will be in direct competition with traditional base load plants and will put pressure on their revenue streams. While this may drive efficiency, it can also impact on the overall financial viability of the generator.

As the penetration of wind and solar generation continues to increase, markets have to increase flexibility to balance surpluses and deficits both within a particular power system and across the interconnections between the systems. In many cases this will be across international borders and this is driving the need to standardize market structures and systems where these interconnections can and do occur.

Variable operating and maintenance costs of offshore wind farms can be more substantial, and potentially be above the prevailing market price. Operators should therefore aim to run the wind farms as much as possible whenever the market price is above their variable operating and maintenance costs, while scheduling outages at times that have the lowest impact on revenue.

As discussed above, the growth of distributed generation, mostly in the form of rooftop solar-PVs and, more recently, associated storage, is opening up opportunities for customers to participate in the market, both in terms of sale of surplus generation and demand management.

18.3.2 The Role of Markets

A market is only useful if it ensures generation capacity is introduced and withdrawn in a timely manner. Markets may signal the right time through prices rising or falling as demand grows. For example, higher prices will encourage higher cost generators to run more often or, if a new lower cost source of power is available, it may undercut operation of an incumbent technology. On the other hand, reducing customer demand may cause prices to fall. Entry and exit of capacity may also be centrally managed in terms of price or volume and market arrangements will focus on efficient use of incumbent capacity. Given the significance of electricity in most countries, if new entry does not occur in a timely manner, reliability of supply will fall suggesting market failure and this will usually lead to government intervention.

A centrally managed process to determine the level of capacity may be used where markets pay for capacity or availability, separately to dispatched energy. In this case, generators are paid to guarantee that their plant will be available at times of system stress even if they otherwise do not run often or at all.

In order to design or understand a market for electricity, it is important to be clear about the definition of the product or products that are being traded, be it capacity, energy, availability, frequency keeping services or network support. Electricity systems are highly integrated and market arrangements for one or more products generally affect others. As noted, transmission and distribution networks are often regulated, even if there is a market in operation for other parts of the industry such as generators and retailers. Network costs can be optimized, and therefore the combined costs of generation and networks charged through to users reduced, by lowering infrastructure investment to optimum levels and by leveling demand to reduce the size of peaks and troughs. Subsidies have been provided to some market participants as part of a (legitimate) social policy agenda of governments. A technology specific subsidy can also impact the operation of markets and ironically, market design activities usually need to consider how these non-market subsidies are applied.

There may also be constraints in the transmission and distribution systems, which may limit the amount of electricity that an individual generator can sell. These constraints can impact on the returns to the affected generators as well as the reliability of supply for some end customers. Examples of these constraints are thermal capacity limitations or operational restrictions in relation to system security or stability. In a market situation these constraints need to be clearly identified to minimize the likelihood of contractual disputes at a later date.

In summary, the overall goal of a market is to achieve market efficiency in both the short term and the long term. In the short term, output should be produced in the right quantities by the lowest cost generators and consumed by those that are prepared to pay the most. In order to achieve this, generators are dispatched to achieve the best economic outcome starting with the lowest short run marginal cost. In the long term the market must provide signals to trigger the timely investment in appropriate quantities of new generation. It must also generate confidence that investors will receive a reasonable return on their investment considering the risks involved.

18.4 Market Models

This section introduces a number of the market models that have evolved around the world. Specifically, the following models are described:

• A vertically integrated utility

• A vertically and horizontally disaggregated utility with only utility plant connected

• A competitive market only at the time of investment

• Separate capacity management combined with central dispatch of energy

• A general overview of energy-only markets and specific descriptions of:

  • A pool system with a marginal system price

  • A pool system with pay as bid.

The section then continues to discuss the operation of retail competition and a brief overview of using a market for transmission expansion.

One way to understand the differences in the models is to consider where each sits on a spectrum of central disaggregated control and the allocation of risk. This is illustrated in Table 18.1. Generally, the more centralized the control the more the risk is retained by central entities and ultimately passed through to the end consumers in the form of fees and tariffs. Conversely, the more devolved the decision-making; the more the individual generators and retailers carry the commercial risks (and rewards) of decisions in the industry.

Table 18.1 Spectrum of market options

Prior to the introduction of markets, the traditional function of a system operator was to manage system security, physical dispatch, system reserves and network loading. To manage the commercial interactions with buyers and sellers a new function emerged – market operator or power exchange. Around the world the activities involved in managing security, dispatch, network loading, network switching, metering, market registration and settlement are combined in a number of ways. A range of labels has emerged for the different activities and combinations, for example system operator, market operator or transmission system operator. The label independent is generally applied when independence of the system operator or transmission operator is required. For the market operator or power exchange, this level of independence is always assumed regardless of the label. The role varies from market to market but can be seen as the entity that runs the trading activities that are not handled by the traditional system operator.

18.4.1 Vertically Integrated Utility

The traditional utility, operated under a centralized hierarchy model, sits in column 1 at the left hand end of the spectrum in Table 18.1. In this model, individual generators are generally owned by or contracted to the utility and carry little risks other than for their own performance. Any errors in forecasting, in the choice of which units to dispatch, in the fuel to purchase or the trades with adjacent networks, stay with the central utility and are ultimately paid for by its owners – be they end consumers or private investors.

18.4.2 Vertically and Horizontally Disaggregated Utility with only Utility Plant Connected

The next step along the spectrum is in column 2 of Table 18.1. Here, vertical and horizontal disaggregation has been used to allow financial separation of the generators and network components of the electricity supply. However, there is no commercial market and the generators are still under direct management control or a form of transfer pricing is introduced.

18.4.3 Competitive Market at Time of Investment Only

This is illustrated in column 3 of Table 18.1 and can be described as a single-buyer/power purchase agreement () model. Traditionally, many utilities operated with a combination of owned generation and other contracted generators. In a single buyer model, competition occurs only at the time of contracting. In this case, the buyer would normally tender for a generator supply and enter into a long-term contract or PPA with the successful bidder. At the simplest level, the buyer may be the utility but there may be concerns that it has some level of vested interest in the outcome. A more sophisticated model is where the entity assigned the role of single buyer is commercially independent. It has no allegiance to any particular generation business. The single buyer will liaise with or be integrated with a market operator.

The model does not readily allow any choice of supplier to end consumers. In principle it does allow for multiple retailers to buy from the single buyer and compete on retail margin and service offering but the benefits of this form of competition are small. A single buyer is often seen as a transitionary step to greater levels of competition.

A variation on the single buyer is a PPA market where individual blocks of demand enter into contracts with generators under long term PPAs. This arrangement differs from a single buyer in that it is demand or customer focused as it is the demand side or customers who decide to purchase, typically in the form of large industrial loads or retailers. A similar mechanism to a single buyer market is often used as a means to coordinate the operation of the contracted generators.

Single-buyer/PPA arrangements need a mechanism to commercially account for times when generators produce more or less than their contracted amounts and this is often termed out of balance. Out of balance can be calculated and priced in a number of ways. For example, all contracted generators may be required to inform the market operator of their intended outputs in advance. Uncontracted or partially contracted sellers may be required to bid to the market operator their willingness to raise or lower output at various quantities and prices. Depending on the needs of the balancing market and the competitiveness of their bids, these uncontracted or partially contracted sellers may or may not physically run.

As the arrangements become more sophisticated, they begin to look more and more like the shared or pooled price arrangements that sit further along the spectrum of industry structures.

18.4.4 Separate Capacity Management Combined with Central Dispatch of Energy

A widely used structure for competitive participation provides for a payment for capacity separate from payment for dispatched energy. This arrangement is often described as a capacity market but is more correctly a capacity plus energy structure. It falls within column 4 of the spectrum in Table 18.1. The arrangement reflects the typical structure of earlier PPA contracts that cover availability plus dispatch payments and provide for close management of the level of capacity present in the market [18.3].

In these market arrangements, one (but rarely both) of the amount of capacity or the price of capacity is set administratively. For example:

• The amount of capacity considered to be needed to meet a reliability standard is calculated and a competitive process run to acquire that amount, with the price being determined by the market responses; or alternatively

• The price that will be paid for capacity is announced and calls for parties to provide capacity at that price are made.

The acquisition process can be a tender, subscription or auction.

These markets typically arrange dispatch on the basis of prices submitted by generators rather than costs of production that are used in PPA and single buyer arrangements. Prices provide much greater flexibility and competitive tension in the market than a cost based regime. However, the typical objective is that sufficient competition will drive prices down to costs and, in the process, create incentives for an economically efficient mix of generation and continual pressure for improvement. Later sections discuss the common situation where this assumption about adequate competition is not valid and measures to control market power are overlaid on the operation of the market.

Typically a central or pool price is derived from the prices of the generators that are dispatched. This real time market price is most commonly set by the marginal value of generation to the system (often approximated by the price of the highest priced generation dispatched). All generation dispatched is paid at this market price.

Different implementations of this design include day-ahead, intra-day and longer-term markets, which set a price for agreed volumes of electricity in advance of dispatch. These markets can be for physical quantities or they can be financial contracts that settle against the real time price. The real time price is then applied only for the volumes not covered by advance markets.

The real time price can thus be seen to be equivalent to the out of balance price of PPA arrangements and the day-ahead markets or the price paid for unders and overs of actual compared to contracted volumes.

In these designs it is usual that a central government or other regulated body determines the amount of capacity or the price that is to be paid for capacity that is passed through to customers. Customers therefore bear the risk to the extent that the central body over or under-estimates demand, the flexibility or price elasticity of demand, or the availabilities of installed generation equipment.

18.4.5 Energy-Only Markets

At the right hand end of the spectrum in column 4 of Table 18.1, a market arrangement that pays only for energy dispatched on the basis of a price offered for dispatch transfers the commercial risk to the industry players. Critically, this relies on competition between these players to contain prices, as without sufficient competition there is a risk of uncompetitive prices. These markets have an energy market that operates along similar price based lines to that of a capacity market, but with quite different price outcomes. Generators earn revenue from the market only when dispatched. This type of market relies on price incentives to ensure sufficient capacity is available. As a result price must rise above the short run cost of generation for long enough for the efficient amount of generators to recover all costs. In principle the price will reflect the cost of generation and the risk of scarcity of supply to customers. When generation reserves are very low the price will approach or reach the value of customer scarcity. This price can be politically unacceptable prompting the use of low price caps but creating a risk of 'missing money', whereby potential investors do not have confidence that they will achieve an acceptable return on investment, aside from other risks such as plant performance and the level of demand.

Typically, day-ahead, intra-day and longer-term markets also operate in conjunction with the real time market.

In these designs, far more of the investment risk of the industry initially sits with generators. Generators and retailers may, however, enter into hedging or other longer term contracts to reallocate that risk in return for price certainty.

There are many variants of each of the basic forms. Within energy only markets, two specific examples are pool system with marginal system price and pool system with pay as bid.

18.4.5.1 Pool System with Marginal System Price

For this system, the sellers bid (confusingly analogous to offers to sell in most financial markets) to the market operator their willingness to sell various quantities at nominated prices. All the energy that they wish to sell is bid into the market and for this reason the market is sometimes described as a gross pool. One form of this design sees the bid quantities accumulated to form an aggregate supply curve. Buyers bid to the market operator their willingness to buy at various quantities and prices. These are accumulated to form an aggregate demand curve. This demand curve has traditionally been relatively inelastic as most customers are currently unable to respond to price signals. The quantity traded is at the intersection of these curves. The price is also at the intersection of these curves. The clearing price is the rate for all bidders for all quantities. This is illustrated in Fig. 18.3.

Fig. 18.3

Supply–demand curve

The system marginal price is the price set by the most expensive unit needed to meet the demand quantity. Only sellers that bid below this price will physically run.

A more sophisticated version is based on a mathematical optimization. Generators and any demand blocks that choose to participate in the wholesale market, offer their prices. The market operator then determines the economically efficient combination of generation and demand needed to meet customer needs, accounting for network losses and security constraints. The market price is one of the outputs of the mathematical optimization.

18.4.5.2 Pool System with Pay as Bid

Pay as bid arrangements differ from pay at marginal price described in the previous section in the way the market price is determined, but are otherwise very similar.

For this system, sellers bid to the market operator their willingness to sell at various quantities and prices. These are accumulated to form an aggregate supply curve. Buyers bid to the market operator their willingness to buy at various quantities and prices. These are accumulated to form an aggregate demand curve. The quantity traded is at the intersection of these curves. Only sellers that bid below this intersection physically run.

Each generator that is dispatched is paid at the price it offered and each block of load pays at the price they offered to buy at. Experience shows, and many commentators note, that in practice the price in pay as bid arrangements often trends towards the marginal price, as participants learn that their dispatch is unchanged until they bid above the marginal price.

18.4.6 Retail Competition

While an objective of wholesale market reform is to facilitate competition amongst suppliers, choice of supplier (retailer) for end customers is very often a key objective. The competition that then results between retailers can put downward pressure on profit margins and encourage the improvement of service standards. Retail margins are generally quite low and there is a need to spread the costs over several products with similar metering and billing requirements. Large companies can achieve economies of scale that tend to encourage the amalgamation of smaller retailers. However, too few retailers may hamper competition and lead to monopolistic behavior. Customers are given freedom of choice over their preferred retailer.

18.4.7 Market for Transmission Expansion

There have been attempts to use a market mechanism to drive down the costs of transmission infrastructure rather than to rely on regulatory oversight. An example of this is in Brazil. In this case, a planning process is used to assess future transmission requirements. The federal regulatory agency then holds auctions for bidders who will build, own and operate the facilities. The winner of this auction is the one who requires the lowest annual revenue over thirty years. While initially successful, recent auctions have failed to attract significant competitive bidders. To some extent this is due to regulatory uncertainty caused by rule changes and the forcing of companies who are renegotiating expiring contracts to substantially reduce prices by placing a ceiling on allowed revenues.

18.5 Market Structures Around the World

This section provides some practical examples of electricity markets around the world. It draws on a recent CIGRE technical brochure number 626 [18.4], which has surveyed the current status of electricity markets in selected countries. In some cases other sources from the various countries have also been used. These markets are evolving and some may look quite different in the future. They do, however, provide concrete examples that relate to the theory discussed earlier. One significant change that is occurring relates to the growing penetration of wind and solar generation coupled with growing opportunities for demand response, demand flexibility and energy storage. These present particular challenges to the current market arrangements due to the variable nature of wind and solar energy. Government policies and a range of direct and indirect subsidies are also driving the growth in these forms of generation.

18.5.1 Europe

While a form of trading of electricity commenced in Norway in 1971, for the rest of Europe, market liberalization began in England and Wales and the remaining Nordic countries in the early 1990s. These early markets, while still visible, are now incorporated into a pan-European market that covers energy and the implicit trading of network capacity.

European directives have been enacted since 1996 to establish a common internal electricity market in Europe, which is still evolving. These directives have established common rules for the generation, transmission and distribution of electricity both for the organization and functioning of the sector and for the operation of the market. Key outcomes have been:

• Unbundling of monopoly activities – i. e., separation of generation from transmission

• Third party access to all electricity networks

• Entry of new suppliers or load serving entities into markets

• Customer rights to choose their supplier

• Establishment of a National Regulatory Authority for each member state and

• Establishment of network codes or rules, derived from a common European root, which address issues such as connection to the network, operation of the market and operation of the power system.

In many cases, trading in electricity is by bilateral contracts, but in some cases, a market operator or power exchange has been created. These power exchanges take various forms across Europe, some of which are private entities. For example, the European Energy Exchange (EEX) in Leipzig, operates spot and derivatives markets and EPEX SPOT operates a spot market. German EEX AG and French Powernext SE own this latter market. There is a forward market for deliveries up to six years in advance and a spot market for day-ahead and intra-day trades. Buyers and suppliers submit their bids by midday on the day before as part of electricity auctions in the day-ahead market. After the day-ahead auction closes further bids can be made in the intra-day market. This trading closes 45 min before delivery. Finally, TSOs procure balancing capacity to cover unforeseeable imbalances through a competitive bidding process in a balancing market.

As these markets have evolved moves have been initiated to couple the various markets. In some cases this has been enhanced by the installation of new physical interconnections, for example across the Baltic and North Seas. The stronger these interconnections are, the less the physical constraints will impede the competitive pressures and the more opportunities there will be to dispatch the lowest cost generation. In the move to an integrated European market, work has been undertaken to develop a common approach to calculating cross-border transmission capacity, defining bidding areas and creating efficient trading mechanisms.

In May 2014, two regions, the north west region and the south west region were price coupled with common synchronized operation as shown in Fig. 18.4. The price coupling of regions () now includes Belgium, Denmark, Estonia, Finland, France, Germany, Austria, Great Britain, Latvia, Lithuania, Luxembourg, the Netherlands, Norway, Poland, Sweden, Italy, Bulgaria, Croatia, Portugal and Spain. The coupling of the day-ahead markets has enabled more efficient allocation of interconnection capacities of the involved countries. Market prices are calculated simultaneously, operational procedures are harmonized and offers are made to the market in a coordinated way. In addition, other countries are being brought into the PCR as soon as possible.

Fig. 18.4

Developments for price coupling of regions. (Source: ENTSOE, PCR Status report, Q3, 2017 [18.5])

18.5.1.1 Nordic Market Overview

A market for electricity trading in Norway commenced in 1971. This took the form of a power pool, which allowed generators to trade surpluses and deficits of power through a central grid. Virtually all power generation in Norway is by hydroelectricity whereas the other Scandinavian countries have a mix of hydro, nuclear, coal and other fuels.

Electricity market liberalization in the Nordic countries commenced in 1991 and now includes Norway, Sweden, Finland, Estonia, Latvia, Lithuania and Denmark. As part of the process, a regulator and a transmission system operator were established in each country. All networks were separated from competitive activities and are now open to third party access by all customers.

Trading of electricity is by direct physical contracts between generators, retailers and customers, with all having the option to access the power pool to trade at the margin. For the pool, buyers nominate the energy needed and the price they are prepared to pay each hour and sellers nominate how much they can deliver each hour and for how much. The hourly price is set for where the sell and buy prices meet. There is no capacity payment unless there is a transmission constraint. Within NordPool there is a spot market (ELSPOT) that settles quantities and price nine hours before delivery, taking into account interconnection limitations. There is also another spot market (ELBAS) that operates in Sweden and Finland and settles two hours before delivery. Both require physical delivery. Each transmission system operator operates a balancing market to enable physical balance in real time. There is an organised futures market for financial hedging up to three years in advance.

There is no capacity market although this may be considered in the future if there is concern that the market does not trigger timely investment in new generation. Stagnating load growth, a rapid increase in renewable generation and high penetration of hydro are mitigating the risk of insufficient generation investment. Most of the recent renewable generation has been intermittent wind generation, however, and the capability to manage dispatch is being compromised. While not a capacity market, Sweden maintains a strategic reserve. This is a capacity remuneration mechanism as discussed in more detail in the CIGRE technical brochure 647 [18.3], which describes the capacity mechanisms used in the various countries.

18.5.1.2 UK Market Overview

The United Kingdom (UK) comprises Great Britain (GB) and Northern Ireland. GB encompasses England, Wales and Scotland. The system in Northern Ireland became part of the single electricity market in 2007. Liberalization in GB was first instituted in 1990 with, initially; different market arrangements in Scotland separate from those in England and Wales.

Historically, fuel for power generation in GB has been dominated by coal, gas, and nuclear. Recent years have seen a growing percentage of renewable generation mostly in the form of wind turbines but also, increasingly, from solar PV. At the time of writing there are interconnections with France, the Netherlands and Ireland with further interconnections under development to Belgium and Norway and still others under consideration. In 2016, the GB system was run for some hours without any coal generation for the first time ever and coal fired power stations are predicted to close completely by 2025.

The first England and Wales market required all electricity to be bought and sold through a common pool. For each trading period there was a single input and a single output price. The pool input price was set by the last bid accepted to fill the quota of capacity for each half hour. Bidding into the market was optional. There was also a capacity payment made to encourage generators to offer capacity. This payment was based on estimates of the value of lost load () and loss of load probability (). Concerns with this mechanism included that it rewarded shortages rather than new investment and that generators could manipulate the pool price by withdrawing generation plant at critical times.

Scotland did not enter into this market. It retained two vertically integrated utilities with contracts in place between them to allow the mix of power plants to be optimum for both companies. Due to the lack of competitive pressure, the businesses were heavily regulated with prices linked to the England and Wales pool price. A similar process was adopted in Northern Ireland. The wholesale market in Scotland was merged with that in England and Wales in 2005 to form a single GB market with a single GB system operator adopting the structure that was put in place in England and Wales in 2001.

Amongst other things, concerns over excessive pool prices, potential gaming of the complex pool rules and the lack of demand side participation led the government to change the market. In march 2001, new electricity trading arrangements (NETA) were introduced. Under these new rules, encouragement was given to the decentralized trading of energy via bilateral contracts between generators and suppliers acting on behalf of end customers. To cover any remaining imbalances, there was also access to an optional power exchange to cover contractual imbalances and a balancing mechanism (). For the BM, participation is compulsory for larger generating units and suppliers and optional for others. The resulting market is similar to that operating in the Nordic power pool. The BM accepts bids and offers up to gate closure at 1 h ahead of delivery. The BM is managed by the system operator part of National Grid, which in 2018 is being separated from the transmission company that owns the transmission network in England and Wales. (There are two further transmission owners covering the north and south of Scotland).

The general view by Ofgem, the GB regulator, was that the NETA reforms had performed well alongside other factors such as falling fuel prices, a generous capacity margin and increased competition in generation with significant reductions in wholesale prices [18.6].

In 2013, the government introduced further reform with the objective of providing incentives to invest in secure, low carbon electricity; improving the security of Great Britain’s electricity supply; and improving affordability for customers. A key aspect of this reform was the introduction of a capacity market to ensure sufficient reliable capacity during periods of higher risk such as when high demand coincides with low wind power generation. The market operates as two auctions, one for year-ahead capacity and the other for capacity four years ahead. The successful bidders enter into capacity agreements that allocate payments for guaranteed capacity availability.

A further support to renewable generation providers is in the form of long term contracts for difference that provide stable and predictable revenue streams for investors. If the wholesale market price is higher than the contract price, the generator makes a payment and if the price is lower, it receives a payment.

The reform also introduced an offtaker of last resort scheme, which facilitates a contract between a renewable generator and a licensed supplier through a competitive auction where the contract must be at a discount below the market reference price. This process is used where the renewable generator cannot secure a contract via usual commercial means.

18.5.1.3 German Market Overview

The electricity market in Germany is Europe’s largest with an installed generation capacity of 184 GW. Full deregulation of the domestic electricity market occurred in 1998. There are four transmission companies that own and operate their respective transmission systems Amprion, ENBW Transportnetze, TenneT and 50 Hz transmission. Transmission is legally unbundled with defined rules for non-discriminatory third party access. Power generation and distribution is provided by a few large companies and a large number of small distribution companies. The larger companies are RWE, E.ON, innogy SE, Uniper, EnBW and Vattenfall.

As mentioned earlier, the German market is coupled with 15 neighboring countries and the exchange price for the day-ahead market is calculated jointly for the coupled markets. The coupled market determines the lowest price from any market zone until the cross-border interconnections are fully utilized and then it reverts to the lowest price in the particular market zone where the load is being served.

In June 2016, the German Federal Parliament approved a number of reforms to the power market and agreements about transferring existing lignite power stations into an emergency reserve. Ultimately these generators will be phased out permanently.

The main driver of the reforms is to ensure efficient power station operation and overall security of supply as the amount of renewable generation continues to grow. The main reforms include [18.7]:

• Strengthen the price signal to investors – in times of supply scarcity, peak prices will not be capped and may increase substantially which should signal investment opportunities to power station investors. They may choose to invest in peaking plant such as open cycle gas turbines, load management or storage.

• Penalties for insufficient power delivery – traders must ensure they buy sufficient power to supply their customers and not rely on grid operators to make up the difference.

• Increased competition for flexible power options or peaking plant – providers of flexible options, including electric vehicles in the future, will have access to the balancing market.

• Reduce network costs – firm access to all renewable generation will no longer be required and network charges across the country will be harmonized.

• Provide a capacity reserve – similar to the strategic reserve in Sweden, this will ensure adequate capacity is retained in constrained network regions and support power system security for extreme events. This plant will not be part of the power market to ensure there is no distortion of the competitive elements of the market.

• Mothballing of inefficient lignite power plants – these will be put on temporary standby for a period of four years before being permanently closed.

• Smart meters will be rolled out – this will commence with the largest customers and be governed by a cost benefit ceiling and strict data security.

18.5.2 Australia

The National Electricity Market (NEM) encompasses eastern and southern Australia (Fig. 18.5 [18.8]). It is geographically one of the largest interconnected AC systems in the world covering 5 200 km from tropical Queensland to western South Australia. The maximum demand of the NEM is now (2017) approximately 37 GW.

Fig. 18.5

The Australian national market (reprinted by permission from [18.8], ©AEMO 2016)

Economic reform of the electricity sector began in the early 1990s. Victoria was the first to establish a market within the state, followed by NSW and later still a form of market arrangement in South Australia. The states collaborated and designed the National Electricity Market (NEM), which commenced December 1998. The NEM covers all of the east coast of Australia but not Western Australia or the Northern Territory, which are not interconnected with the east coast system.

At the time of writing the NEM consists of five major and two minor transmission companies, seventeen distribution companies, twenty-five major and twenty-five smaller retailers and individual market customers as well as eighty-one generators.

The market design is similar in some ways to early versions of the UK model. The participants in the NEM are a mixture of private and government owned utilities. A key industry change was the disaggregation of generation, transmission, distribution and retail during the early 1990s in the lead up to the start of the NEM.

Western Australia followed later with disaggregation into generation, retail and a combined transmission distribution business occurring in 2006. More recently the disaggregated generator has been re-combined with the retailer and this larger business is intended to compete with other smaller combined generator retailers. The size of the market and the lack of diversity limit the extent of the competition that can occur.

The current market model in the NEM is consistent with the above description of a pool system with a marginal system price [18.9]. It is a gross pool, trading in energy only and it is broken down into regions, which follow state boundaries and each has its own spot price. There are no capacity payments. The intention is that as the spot prices increase, or rather forecasts of future spot prices increase, this signals to investors that the addition of new generation will be a profitable venture. While the visible trading in the market is seen as a spot price, behind the scenes there is extensive bilateral contracting to reduce the risk from short-term price variations. Today, many of the separate generation and retailing businesses have merged into a few very large gen-tailers. There is a reliability and emergency reserve trader, which is a function available to the market operator to contract for electricity reserves ahead of a period where there is a predicted shortage of generation. This is a form of strategic reserve as described for Sweden and Germany and discussed in more detail in [18.3].

In the NEM, the Australian energy market operator (AEMO) is both the market and system operator and is required to be independent of market participants. The market is intended to operate commercially at all times. AEMO has powers of direction to maintain reserve margins and ensure secure operation of the transmission network. There is also a spot market for frequency control services.

A snap shot of the data dashboard for Queensland in the Australian national electricity market is shown in Fig. 18.6 [18.10]. The chart shows 30 min data including current and historical spot prices, forecast spot prices, current and historical scheduled demand and forecast scheduled demand for a twenty-four hour period. Figure 18.6 is regularly updated. To obtain the latest figure, go to: https://www.aemo.com.au/energy-systems/electricity/national-electricity-market-nem/data-nem/data-dashboard-nem. Then select price and demand and QLD.

Fig. 18.6

30 min data for Queensland in the Australian national market: Electricity price and demand, Queensland, 01.06.2018 (reprinted by permission from [18.10], ©AEMO 2018)

The spot market is used to instantaneously match wholesale electricity supply and demand in real time. AEMO centrally coordinates the dispatch process and determines which generators will meet the demand based on 5 min bids from the generators for quantity and price. AEMO then arranges the dispatch of the generation accordingly.

Currently, every 5 min, participant bids are used to determine a dispatch price and the average of six consecutive bids is used to determine the spot price for each half hour of trade. Prices are submitted the day ahead for each trading interval but can be varied with around 10 min notice. The dispatch system assesses the demand and constraints at the start of each 5 min interval and then calculates the dispatch solution and price. The spot prices are used for financial settlement of all energy traded. From 2021 the spot price for settlement of trades will be moved to the 5 min price.

During times of generator shortage, where there may be inadequate reserves of generation to meet the customer demand, the bid prices may go very high. In order to protect the customers, the spot price is capped to nearly 15 000 $ (AUD)∕MWh (2019–20) [18.11]. This cap is periodically reviewed and is set to create incentives for sufficient capacity to meet the market reliability standard. A temporary lower cap is set if the spot price is consistently high over a rolling seven day period.

The NEM has now been in operation for nearly twenty years. It was introduced at a time of excess generation capacity in most regions followed by a period of stagnating demand coupled with a significant surge of investment in renewable generation in the form of wind and solar-PV.

As in most parts of the world, electricity is considered to be an essential commodity and loss of supply due to a major system blackout or a sudden surge in electricity prices are guaranteed to receive intense political focus. Both of these events coincided recently in Australia, mainly focused in South Australia, which experienced a statewide blackout at a time when electricity prices have been rising excessively. To some this is an indication that the electricity market is not working very well.

On Wednesday September 28, 2016 a major storm in South Ausralia, including two tornadoes, brought down transmission towers and caused a number of transmission lines to trip, including the primary interconnector to the adjoining state of Victoria. During this period, a large number of voltage dips caused a number of wind farms to power down. The sudden loss of this generation, coupled with the loss of transmission lines and the input from interconnection, caused frequency drops in excess of those allowed for in the under frequency load shedding schemes. This led to a state wide blackout. At the time of the event, wind power was supplying almost half of the state’s power needs.

Analysis of the event by AEMO [18.12] determined that settings on the wind farms coupled with a large number of network outages due to the storm exceeded the systems ability to deal with the resultant rapid fall in frequency. To some extent this can be remedied by modifying the wind farm settings. However, there is a longer-term issue developing due to the changes to the nature of the grid where less synchronous generation is leading to more periods with low inertia and low fault levels. This has been exacerbated by the unexpected size of withdrawal of older coal and gas fired generation. AEMO is recommending that frequency response services traditionally provided by synchronous generators should be procured from non-synchronous generators where feasible or from services such as demand response or synchronous compensators.

As a result of this blackout and the rapidly rising prices across the NEM, the government initiated a review into the future security of the NEM. This has led to the recent publication of the review report by the chief scientist in Australia, Dr. Alan Finkel. Of direct relevance to this chapter is a recommendation that all new generators must meet technical requirements to contribute to fast frequency response and system strength and that there must be a minimum level of inertia maintained within each region. In the future it recommends a move to a market based mechanism for procuring fast frequency response if there is a demonstrated benefit. It also recommends an orderly transition to a clean energy target via an emissions reduction trajectory. All large generators should provide at least three years notice of closure. The need for a strategic reserve to act as a safety net in exceptional circumstances should be considered. In addition, an integrated grid plan should be developed to promote the development of renewable energy zones. There should also be greater transparency and clarity associated with the setting of retail prices, together with stronger governance over the NEM. For consumers, issues associated with demand management, distributed energy resources and improved energy efficiency should be examined. If adopted the review panel believes its recommendations will lead to lower costs, greater security and reliability coupled with lower emissions. We can expect there to be a number of changes to the NEM over the next few years as the industry grapples with these recommendations.

Another recommendation of that report was to establish an energy security board (ESB) comprising the three peak regulatory bodies (the Australian Energy Market Commission – the rule maker, AEMO – the System and Market Operator and the Australian Energy Regulator – the economic regulator and market surveillance entity). The ESB proposed the development of a National Energy Guarantee (NEG) which would combine a requirement to contract capacity that is able to be dispatched with a requirement to contract low emissions plant. This simultaneous requirement on retailers was in response to reliability problems that were then plaguing the market. The problems had developed as a result of uncertainty and extended policy inertia that placed future investment at risk. The lack of investment hampered responses to the operating challenges of the NEM due to the change in technology mix in the market and the withdrawal of coal fired plant. In the event, only the obligation on retailers to contract plant that could be dispatched to meet their loads, the Retailer Reliability Obligation, was implemented (in July 2019). Mechanisms to limit emissions are still being debated by governments.

18.5.3 North America

Market liberalization in North America began when the US federal energy regulatory commission (FERC) mandated transmission open access across the US in the late 1990s. At this time, wholesale trading was permitted between all utilities with some basic rules governing use of the transmission system; there were no rules for an electricity market.

More formal markets began to evolve by the early 2000s with rules in place to govern how the trading should occur. In parallel with this, parts of Canada also established markets to allow trading with their US neighbors.

There are now nine markets as illustrated in Fig. 18.7. Each has an independent system operator (ISO) or regional transmission operator (RTO), which manage both the market trading and the system operation. These organizations are independent of the generator or network functions.

Fig. 18.7

Markets in North America (reprinted by permission from IRC)

18.5.3.1 Pennsylvania, New Jersey, Maryland (PJM)

The PJM interconnected power pool was formed in 1927 and is one of the oldest and largest centrally coordinated power systems in the world. It now operates as an independent regional transmission organization operating the wholesale electricity market and managing the high voltage transmission grid. In 1997, PJM began to expand from the long standing group of eight investor owned utilities and now has members operating in all or parts of Delaware, Illinois, Indiana, Kentucky, Maryland, Michigan, New Jersey, North Carolina, Ohio, Pennsylvania, Tennessee, Virginia, West Virginia, and the district of Columbia and supplying more than 65 million people.

In 2018, PJM had an installed generation capacity of 180.086 GW with approximately 10 GW of demand side management also available [18.13, 18.14]. PJM operates a day ahead spot market with energy only prices and a real time (5 min) market, also with energy only prices. The major portion of electricity is traded via bilateral contracts between generators and customers or is self generated and consumed within vertically integrated utilities. There is also an ancillary services market.

FERC oversees the PJM market and loads or retailers are required to have or contract for capacity plus a reserve margin or pay a penalty. There are a number of points of congestion across PJM and a system of financial transmission rights () is applied within a number of zones that allows market participants to cover potential losses related to delivering energy to the grid. The FTR holder collects revenue based on the day ahead hourly congestion price difference across an energy path. These FTRs are traded along with the energy produced.

18.5.3.2 California

It is informative to examine the market evolution in California, as there were some significant challenges with the initial set up. California started deregulation in March 1998 and intended to complete the transition by 2002. Before deregulation, eighty percent of electricity was supplied by three vertically integrated utilities. These were Pacific Gas and Electric (PG&E), Southern California Edison (SCE) and San Diego Gas and Electric (SDG&E). The remainder of power was supplied by a number of municipalities and public power utilities. At this time, peak demand was about 45 GW and installed generation was about 44 GW with the remainder being imported, mostly from the Pacific North West region. The generating resources are a mixture of gas, nuclear, hydroelectric, coal and renewables.

The California market was initially modeled on the one established in the UK and was established roughly at the same time as the one in Australia. As part of the market development, vertically integrated utilities had to divest most of their generation. There was a short-term day-ahead market, which closed sixteen hours before delivery and a day of market that closed eight hours before delivery. In addition there was a real time market for imbalances 45 min before delivery and an extensive ancillary services market.

About eighty-five percent of the energy was traded through the market. In the original design, most of the energy was bilaterally traded and settled via the power exchanges. Like PJM, there were two stages; the development of the dispatch schedules a day ahead and then the on the day balancing. Initially, only one power exchange was developed, CALPX, and so there was only one day-ahead schedule.

The day-ahead schedule was driven by the planned dispatch and usage of participants, which were lodged with CALPX – based on bilateral agreements and adjustments bid into the day-ahead process. A feasible solution was developed, with a day-ahead price, which was given to the operator. On the day the market ran, the programmed schedule was varied to match changes in load and generation.

There was a cap on the price for the day-ahead market, which was to be the main price. While there was no formal limitation on financial products for the balancing pool, the existence of the power exchanges meant that risk management was not required for the (expected) low amounts of energy to be settled in the pool.

A further complication was that there was a separation of the physical and market functions. An independent system operator (ISO) was established to deal with the actual technical management of the power system and generation dispatch and to financially deal with occasional imbalances in the real time market. The market functions were incorporated into a power exchange, which dealt with the actual financial energy transactions. The market only traded in energy and not capacity and there was a cap on the market price.

Concerns about pollution in California had increased the reliance on power imports from other states but the transmission infrastructure had not kept pace with this leading to transmission congestion with other states and within the state. Customers were allowed full retail choice from day one, however, some of the investments in generation were considered uneconomic in a competitive environment and the retail rates were frozen until the stranded assets were paid off. The general expectation was that lower prices would eventuate over time, particularly due to the entry of new high efficiency gas turbines. The pool was designed to be compulsory, where generators lodged hourly bids to meet demand at prices of their choosing and a common pool price was set by the last generator to be loaded in that hour.

There were a number of issues worth noting which would later prove to be of concern. None of the markets were physically binding. Ultimately, real time dispatch compliance was purely voluntary which resulted in difficulty maintaining system stability and reliability. There was a lack of a single unambiguous price for electricity to be used for hedging and investment decisions. There was some concern about the use of the fixed cap in that it may have sent adverse signals to participants for investment decisions and could also act as a target for bidders when prices are on the rise. There were long lead times on approvals for new transmission lines and generators.

California is a summer peaking area and in the 1998 and 1999 summers, temperatures were typically average. There was above average runoff into the hydroelectric dams of the Pacific North West and California was allowed to import low cost hydropower from these areas. The three major utilities PG&E, SCE and SDG&E were starting to divest their generators but still owned most of them. Natural gas prices were low to moderate, averaging around 2 $US∕GJ. Wholesale electricity prices generally remained in the 20–30 $US∕MWh range. Electricity pool prices were well below the levels of wholesale prices previously embedded into the tariffs that had been frozen for the major utilities. These utilities were therefore quite profitable at this time and were able to pay off the stranded asset charges more quickly than expected.

All this changed in the summer of 2000. Average temperatures were higher than normal with several heat waves. Water inflows in the Pacific North West returned to their long-term averages and reduced the amount of hydropower available. Natural gas prices increased to 10 $US∕GJ, a five fold increase and gas transport capacity reached its limits. In California, all gas fired generators needed to purchase nitrous oxide emission credits when operating and the price of those credits increased. The marginal cost of operation of an open cycle gas turbine exceeded 150 $US∕MWh. As a result of all these coincident events, price spikes began to occur in May and by June prices had reached record levels.

In reaction to these events, the California ISO lowered its wholesale price cap from 750 to 500 $US∕MWh and then a month later to 250 $US∕MWh. However, there was no noticeable effect on the pool prices. At this time, SDG&E had moved out of the rate freeze. The average monthly residential bill in San Diego went from 40 to 68 and then to 130 $US from June to July 2000—more than trebling over the summer months. General price levels throughout the western states, caused major energy intensive industries, including most of the aluminum smelters in the Pacific North West, to curtail output or cease production.

Not surprisingly, there was considerable anger amongst customers who forced the state government to cap residential prices around pre existing levels. Price caps were also implemented at the wholesale level leading to almost 12 billion $US of costs that retailers were unable to recover from customers. These retailers were forced into bankruptcy (or would have been without government support) while owners of generators reported record profits.

A number of factors in the design of the market can be identified as contributing to the surge in prices. There was limited demand side response. The rate freeze prior to the sudden increase in prices had given no signal to conserve energy or in fact develop strategies to manage varying prices. There was a sudden increase in power production costs coincident with an unexpected increase in demand due to unusually high temperatures. The fact that the three major utilities had to buy and sell all their energy needs through the California Power Exchange, coupled with restrictions on their ability to forward contract, removed their ability to mitigate price volatility. There was also withdrawal of large capacity from the market close to days of high prices indicating a possible abuse of market power. This last issue was a practice extensively used by Enron, an aggressive market trader at the time.

Traders looked for ways to maximize profits within a market structure that was new, complex and had little oversight or governance. Enron found a way to play the California power exchange off against the California ISO. The ISO was only meant to deal with occasional imbalances in a real time market. Daily schedules of load and generation were expected to be more or less equal. However, Enron and others would deliberately overstate expected customer loads, leading the ISO to pay a premium for delivering more power and a premium for removing load from the grid.

Ultimately Enron filed for bankruptcy in December 2001 becoming the largest bankruptcy in US history at that time. This was driven by illegal corporate practices and most of their top executives were tried for fraud after it was revealed that Enron's earnings had been overstated by hundreds of millions of dollars. The risks inherent in the market design had ultimately captured one of its most aggressive players who tried to hide their financial position with the hope that they could trade their way out of trouble.

A number of changes were put in place to address the problems. There was a move from compulsory spot market trading to bilateral trading as used in the other electricity markets in the US. FERC was given the authority to investigate abuse of market power where any bids over 150 $US∕MWh need to be justified and may be investigated. New generation was fast tracked. The oversight of the California ISO was changed from one representing stakeholder interests to an expert panel appointed by the governor. Finally, the California power exchange was dissolved and the market and operational roles were combined into the ISO.

18.5.4 Brazil Market Overview

Brazil has the largest electricity market in South America with a generation capacity of more than 137 GW, dominated by hydro power with most of the remainder supplied by fossil fuels, biomass and small amounts of wind and solar. There is limited interconnection with neighboring countries. Unbundling of generation, transmission, distribution and retailing occurred in the 1990s, although generators and retailers may be integrated. Third party access to the transmission is available to all parties on commercial terms.

There are two market environments. The first requires bilateral energy contracts to be established between generators and retailers that are considered to have captive customers. These are conducted by auctions in a regulated environment. The second allows unregulated contracts to be established between generators, uncontracted customers (generally if the load is greater than 3 MW), retailers and traders.

Generators, through long and medium term contracts, must supply 100% of the total demand of each retailer. A spot market allows for short term contracts to cover the difference between longer term contracts and real demand.

Auctions for new and existing generation capacity are carried out annually with new generation offered long term contracts. The regulator sets a ceiling price for these auctions.

There is a separate market operator and system operator. The market operator manages the long term and spot markets, with the marginal cost for the spot market provided by the system operator used as a reference for the spot market. The system operator is responsible for generation dispatch, power balance management and the coordination of ancillary services. The regulator oversees both operators.

There are four large subsystems as shown in Fig. 18.8 and there is a procedure to deal with transmission congestion within and between these regions where this impacts on lowest cost dispatch and contractual arrangements.

Fig. 18.8

The Brazilian subsystems and regions (reprinted by permission from [18.4], ©CIGRE 2015)

There are some specific market challenges that are currently being reviewed. These include the timely development of adequate transmission for sufficient generator competition and cross border flows. This is coupled with the growing interconnection of renewable energy sources, particularly when these occur as distributed generation that is not under the control and monitoring of the system operator.

18.6 Regulation

This section will examine the role of regulation in the operation of power systems.

18.6.1 Electricity as an Essential Service

Electricity is unique in that it must be available the instant it is required and must be consumed or stored the instant it is produced. Failure to meet these requirements results in a disruption to the power system where loads or generators may be tripped and, in the extreme, major power blackouts may occur. There have been examples of these blackouts in virtually all developed countries. They often lead to front-page news and government led inquiries.

There is no doubt that electricity is now considered an essential service in all developed economies. Frequent loss of power or loss of power to customers for extended periods can lead to significant economic loss and have major political implications. Customers rely on electricity in almost every aspect of daily life from lighting and air conditioning through to the operation of most appliances as well as most aspects of manufacturing. It is becoming more and more important for transport and electric vehicles are predicted to overtake petrol and diesel fuelled vehicles as the preferred form of transport within the next twenty years. A number of countries have announced plans to phase out petrol and diesel vehicles including France, Germany, UK, China, India and Norway. While the timing for some countries is still being finalized, some have already declared a target year. For example, France has recently announced that it will outlaw the sale of petrol or diesel vehicles by 2040 [18.15]. China is currently studying the timing of a move to phase out fossil fuel vehicles and has commenced using incentives and subsidies to guide this outcome. It is currently the world’s largest manufacturer of electric vehicles [18.16]. This has been driven partly by the growing smog problem in the major cities, partly by industrial policy and by commitments to acting on climate change.

Electric vehicles and a growing number of appliances rely on battery power as their energy source. While this is generally not the case at the moment, there is the opportunity to control the timing of charging of these devices so that the power demand is aligned to the capacity of the power system. There is also the possibility to use the electric vehicles as a source of ancillary services to the electricity system in the future. Accommodation of large scale charging of electric vehicles will require changes to industry operation practices and installed infrastructure capacity. A recent study by Vector in New Zealand estimates that power demand per dwelling could increase between 100% for slow trickle charging and 2000% for rapid charging [18.17]. Minimization of peak power increases and optimization of infrastructure usage will require a range of market design changes coupled with effective regulatory controls on both the utilities and the electric vehicle industry.

18.6.2 Vertical Integration and Disaggregation

As previously discussed, power system industry structures have evolved to take advantage of economies of scale, with large interconnected transmission systems allowing the shared use of generation reserves and larger, more efficient generators. In many cases this has resulted in vertically integrated business structures where the utility generates, transmits, distributes and sells the electricity. As discussed below, the introduction of competition between interconnected generators and also between retailers has placed more attention on the interconnecting networks. In some cases this has led to rules that financially ring fence the networks from the rest of the business. In others it has led to complete separation of the businesses.

18.6.3 Transmission and Distribution as Natural Monopolies

The transmission and distribution or network components of the utilities are considered natural monopolies as it is not economic to duplicate these components, and even if it were, it would be unlikely to be publicly or environmentally acceptable. As a result, the networks are in a privileged position and able to charge monopoly rents for the use of their services. It could be argued that too high a charge for service would result in the end customer going elsewhere for their power. However, this would almost certainly be at a much higher incremental cost and result in inefficient use of assets. Governments are well aware of this problem and in most countries have installed a regulator to oversee the investment in new assets and the allowed return on any investment in those assets. In other countries such as Brazil and Chile, competitive auctions are used to try to ensure the lowest cost provision, operation and maintenance of transmission infrastructure.

18.6.4 Economic Regulation of Monopolies

It is difficult for an external party to fully understand the internal costs of a utility or the link between investment and the quality of service provision. Attempts to build this understanding could lead to extensive duplication of resources, add significant costs to the process of oversight and therefore limit the benefits of regulation. The overall aim of regulation is to drive the business to deliver the desired quality of service at the lowest cost that is sustainable over the long term. Regulators therefore tend to focus on the outputs in terms of cost of service and delivery against prescribed service standards. At the simplest level, this may be a monitoring of the prices charged for the service, any annual increases relative to inflation and the performance against agreed service level targets. In addition, benchmarking with other similar organizations may be used to assess their relative efficiency.

While this may result in a lower cost of regulation, there may be a concern that the initial prices at the start of the regulation process may be too high. While benchmarking can help here, it is difficult to ensure a level playing field when comparing organizations. The historical timing of large investments and past decisions on technical issues are linked to the perspective on future conditions at the time the investment decisions were made. For example, an expectation of a high load growth may lead to a choice of a higher standard voltage level and the use of larger standard transformers. Accurate load forecasting is notoriously difficult and this may lead to the investment appearing inappropriate in hindsight. In addition, utilities vary in size and small utilities have fewer opportunities to enjoy the benefits of economies of scale.

Services are normally classified as regulated or non-regulated services. Regulated services are those not subject to adequate competition to drive lower prices and are normally classed as monopoly services. Conversely non-regulated services are those that are subject to competition (say by means of an auction) or can be provided by a number of entities. Classification of these services can be complicated further when the utility is vertically integrated. In this case it is possible for the utility to load additional costs into the network part of the business, which has the added benefit of allowing the utility to charge lower prices or make larger profits in the unregulated generation or network parts of the business. To some extent this can be overcome by requiring financial ring fencing of the regulated network components of the business but this still carries the risk associated with the regulator having less information on the internal workings of the business. In a number of cases, this has led to a requirement for the vertically integrated business to sell its network assets.

18.6.5 Rate of Return Regulation

For this form of regulation, the regulator scrutinizes the utility investment to ensure that the investment is required and efficient. It then allows sufficient revenue to ensure the utility fully recovers its costs. This form of investment is popular with investors as it allows a consistent return despite fluctuations in the economy. However, it does not provide strong incentives for the utility to operate efficiently.

18.6.6 Price Control

To reflect the information asymmetry between the regulator and the utility, regulators often choose to use incentives to drive the business to improve its efficiency. In this case, if the utility spends less than forecast for the regulatory period, it will be allowed to keep a portion of the difference between actual and forecast revenue as a reward. Conversely, if the utility spends more than forecast, a portion of the difference between actual and forecast revenue will be applied as a penalty. In both cases, this will then be taken into account when setting desired revenue and consequent price settings for the next regulatory period.

18.6.7 Revenue Determination

A more intrusive regulatory model is to require the network business to propose and justify its required revenue for a regulatory period that could typically be five years. This revenue would be determined based on the total anticipated efficient capital, operating and maintenance costs over the regulatory period together with asset depreciation, tax liabilities and a commercial return on the capital investment. To determine the capital costs, an asset valuation of all existing and proposed new primary and secondary assets would be required. Then a rate of return on this capital would need to be proposed based on identified risks and known costs of capital. This would typically be based on allowing a reasonable return on equity to reflect efficient equity finance costs and a return on debt that reflects the costs of regularly sourcing debt within the regulatory period. As part of the revenue determination model, revenue cap or price caps are usually applied for the regulatory period to apply incentives to the utilities and provide price certainty.

18.6.8 Revenue Cap

Revenue cap regulation fixes the revenue that the utility can receive regardless of actual output or demand. Prices are set based on the allowed revenue and forecast electricity consumption but if actual consumption differs from forecast consumption, the allowed revenue is corrected in the following year. There is therefore no incentive to incorrectly forecast the expected consumption. In this model, prices may be more volatile if demand is less predictable. It is also possible to apply an incentive in the form of CPI − X, where CPI is the general inflation rate in the economy and X is an estimation of the utility’s expected efficiency gains. This provides a very strong incentive to find efficiency gains.

18.6.9 Price Cap

For price cap regulation, prices are regulated rather than the revenue. The initial starting price is set and then it is adjusted each year by CPI − X as described for revenue cap regulation. If electricity consumption rises in the regulatory period, revenue will increase and if it falls, revenue will reduce. This should encourage the utility to adjust the timing of investments to suit changing load expectations and to drive efficiencies harder when consumption falls. On the other hand the X is known in advance of the regulatory period, which allows the utility to plan and deliver efficiencies over the period.

In reality there may be limited opportunities to take action on investment timing, as is the case for transmission, which usually has long lead-time projects. It is therefore more common to use price cap regulation for distribution investments, as these are usually more short term.

18.6.10 Pricing Methods

The subject of electricity pricing is complex and varies from country to country. This section will provide a high level overview of this topic. More detailed information can be found in the reference documents referred to in Sect. 18.7.

Ideally pricing should be efficient in that it recovers all the required revenue of the utilities while providing signals to customers that support the efficient use of electricity. Unfortunately, many practical issues may interfere to tarnish this objective. Governments are well aware that any large increase in electricity prices can cause headaches for them and therefore often place controls on the actual prices that can be applied. In some cases the government will subsidize the end price in order to deliver on social issues. In addition, ambitious renewable energy targets have been set by governments to combat climate change. To help increase the rate of installation of the renewable generators, subsidies have been applied either as payments for the surplus energy generated or by direct government payments that effectively reduce the capital cost of the renewable generator installation. As the efficiency of both the technologies involved and the production processes have improved and significant economies of scale have been achieved, the level of subsidy has been reduced.

The cost components of the pricing are driven by the wholesale prices from the generators, the network prices from the transmission and distribution companies and by a charge from the retailers. All of the participants will need to recover fixed and variable costs. However, accurately modeling the end charges on each of the total fixed and total variable costs will not necessarily produce the ideal result. For example, one could argue that the majority of costs incurred by a network business will be fixed costs as they are either incurred as a result of capital investment in the infrastructure or maintenance costs incurred to support the infrastructure. Once the investment decisions are made, neither of these costs varies over time. Network costs generally make up a proportion of between 30 and 50% of total costs.

Fossil-fuelled generators will have a higher proportion of variable costs. While they still need to recover the capital costs of the generator, the dominant cost will be the fuel. In the case of renewable generators such as wind and solar, the majority of the costs are fixed as the fuel is free leaving only maintenance as a variable cost. However, maintenance costs can be substantial for offshore wind turbines. Hydro is also dominated by fixed costs and nuclear is mostly a fixed cost, as it must continue to operate regardless of the demand from the end customer. If these costs were all passed through directly to the end consumer, the main component would be a fixed charge. There would then only be a small price signal to the end customer to vary their consumption at various times of the day.

variable prices can be used to encourage customers to apply demand management to reduce the peaks and troughs of daily load consumption. This should reduce the need for investment in new generation and network capacity as well as improve the utilization of existing infrastructure. Despite the difficulties in accurately providing cost reflective pricing, variability in pricing is a useful tool to encourage customer behavior that will lower overall costs.

Generators would prefer to operate at their maximum output whenever they are available, particularly in the case of base load generators such as coal, nuclear and combined cycle gas turbines. Open cycle gas turbines and large storage hydro generators are generally more flexible but are still keen to maximize their revenue, whereas solar and wind want to operate at their maximum possible output whenever the sun shines or the wind blows. In a market context, these parties are likely to price their output to ensure optimum output. This price will also be influenced by any longer term bilateral or hedging contracts that are in place.

End customers, on the other hand would like the electricity to be available at all times of the day and night at 100% reliability. They are however open to price signaling and, in many cases, would be prepared to vary their consumption to reduce their overall costs. This is becoming more and more possible with the advent of new technologies such as smart metering and remote appliance switching. Many household appliances are now available with batteries and some industries may be prepared to reduce demand occasionally for a fee.

Tariffs are determined by considering each class of customer. For example, domestic and other low use customers require more infrastructure and their peak loads are concentrated at times of the day that match system peak loads. On the other hand, industrial loads are generally fed by less network infrastructure and consume a more uniform load throughout the day. They can therefore be charged a lower tariff. Usually the tariff will have demand ($/kW) and energy (c/kWh) components with the demand charge higher for more peaky loads. Where meters have suitable capability, time of use tariffs are used to try to influence demand and reduce the overall system peak or to shift demand to times when renewable energy is abundant. Overall the setting of tariffs is partly political and social and is not an exact science. However, as technology improves, there are increasing opportunities to provide customers with more choice about the reliability and cost of their electricity and suppliers with a greater understanding of the impact of price signals.

Electricity pricing is set to try to achieve a balance between adequate signaling of the generator's desired mode of operation and meeting the end customer's needs for electricity availability at the best possible price. This applies regardless of whether the generators signal their requirements through a market structure or as regulated entities. Governments often intervene in the price setting to meet social objectives, particularly at the domestic customer level. For example, they may apply a uniform tariff across the entire customer supply area so that customers in the central business district pay the same price as those in remote country areas despite the fact that it costs much more to supply the remote customers. If the remote customers were to be charged based on the real costs to supply them they may choose an alternative such as a stand-alone supply.

The approach to pricing across the world varies depending on a range of factors, particularly related to the level of power industry restructuring. CIGRE Working Group C5.16 [18.18] has surveyed a number of countries to determine the types of costs that impact on end user billing and the methods and trends for practically allocating costs in both regulated and competitive market environments. The survey concluded that the main electricity cost drivers are a mix of fixed costs, capacity costs and variable energy costs. The retail billing components therefore include a balanced mix of fixed charges for common shared customer costs, demand charges for capacity, energy charges for fuel and other operating costs and policy charges for taxes and other externally imposed costs. The survey revealed that the extent of using demand charges varied widely, driven to some extent by the desire for rate simplicity, metering limitations and historical perspectives on electric service.

Large scale embedded generation has a significant impact on the use of the network. When combined with storage and smart meter enabled load control, the use of the network can be reduced to zero for extended periods. This has significant implications for transmission and distribution pricing. Most domestic electricity consumption is measured in kilowatt-hours and virtually all network costs are fixed, based on a return on the large capital investment and operating costs to operate and maintain the infrastructure regardless of the energy that flows through the system. To maintain the required revenue to service the network costs when energy consumption is falling requires the price on a unit of energy to be increased. This may lead to more customers investing in distributed generation with consequent higher energy delivery prices. This leads to what has been termed a death spiral, where the rising costs increase exponentially as the energy reduces (Fig. 18.9). Ultimately this could lead to prices for connection to the network being so high that no one can afford to connect.

Fig. 18.9

The death spiral

Rapidly increasing prices is one possibility and will depend on the responses of the network utilities and the development of new technologies. While significant installations of roof top solar are likely to continue, helping to exacerbate the above problem, large-scale renewable generators are still more economic than small-scale generators and these will require transmission lines to bring this energy from remote locations to load centers. This may encourage greater use of the network resulting in lower per kWh customer charges. In addition, individual houses that have surplus generation may want to trade this with others in the neighborhood. This will again require greater use of the network and will exert downward pressure on network prices. Electric vehicles and electrification of heating are also likely to increase demand on the networks. Ideally incentives will be placed on this increasing load to minimize the impact on peak loads. A key factor will relate to how the pricing structures adapt and how the customers respond to these changing opportunities.

18.6.11 Performance Reporting

While keeping the end price of electricity as low as sustainably possible is a key focus of regulators, it is also important to ensure technical performance is maintained or, in some cases, improved. One of the ways that regulators can keep tabs on the relative performance of monopoly network businesses is to require performance reporting. As discussed earlier there may be some difficulties comparing different organizations, particularly if they have different histories and current environments. However, comparisons will provide some useful information and the tracking of performance of a particular utility from one year to the next is certainly an effective model.

Utilities will generally use a number of both financial and technical performance indicators to help drive performance improvement in the organization. A number of these are likely to end up in the utility annual report as direct feedback to staff, stakeholders and shareholders. The regulator may use the annual report or even ask the utility to nominate certain indicators. They may also have additional requirements, especially where there is required ring fencing of monopoly components within the organization.

Regulators may also use a financial reward and penalty scheme to provide an incentive for the utility to improve performance. For this to work well, it is important to ensure that performance targets are reasonable and the rewards are sufficient without being excessive. Incentives may also be provided to drive innovation. Network utilities are necessarily conservative in their operations as large scale extended black outs can be catastrophic for both the utility and the country. With this rider, it is important that the regulatory controls do not stifle innovation, which can improve performance or reduce costs. In some cases, a value of loss of load may be used to justify network investments. This is often based on survey results to determine the value of the electricity to the customer at a particular time of day. This perceived value would tend to vary depending on how recently the respondents have experienced a blackout. Regulators therefore tend to prefer a lower value for this, although it is still usually significantly greater than the normal energy charge and will vary depending on, amongst other things, whether the supplied area is the central business district, a commercial area or a suburban domestic load area. Smart meters and dynamic pricing may also provide useful data for the customer electricity valuation adopted by the regulator.

18.6.12 Regulated Versus Non-Regulated Services

Within the transmission or distribution business there may be services that the utility provides that are not obligatory and can be provided on a contestable basis from a range of suppliers. These are not regulated but need to be ring fenced from the regulated business to ensure no cross subsidies. In some cases, transmission and distribution connections are not part of the regulated income and the connection can be either built by the user in accordance with prescribed standards or be subject to a separate competitive tender.

Similarly a utility may choose to build a transmission interconnector as an unregulated asset or merchant line. In this case, it would not have a regulated income but derive its revenue from users that wish to utilize the asset on a pay for service basis. These kind of interconnectors are not common but have generally been used to link power systems where the price in one system is different to the price in the adjacent system. The transmission business then leverages this difference to fund the cost of the interconnector. These types of interconnector are less popular due to the much higher risks involved as the price differentials between the adjacent systems can vary over time and investors tend to prefer the long term certainty of a regulated return on the asset.

18.6.13 Treatment of Losses

Electricity is produced and used at many different locations and there are product losses as it is transported to the loads. Generation must make up these losses at all times, which, on a typical system, tend to be on average in the order of four to five percent. The value of these losses will vary from one location to another, depending on the distance and transmission or distribution voltage. These losses have an economic and environmental cost. Network businesses may have an incentive within their revenue or price control to reduce the losses. However, these are not always under their control. As dispatch of the generators in different locations changes so too will the current flows in the network and the losses will increase or decrease with the square of the current. This may occur in relation to large generators connected at transmission level or for distributed generators such as rooftop solar. In some countries the penetration of roof top solar is so significant that at some times of the day the current flow reverses away from the load back into the transmission system. A useful discussion on the management of network losses can be found in a recently published CIGRE reference paper [18.19].

There are some losses that can be controlled by the network businesses but for this to be meaningful there needs to be accurate metering at the points of connection of loads or generators. While the end customer load will have tariff quality metering, the generators may not. Certainly, many rooftop solar systems will not currently have metering and their input will simply be seen by the customer tariff metering as a reduction in load.

Where the metering issue can be resolved, there may still be issues such as electricity theft and the reduced accuracy of meters as they age to deal with, although this latter aspect is less of an issue for digital meters. It may therefore be more practical to provide a reward for the utility that reduces losses when it provides specific evidence of the cost benefit for the particular location where the action has been taken. For example a transmission line voltage could be increased or a high loss transformer could be replaced with a low loss transformer.

18.6.14 Metering/Smart Meters

As mentioned in the above section on losses, tariff meters provide a very important role in the management of the power system. They ensure that customers are paying the correct amount for their service and that the utilities receive the required recompense for the service they provide.

In recent times, smart meters have been introduced into an increasing number of countries. A smart meter may vary in complexity but fundamentally it consists of a tariff meter that records electricity consumption in intervals of one hour or less and is able to communicate with a central control system. In some cases the meters have been introduced to combat theft and in other cases for a range of technical solutions that enable customers to manage their load and reduce their cost based on price signals provided by the utility.

For example there may be a shortage of generation resulting in a very high wholesale price. The smart meter could be sent a signal from the retailer, the supplier or the load aggregator asking for load to be reduced. This would normally be based on the contractual relationship between the company and the customer. One input to the company’s decision-making could be scarcity price signals from a power exchange or congestion price signals from a system operator. The smart meter could then send a signal to various appliances either switching them off or reducing their load. One of these could be a non essential load such as a pool pump or an air conditioner that could be switched to run with a fan only for say 15 min in the hour. The customer would receive a reduced tariff to compensate for the inconvenience of the power reduction. Various appliances can also be set in advance to operate at times of low demand. Ultimately, with the use of batteries at the load point, it would be possible to match the load to the ideal generation profile. In the case of wind and solar, this would be when the sun shines or the wind blows. While some countries have taken major steps to roll out smart meters to all or most customers, the technology is still considered expensive by some, especially if it requires the replacement of perfectly good meters that are already installed at the load points. Evaluating the full costs and benefits is complex and the subsequent large capital investment may prove difficult to justify without some overriding government policy directive.

The European Commission has directed that 80% of customers should have smart meters by 2020, providing an incentive for member states. Some countries (e. g., Italy and Spain) had already rolled out smart meters and other countries are well on the way (e. g., France and the Netherlands) but others are well behind (e. g., UK and Germany) [18.20]. Figure 18.10 shows the current (2017) responses from member countries, as reported by the EC Joint Research Center Website.

Fig. 18.10

Smart metering rollout (EC Joint Research Centre, 2017) [18.21]

Other parts of the world have also made progress with the installation of smart meters. For example, in Australia, Victoria has achieved a 96% rollout and there has also been a significant rollout in South Australia and New South Wales. A national electricity market rule now requires smart meters for new and replacement meters. In China more than 400 million meters had been rolled out by 2017 whereas in other parts of the world such as Eastern Europe, Latin America, the Middle East and Africa, the rollout has been quite limited [18.22].

18.6.15 Technical Regulation

In parallel with economic regulation it is also desirable to monitor the technical aspects to ensure the power system delivers the required reliability and quality. Instantaneous matching of supply and demand is required if the power system is to meet the required standards. It must also withstand sudden disturbances such as system faults and deliver electricity within prescribed bands of voltage and frequency. In addition, interference such as harmonic distortion must be managed to ensure customer equipment does not mal-operate or become damaged. In order to manage these impacts, power systems are designed with a level of redundancy, installed equipment must meet prescribed standards and steps are taken to reduce power quality issues such as harmonic distortion to within prescribed limits. The level of reliability and quality is usually chosen to meet customer needs and will usually vary from major city central business districts where high levels of redundancy and power quality are required to remote rural loads, which would normally have lower standards. The chosen standards are also limited by economic limitations.

It is also important that customers that connect to the network meet required standards as they may impact on other customers that are connected. These standards are designed to assist the system and network operators to meet system standards. For example, the sudden switching of a large load onto a network that is not rated to take it may result in large unacceptable voltage swings for nearby customers and in the extreme, may cause the supply circuit to trip causing loss of supply to other customers on the same feeder. There are also safety standards that must be met to ensure public safety.

While the power systems are designed and built to meet the prescribed standards, they must also be operated within defined rules to ensure the required power system performance is delivered. Prior to the deregulation of the electricity industry and the introduction of electricity markets, many of the technical issues were managed internally by the various electricity businesses. Open competition amongst generators and amongst retailers has meant that the technical rules and standards are required to be public. This ensures clear accountability and avoids accusations of bias, particularly if vertically integrated utilities are involved.

18.6.16 Technical Rules

Published technical rules or grid codes now governs many utilities. These documents may have been produced by utilities but they will normally have been subject to review and public consultation by regulators prior to publication. It is beyond the scope of this chapter to go into detail on the technical rules; however, the section on regulation models around the world will highlight some of the differences between countries and provide references to the various documents that are applicable in those countries.

A recent CIGRE publication on the review of drivers for transmission investment decisions [18.23] has established that the main reason for expansion or new build projects is security of supply followed by new connections, generation integration and economically motivated projects. The report also noted that compliance with technical planning criteria is the primary determinant of reinforcement and that most transmission investments are made on a long-term basis. In a number of cases, changes to planning criteria are being made to reflect the changing industry, particularly in relation to integration of high levels of renewable energy sources. In addition, changes are also being made to accommodate new technology or industry wide practices. In many cases, economic principles such as the cost of unserved energy and societal impacts are considered as part of the review of the planning criteria.

18.6.17 Service Standards

The earlier section on performance reporting referred to the technical performance indicators that regulators monitor to ensure that cost cutting does not result in poor service outcomes for customers. This requires the relevant utility to publish its performance against the various service standards. These service standards may relate to the frequency of interruptions to supply or the duration of the outages.

18.6.18 Environmental Regulation

In addition to all the technical requirements necessary to deliver a sustainably reliable power supply at a reasonable cost, many countries have imposed additional environmental requirements on the electricity utilities and their customers. These may be at a local level, say for an individual town, a region or state level, an individual country or for an entire continent. While the various political leaders may agree these at a high level, the detailed interpretation and implementation of the requirements is left to individual areas. In most cases, satisfying these additional requirements will add costs to the delivery and consumption of power. While some attempts have been made to quantify the costs of inaction on these issues, they are not yet universally accepted.

18.6.19 Climate Change

Climate change is an issue that has maintained considerable global attention for a number of years. In December 2015, 195 countries agreed to a global climate deal that is legally binding [18.24]. This agreement aims to limit the increase in CO₂ emissions from those countries to a level that scenarios indicate would restrain the increase in average global temperatures to significantly less than 2 °C above pre industrial levels and ideally as low as 1.5 °C above. While the global commitment has remained resolute for most signatories, at the time of writing the US appears to be reducing some of the previously committed actions where it believes they will have a negative impact on its economy [18.25] and, it is reported that many countries are expected to fail to meet their commitments.

The actions taken by countries in relation to these climate change commitments will directly impact on the electricity sector, particularly where they are reliant on fossil fuels. Common actions to be taken relate to use of less fossil fuels, an increase in renewable energy, control of CO₂ production and a drive to improve energy efficiency. Each of these actions can have a direct impact on costs of electricity and interfere in the pure economic drivers in a commercial market. Regulatory oversight is required to ensure actions and related cost impacts are clear, open and accountable.

18.6.20 Energy Efficiency

Energy efficiency refers to the ability to produce more for the consumption of less energy. The World Energy Council has produced a useful report on the current status of energy policies around the world [18.26]. It considers the policies and trends that are occurring in relation to this topic. Energy efficiency is one of the simplest ways to reduce the impact of fossil fuels on the environment as well as ensuring a more sustainable approach to the use of the world’s scarce resources. Often it can pay for itself in a relatively short period. Some countries have set aspirational targets and others have legislated to eliminate some of the less efficient plant, equipment and appliances. Others use labels to advise consumers of the appliance efficiency at the time of purchase.

A key problem in relation to energy efficiency relates to the disaggregated nature of the supply chain. If the maximum end customer benefits are to be achieved, the reward must be spread appropriately across all parties that contribute to the efficiency. This will require clear price signals and profit incentives that encourage an integrated approach to the problem.

Within the electric power system, efficiencies can be gained by modifying the design, construction and operation of the system components, reduction of network losses (while avoiding suboptimal generator dispatch) and the use of energy storage. The use of smart meters and intelligent devices allows the ability to control and influence consumption rates and timing, which can improve efficiency of the whole system. Choosing the optimum mix of these factors will determine the ultimate system efficiency.

The various aspects can be described as shown in Fig. 18.11 as presented at the CIGRE symposium in Bologna [18.27].

Fig. 18.11

Energy efficiency pyramid (reprinted by permission from [18.27])

Various approaches can be used to encourage energy efficiency. The electricity market can send signals and reward efficient operators; although to work well, this would require access to storage and demand side management together with clear price signals. Regulatory targets can be set, although this can be distortionary as it may lead governments to back politically attractive technologies. International standards can improve interoperability as long as they don’t stifle innovation and competition.

18.6.21 Visual Impact

As power consumption around the world has increased so have the number, size and density of transmission lines and substations. In addition the growing number of large-scale renewable energy projects that are adding to and sometimes displacing existing fossil fuel generators are driving further construction of new transmission lines. In some cases transmission interconnectors are being built to connect the diverse generation and load portfolios of adjacent countries in order to lower the overall cost to customers. Line routes to supply the central business districts of large cities are also becoming more difficult and expensive to procure. In many of these cases there is considerable public opposition to the construction of these assets due to the visual impact. As a result, more lines are being placed underground or, at the very least, suboptimal line routes are being used to avoid contentious areas. These changes will generally lead to increased construction costs, which will need special consideration by the regulator. Evidence needs to show that the chosen route was the lowest practical cost for the service provided and that the utility has not simply taken the easy way out.

18.7 Regulation Models Around the World

This section considers a few examples of the regulation models applied across the world. There are similarities between them as well as an overall consistency with the descriptions above. Some detail has been provided to demonstrate the differences between the countries that often reflect the history of evolution of electricity markets and deregulation within the different regions.

18.7.1 United States

In general the US follows the methodologies discussed above although this varies between states, unless it falls under the jurisdiction of the federal regulatory body. The following discussion highlights a few general points in relation to the US. One source of a more detailed examination can be obtained from [18.28].

The majority of utilities are privately owned with the remainder mostly owned by government entities or local communities. Electricity generation is generally fuelled by coal, gas and nuclear with some hydro and a growing proportion of solar and wind.

In recent times there has been a huge surge in gas generation, which has coincided with the rapid expansion of the use of shale gas. This has resulted in the replacement of a large number of aging coal fired generators. The US is now considered to be the world’s leading natural gas producer. When operated in parallel with intermittent generation from wind and solar, open cycle and to a lesser extent combined cycle gas turbines are more able to follow rapid changes in generator output than coal and nuclear and gas has the added advantage of emitting lower greenhouse gases than coal.

Some states have unbundled their transmission and distribution monopoly components from the contestable generation components.

Regulation generally occurs at a state level unless it relates to interstate transmission. This will include any part of the state system that has an impact on the interstate transmission as well as the transmission interconnector itself. The federal regulatory body is known as the Federal Energy Regulatory Commission (FERC). Third party access is allowed to the transmission system by law and the tariff for access is approved by FERC. The North American Electric Reliability Council (NERC) oversees reliability and the adopted standards are legally binding.

Generally in the US, rate of return regulation is used and this provides a steady stream of revenue to cover the costs to the utility of producing, transmitting and distributing electricity. Electricity regulation is used to protect public interest and allow access to all at an approved price.

The scope of regulation in the US generally follows the principles described earlier in Sect. 18.6. This includes revenue determination and the setting of prescribed service standards and prices to customers. Many states are being influenced by legislated environmental requirements, particularly in relation to the use of renewable energy. The costs of implementation of these requirements fall under the jurisdiction of the regulators assuming the utilities wish to recover these costs as regulated revenue. There is also some oversight of the boundary between regulated and unregulated revenue to ensure no cross subsidies. This may require the use of subsidiary companies.

In general there is no prescription on the timing of when a utility should seek regulatory approval of a major review of prices and service standards. However, this normally occurs every two to five years.

The review process generally involves a hearing where the utility presents evidence to justify the requested changes. A process of interrogation then follows where the utility may present expert witnesses and, in some cases, other affected parties may provide evidence. There is also usually an opportunity for the general public to provide comment or evidence. Negotiations then commence with the goal of achieving an agreed settlement on the final outcome. Ultimately this leads to a final order by the regulator.

Annual costs are determined based on analysis of actual expenditure and revenue in a past year, taking into account the effects of weather and any other significant disruptions and expenditure forecast for a future year. The process requires the utility to submit these costs as evidence to support any required rate increase.

Utilities are allowed to earn a reasonable rate of return on their regulated asset base considering the level of risk they are facing. Traditionally this would have been a relatively low risk, although there may be some argument that this risk is changing with the rapid growth of renewable and gas generation leading to a lower demand for existing coal generation. This new generation may be installed at different locations to the existing facilities leading to some level of stranding of transmission assets. The rate of return is normally calculated as a weighted average cost of capital taking account of the varying rates for equity and debt.

While traditional costs are relatively consistent each year, significant events such as major storms or the closing of a nuclear power station are usually considered separately.

Tariffs are determined by considering each class of customer as described in Sect. 18.6. Charges for connection to the electricity network are subject to regulation and may include cross subsidies if this is considered an economically efficient way of recovering costs. The charges are required to have reasonable terms and conditions set at a fair price.

18.7.2 Great Britain

In Great Britain, generation assets were split from network assets as part of major reforms to the electricity sector in 1990. Transmission was divided into three parts each of which is now privately owned. One sector was a combination of England and Wales, another was northern Scotland and the third was a combination of southern and central Scotland. There is one system operator for the whole of Great Britain, National Grid Electricity Transmission plc (NGET), which is also the owner of the England and Wales transmission system. There are 14 licensed distribution companies, which are owned by six different groups. There are also a number of smaller distribution network operators.

All of these transmission and distribution companies are considered natural monopolies and are regulated by Ofgem that also has a role to oversee the operation of the markets.

The transmission and distribution networks are regulated by an RIIO framework where revenue = incentives + innovation + outputs. This model is intended to drive improvements in performance in relation to both safety and reliability and lower costs for customers by providing incentives for the network businesses. It is also intended to encourage addressing wider environmental objectives such as those associated with climate change. To this end it encourages innovation, recognizing that this may have a higher risk than traditional investment. The framework is used to set price controls for a period of eight years.

While Ofgem requires each network business to submit proposals on how it will deliver within the RIIO framework, it also provides a number of documents on how the submissions should be structured and the information that should be provided. Amongst other things, these include comment on expected content in relation to the environment, stakeholder consultation, losses and innovation. Network businesses are required to propose for approval, performance targets with financial penalties and rewards, an allowed revenue requirement and mechanisms that would be used to cover unforeseen developments that may require changes to the allowed revenue.

The success of the latest scheme may not be fully understood until the next regulatory period is reviewed in 2021; however, Ofgem is already proposing that a more onerous scheme will apply in the next review. It is claiming that there have been significant improvements in reliability, costs to customers and customer satisfaction under Ofgem’s current and past regulation [18.29].

The framework also gives an incentive for the network businesses to reduce losses, which are the main opportunity to improve energy efficiency, provided the solutions do not lead to suboptimal generator dispatch. Losses include theft as well as those on the network. Network businesses are required to publish their strategies to reduce their losses as part of their compliance with the RIIO framework. A useful reference has been published by Ofgem titled Energy efficiency directive: An assessment of the energy efficiency potential of Great Britain’s gas and electricity infrastructure [18.30].

As regulation has developed in the UK, so has its complexity with regulatory submissions running into hundreds of pages that provide considerable detail on how they propose to deliver on their expenditure and performance targets.

One of the changes made under the new regime is to consider the total proposed expenditure rather than separately considering operating and capital expenditure. This reflects a view that there has been cost shifting between the two categories in the past.

A further change is the eight-year period rather than the previous five-year duration. Longer time frames are possible as the industry and Ofgem are now more mature and both sides have considerable experience. It also reflects the fact that transmission and distribution investments are generally long term. There is still some risk that technological change will accelerate the transformation to alternative forms of energy provision and produce an unforeseen risk on the network companies. However, this can be dealt with to some extent by the change mechanisms that are built into the regulatory process.

18.7.3 Australia

The Australian Energy Regulator (AER) [18.31] carries out regulation of the electricity sector in Australia other than in Western Australia. This includes monitoring the operation of the electricity market and the allowed revenue of the network businesses. In Western Australia the Economic Regulation Authority (ERA) [18.32] carries out a similar but independent role. Consideration was recently given to moving the role to the AER, but it remains with the ERA at the time of writing. Network businesses are considered natural monopolies and have to submit proposals to the AER on their required revenue in accordance with a prescribed framework and timeline. This framework is set out in national electricity law and rules. The approach is similar to the RPI − X incentive regime that was used in the UK prior to the implementation of the new RIIO regime. In this case, the X is an efficiency factor and the regulator expects the business to find a range of efficiencies that exceed those driven by this framework. As the revenue is approved for a five-year period, there is an incentive for the network business to reduce expenditure by more than the regulated X factor over the period and therefore increase its profits. The regulator can also look at these found efficiencies and factor them into the next review.

During the review process, factors considered include the projected electricity demand, the age of the infrastructure, operating and financial costs and network reliability and safety. Decisions usually apply for five years. Under the prescribed framework, the AER determines the allowed revenue for the business that covers its efficient costs by using a building block model. These building blocks include capital expenditure, operating and maintenance costs, asset depreciation and taxation liabilities. The AER also determines an allowed rate of return on capital taking into account debt, equity and risk issues amongst other things. The incentive to defer cost savings towards the end of the regulatory period that may occur in a traditional CPI − X regime is counteracted by an efficiency carryover mechanism in the building block approach.

The AER also reports on the performance of the market and electricity businesses including customer issues relating to affordability and disconnection of customers for non-payment of bills. It also administers a retailer of last resort scheme as a protection for customers in case an electricity retailer fails.

As for the UK model, the Australian regime is evolving. The current rules are quite prescriptive and revenue submissions run into many hundreds of pages. The prescriptive nature of the regime makes it quite difficult to make changes that may produce more favorable outcomes for the customer. Under the original process, the network business could appeal if it perceived the regulator’s decision to be unreasonable. On the surface, this appears acceptable as the regulator is attempting to strike the correct balance between the long-term interests of the customer and those of the network business and various decisions may therefore be subject to challenge. Also, an appropriate return is necessary to ensure that the network business owner will continue to invest its capital. However, there were a number of concerns raised that, amongst others, the process produced unjustifiably higher prices for customers, the long process led to significant regulatory and price uncertainty and that the appeals may be related to procedural correctness rather than the merits of the outcome. As a result, the appeal process has been removed, except for decisions related to the disclosure of confidential or protected information. The lawfulness of the regulator’s decision can, however, still be subject to a judicial review in the Federal Court. This limits the review to breaches of the rules of natural justice or errors of law rather than the merits of the regulator’s decision.

18.7.4 Nordic Countries

Deregulation in the Nordic countries started in Norway in 1991 followed by Finland in 1995 and Denmark and Sweden in 1996. There is one transmission system operator in each country and a large number of distribution companies (varying from 84 in Denmark to 380 in Sweden). Consistent with previously discussed models, revenue cap electricity regulation is now used in all the Nordic countries.

Differences in approaches in each of the Nordic countries are in the detail. For example different models are used for deciding the allowed rate of return. In addition, the assessment of the regulatory asset base varies leading to large differences in the applied revenue caps. Similarly, benchmarking and efficiency targets may be applied differently. The regulatory period is one year in Denmark, four years in Finland, five years in Norway and four years in Sweden.

Incentives are provided to ensure reliability is maintained despite incentives to lower costs. These vary between countries, but in all cases attempts are made to relate the rewards and penalties to the customer value of the reliability.

NordREG, the Nordic energy regulator, has produced a detailed overview of the economic regulation of electricity grids in Nordic countries [18.33].

18.8 Conclusion

This chapter has provided an overview of the current state of electricity markets and associated regulation. This has included the development of the electricity industry structure and its organization. A more detailed explanation of electricity markets and the various models that have been adopted to facilitate competition then follow, as well as some examples from around the world. A similar approach has been used to explain the use of regulatory oversight.

The chapter is necessarily at a reasonably high level and the reader is encouraged to further explore at a more detailed level to view the latest developments and in depth explanations regarding the design and operation of market and regulatory schemes that are used across the world. Widespread adoption of electricity markets is relatively recent, occurring over the last twenty-five to thirty years and the models used continue to be developed and refined.

The drive to reduce the carbon footprint of electricity supply while ensuring the end price to customers remains affordable is ongoing. To some extent this is dependent on the evolution of existing and the development of new technologies. This includes storage mechanisms such as batteries, new distributed generations and a range of new smart load control processes. Some of these will increase the number of competitive generators and loads in the market as well as change the nature of their operation. In some cases this may put pressure on the traditional monopoly status of the network businesses. Electricity markets will need to evolve to facilitate the new forms of trading and customer expectations. In addition, regulatory oversight will need to change to ensure it supports market developments that benefit the end customers.