Learning Objectives

  • Be able to understand the key energy economics aspects of the Energy Systems Integration paradigm,

  • To get a grasp on the system changes involved due to the use of smart grid (energy and ICT) technologies,

  • To be able to describe the types of new actors involved in smart energy systems and their roles,

  • To have a solid understanding of the major impacts on the economics of competing supply and demand-side flexibility options,

  • To obtain a sense for the potentials of the different kinds of distributed energy resources and their possible value for actors in smart grid systems, and

  • To be able to explain the role of machine learning in the context of energy economics and management of smart grids.

1 Introduction

The question why energy economics was established as a dedicated subdiscipline of economics is typically answered by pointing to the many specifies of energy markets, regulation and politics. Likewise, one could ask the question why there is a need for smart grid economics? Put differently, which tools in the toolbox of an energy economist are missing entirely, and which ones have to be adapted or otherwise modified?

Several of the smart grid topics addressed rather broadly are elaborated in some more detail in dedicated chapters, such as demand-side management (incl. dynamic pricing) (\(\rightarrow \) chapter “Demand Side Management”), energy market engineering (\(\rightarrow \) chapter “Market Engineering for the Smart Grid”), regulatory and institutional aspects (\(\rightarrow \) chapter “Regulatory and Institutional Aspects of Smart Grids”), system modeling (\(\rightarrow \) chapter “Modeling Smart Grid Systems”), (business) analytics (\(\rightarrow \) chapter “Smart Grid Analytics”), and business model design (\(\rightarrow \) chapter “Business Model Design”), and are therefore only discussed in a concise manner and mainly from an economics perspective.

The aim of this chapter is to point out not only some specialties of smart grid technologies and systems relevant from an energy economics perspective, but also the drivers of the paradigmatic changes ahead. These can be summarized as decarbonization, decentralization, digitalization, and democratization. While there are large potentials for efficiency gains, the challenges are formidable for the system operators for whom both distributed supply, demand, and storage/demand response will become less controllable and predictable overall, compared to the traditional top-down hierarchical system.

Smart grids enable to deliver energy in a much better controlled (“smart”) manner, from different locations of generation to (active) consumers. Ideally, such a smart grid system is even more reliable and resilient than the classical hierarchical, top-down system (although this still needs to be proven; after all, the transition from system A to system B can be expected to be a lengthy process that is prone to many uncertainties).

Energy consumers today are increasingly empowered (and potentially also overwhelmed) by a growing number of choices including DER—such as rooftop solar PV, smart loads and energy storage systems, and energy service-oriented options, such as peer-to-peer (P2P) trading and participation in microgrids or some virtual power plant (VPP). Likewise, more cost-reflective tariffs aim at enhancing system efficiency and potentially offer cost-saving opportunities by acting flexibly and system-friendly (Gui and MacGill 2018). Such a continuously widening range of choices allows consumers self-selection in order to satisfy their individual energy (services) needs, affecting their energy consumption, production (prosumer households), and energy asset investment. Energy consumers/prosumers, however, have to make highly complex choices under considerable economic, technical/system, and social uncertainty. Still, to this end, a growing number of consumer-centric service innovations are available to assist energy consumers in their decision-making processes. And while some consumers/prosumers may prefer to get assistance only for individual services (for their personal, customized preferred solutions), others will prefer more fully integrated services (Gui and MacGill 2019).

Smart energy economics, in a way, combines standard energy economics with power system economics and elements of energy information science. It also brings together new business models and management concepts enabled by smart grids with analytical concepts that have been well established over the last decades—but somehow pushing it further to new levels. Examples of new analytical concepts are local market designs for peer-to-peer trading, the economics of automated energy management systems, and energy hubs, to name just a few.

The spheres or domains that can be distinguished are the following:

  • Generation and storage,

  • Transmission and distribution,

  • Customer/end-use,

  • (New) Service providers,

  • Markets, and

  • Regulation and policy-making.

Smart grids are part of an emerging paradigm change toward a more holistic, system-oriented perspective of the energy supply system. It will also require the development of new economic, regulatory, and policy frameworks to ensure efficiency. The question is out there whether the enabled sustainable energy transition and required transformation of the socio-economic and technical systems will be incremental or disruptive, but it makes sense in any case to discuss some of the issue at hand, to develop new analytical concepts and models, and to aim for proactive regulatory and policy guidance.

Different strands of the economics literature, not just neoclassical economics, are potentially useful for a better understanding of the issues and potential pathways and solutions at hand. For instance, we could think of ‘new energy economics’ (e.g., in the sense of making use also of new institutional economics, evolutionary economics, and behavioral economics) as a new school of thought, and in contrast to more conventional energy economics inspired mostly by neoclassical economic thinking (e.g., Zweifel et al. 2017). Hence, this chapter mainly points out the key characteristics of smart grids and related issues, as these are not typically covered in standard energy economics textbooks. It also draws on (often very recent) literature where aspects of smart grid economics and management have been discussed, typically from a very specific angle not easily accessible, and thus not well suited, as a reference for readers who want to effectively and efficiently build up their knowledge on smart grid economics.

The remainder of this chapter is organized as follows. In Sect. 2, we discuss objectives of smart grid economics as a new strand in energy economics, and energy system integration as a new paradigm, and also reflect on the potential benefits of multi-business utilities in a smart grid world. Section 3 takes the perspective of structures and actors, not only discussing the impact of smart grids along the value chain generation–transmission/distribution–consumer, but also discussing the role of new (energy) service providers, including aggregators, and the emergence of new markets. Section 4 tackles some governance, policy and broader issues related to smart grid economics, whereas Sect. 5 addresses the investment needs and related issues. Section 6 then briefly discusses the various time scales involved for economic analysis and value creation, including the need and value in short-term balancing. Finally, Sect. 7 focuses on the smart grid initiative in the United Kingdom as a case study.

2 Energy System Integration (ESI) and the Need for Smart Grid Economics

Smart grid technologies enable a restructuring of the energy systems as we know them in a way that has not been seen since the early days of electrification. Still, the co-evolution of the existing energy system/sector—with all innovations, standardizations, and regulatory adjustments—has taken many years. It can be expected that also the co-evolution toward a much more integrated, digitalized energy system will be a lengthy process (not least due to the massive investment needs and the fact that, at least modern societies, cannot afford a non-reliable energy supply system). But the new system does not only have to be reliable (and resilient), but it also needs to be affordable and environmentally sustainable (the energy sector, e.g., has been a major contributor to global greenhouse gas emissions).

The challenge of restructuring is all the more ambitious as it involves achieving or maintaining sustainability (along the economic, social, and ecological dimensions), energy justice, and supply security/resilience. It also implies that the inherent trade-offs (dilemmata) are actively managed and that the system remains adaptive (avoidance of undesirable lock-ins) and manageable (acknowledging the curse and limits of complexity).

2.1 Smart Grids and ESI as a New Paradigm

Smart grids enable system integration at an unprecedented level. Apart from new ways of coupling energy carriers and infrastructure systems, exploiting numerous horizontal synergies and raised efficiencies at all levels, in principle they will also allow to more effectively balance sustainability, energy security, and equity considerations. Apart from technical issues that still need to be resolved, there is an urgent need to think about the economic, regulatory, and policy frameworks that will enable to exploit these new potentials and to continuously safeguard the efficient performance of such SG-enabled integrated energy systems over time. In the following, a synopsis is given on the ESI paradigm (cf. Jamasb and Llorca 2019).

Aside from technical issues, there are needs to better explore not only the present and expected future economic and regulatory needs of smart grid systems but also unexplored business models and policy-making needed. The performance and desirability of a smart, integrated energy system will depend not only on the technical setup but also the economics behind it for both private actors and the public sector. In other words, there is no guarantee that such an integrated system will be superior unless it is well defined, designed, and orchestrated. Regulators are particularly challenged to steer the course broadly and avoid the worst consequences of a failure of a system that is many times more complex than the traditional ones.

Starting in the 1990s, with some earlier exceptions, market liberalization was becoming a dominant strategy to improve energy services through competition (not only on the production/generation and retail side but also in terms of capacity usage of so-called ‘natural monopolies’). ESI will challenge this existing paradigm by calling for the exploitation of synergies. ESI need more synchronization of the individual parts.

Figure 1 shows an exemplary architecture of an integrated energy system at the transmission and distribution levels, providing an idea of the enormous potential for flexibility and efficiency gains by means of potential substitution between energy sources to provide certain energy services to end-users. Note, however, that such benefits arising from a more integrated and thus more flexible system might come at higher capital and operating (in terms of managing) costs.

Fig. 1
figure 1

Example of an integrated energy system architecture (Jamasb and Llorca 2019, p. 12)

Such an integrated architecture can be seen as a ‘network of networks’ or ‘system of systems’ (cf. O’Malley et al. 2016), and an ESI may be extended to include also other network infrastructures (e.g., transport, telecom, water, and compressed air) than energy ones (electricity, heating, cooling, gases, etc.).

Since the 1990s, such network industries have been liberalized and unbundled (legally, or only in terms of accounting), in order to enhance competition but requiring effective and efficient regulation to avoid excessive profits and market power abuse. Obviously, the benefits of market competition and incentive regulation need to exceed the economies of coordination lost from ruling out vertical economies of scope. Traditionally, vertically integrated companies were able to benefit from horizontal economies of scale as well as economies of coordination stemming from vertical economies of scope (Jamasb and Llorca 2019, pp. 9, 14). Horizontal economies of scope typically arise from joint utilization of resources. Multi-utilities can benefit from providing services where the same network is used or where similar products, or services (e.g., billing), are provided jointly to the customers. In network industries, horizontal economies of scope can also be reaped through joint management of knowledge, e.g., regarding regulation, planning, R&D, and strategy/policy development.

As in many countries, market liberalization has led to some vertical separation of energy industries (so-called ‘unbundling’), and there is obviously a dilemma: On the one hand, reducing institutional and regulatory barriers would help not only to unleash flexibility and efficiency potentials as well as synergies by means of vertical integration but also to integrate a much wider range of resources. On the other hand, it may lead to cross-subsidization and market power abuse, requiring new, and much more integrated models of regulation as well as to mitigate such inefficiency problems (and related welfare losses).

The evolutionary nature of the system development requires an equally dynamic system evaluation for identifying further/remaining coordination potentials and any (remaining or emerging) flaws. The quest is for a dynamically optimal and optimized system arising from the interplay of its constituent parts.

Overall, ESI enables the use of (technical, economic, social, etc.) synergies but requires some smart form of technical and economic/business coordination, both horizontally and vertically. Due to the energy system’s increasingly distributed nature, it makes sense to organize it as much as possible based on well-functioning markets and transparently regulated and dynamic incentives that ideally not only ensure efficiency but also some degree of energy justice.

2.2 Synergies and Multi-business Utilities

An interesting question is whether the integrated energy company is outdated due to the neoliberal paradigm of introducing competition and low market entry barriers by unbundling. Still, capital markets might favor either more integrated or more focused utilities. The goal of multi-business enterprises is to create more value than can be achieved with stand-alone businesses. Both liberalization and re-regulation have been taking place simultaneously, and in different sectors of the energy market. The scope of synergy exploitation can be expected to have considerable influence on the value of business models in light of the further development and implementation of SG technology.

The challenges inherent in the need for massive structural change of the energy system in order to enable a sustainable energy transition toward a zero carbon energy supply also provide room for a multitude of new business models and opportunities. As discussed in Fuhrmann and Madlener (2020), synergies are subject to permanent change, thus requiring continuous evaluation of the corporate and social welfare surplus of multi-business utilities compared to focused utilities.

Fig. 2
figure 2

Taxonomy of synergy types (Fuhrmann and Madlener 2020; Müller-Stewens and Brauer 2009, p. 376)

Fig. 3
figure 3

Cross-functional synergies between classes of assets within midstream/trading and other asset classes (for larger utilities often supra-regional ones) (Fuhrmann and Madlener 2020, p. 29)

A classification of synergies is shown in Fig. 2, whereas Fig. 3 depicts the main cross-functional synergies between different asset classes (midstream/trading and others) of a multi-business utility. From Fig. 2, it becomes clear that there are many types of synergies (operational, managerial, financial, and so-called ‘synergies of market power’). The operational ones have to do with costs or revenues and the sharing or combining of resources. The managerial ones are related to functions, strategies, and organizational aspects. The financial synergies have to do with risk mitigation, company-internal capital market advantages, and tax benefits. Finally, the synergies of market power involve multi-point cooperation (taxit collusion, mutual forbearance), predatory pricing, and reciprocal dealing. A discussion of how to interpret these synergies in the context of a german multi-business energy utility is provided in Fuhrmann and Madlener (2020).

As can be seen in Fig. 3, which is adopted from that study as an example, all the asset classes are interrelated with trading, and within the trading business/stage each asset class has interdependencies with each other (mainly due to shared infrastructure and resources, such as cross-commodity trading desks or jointly used information services). A big advantage of exploiting synergies between the asset classes depicted can be reducing the cost of capital (e.g., due to internal value chains), and thereby increasing a company’s liquidity.

3 Structures and Actors

Smart grids integrate energy and ICT networks, dedicated hardware and software, as well as numerous distributed (and centralized) assets that enable to monitor, control, and aggregate (‘pool’) generation, storage, and end-use assets (see also Fig. 1). The development of smart grids requires different expenditures in the energy supply industry (transmission & distribution grids, power plants) and in the end-user sector/s (industry, private households, transportation, etc.). Investment considerations are discussed in Sect. 5.

The smart grid/ESI paradigm involves the efficient exploitation of renewable energy, network automation, demand response, reactive power management, and much more. This increased use of smart grid applications enabled by smart grid infrastructure investments (for Europe until 2020 estimated to be in the order of 600 bn Euros between 2014 and 2035 alone; cf. Cambini et al. 2016), if well-designed and well-managed, enables to enhance system efficiency, resilience, and social welfare.

Integrating higher and higher shares of RES into the power system is a challenge in light of their intermittency. Smart grid technologies enable to exploit also many new sources of flexibility, which in sum will increase the elasticity of (flexibility) demand. It will require some accurate estimation of the available (i.e., individual and, if available, also aggregated) flexibilities on the supply and demand side of the market and the system operator’s side. A flexibility merit order, as illustrated in Fig. 3, would take into account the demand and supply side of some flexibility market in order to determine the short- and long-term equilibrium in the flexibility market, and to find out the response behavior to any changes in the flexibility market price signal revealed.

Apart from scrutinizing the net economic gains that can be reaped, it is necessary to understand the major actors and types of systems and concepts involved. In the following, therefore, an overview is given of the structures and actors, their characteristics and functions, and some economic considerations, in light of the co-evolving smart grids.

3.1 Operators/Owners of Distributed Generation and Storage Units

Smart grids will enable small producers of energy (distributed generation, distributed storage unit operators) to participate in the energy distribution. Such local generation and distribution of electricity will change the low/er voltage level of the grid to an active layer with multi-directional power flows. Open, and ideally competitive, local energy markets, enabled by an ICT-upgraded (‘smart’) distribution grid, will enable to balance local supply and demand locally.

Up to now, even modern network-based energy supply systems (e.g., for electricity, natural gas, and district heating) are commonly designed and operated independently from each other. (Smart) Energy Hubs (EHs) as a potentially important element of future multi-vectoral (or integrated) energy systems provide an opportunity for system planners, operators, and also prosumers to decentrally couple, and technically and economically optimize, heat and power generation, conversion, and storage (Geidl et al. 2007).

Distributed Generation: Flexible Power Plants

In the future smart grid, it can be expected that many end-use devices and appliances will be connected in real time and obtain grid parameter values (voltage and frequency information). In addition, some supervising algorithms will be installed that serve to maintain resilience and prevent damages/outages by monitoring the real-world performance of the system against the control signals. This enables dynamic end-user involvement and can help to prevent any security of supply problems, e.g., in the case of management system failures as well as cyber-attacks or other unfriendly intrusion.

Figure 4 presents a conceptualization of increasing degrees of decentralization, viewed from a technical (decentralized vs. decentralized need for flexibility), managerial (central vs. zonal/local operation), and economic perspective (fixed/grid-wide grid tariffs and/or centralized markets vs. dynamic/local grid tariffs and/or local markets). The transition could be from centralized large-scale markets for (aggregated) DER to an increasingly decentralized techno-economic management of the electricity system (Eid et al. 2016, p. 245).

Fig. 4
figure 4

Technical, managerial, and economic challenges with increasing levels of decentralization (Eid et al. 2016, p. 246)

The Economics of Storage

The potential economic value of storage has been discussed in the literature in recent years (e.g., Sioshansi et al. 2012; Crampes and Trochet 2019; Giulietti et al. 2018). Still, there are many storage technologies available, some of which can be installed in a centralized or decentralized manner, including such where some energy supplier or aggregator may not have access to (which reduces the potential, e.g., by trading with aggregated DER in the wholesale energy-only markets or some of the ancillary services markets). An essential question regarding the use of storage units in smart grids is whether there is enough economic incentive/value to operate profitably by, e.g., exploiting intra-day arbitrage opportunities (the time horizon of interest again depends on the technology). Storage technologies vary a lot in their relative characteristics and merits, but are typically characterized by their energy and power capacities, response times, ramping rates and per unit costs. Economics, regulation, market structure (and its development over time), and natural resources can vary a lot between countries as well, and so do the barriers for storage use and the creation of new business models for storage units, rendering generalizaions difficult (Madlener and Specht 2018).

The optimal level of storage use in a smart grid may change considerably over time, as the smart grid evolves, and it can be analyzed from an economic perspective either in terms of private and societal benefits and costs resulting from investment in and operation of a storage device.

Smart grid technologies in combination with storage units offer new and intriguing opportunities. However, regulatory and policy adjustments are necessary to enable the introduction of this important additional asset class in the supply or value chain (Zame et al. 2018, p. 1650).

3.2 Transmission and Distribution System Operators (DSOs)

A classical electricity supply chain categorization is generation–transmission–distribution. With the increasing integration of ICT into the electricity distribution system, and the change from a passive to a more active and ‘smart’ network, more system management is needed. As a consequence, the role of transmission system operators (TSOs) and DSOs will change. At the same time, the need for cooperation between the generators/suppliers of electricity and the DSOs and TSOs (and also between the latter) will increase; security of supply and system resilience will become a joint responsibility of DSOss and TSOs (Faerber et al. 2018).

The main investment burden toward a smart grid will be on the DSOs (Cambini et al. 2016). As regulated entities, DSOs will benefit from regulated tariffs/revenues to pass on to the grid users. While the path toward a full-fledged smart grid and integrated energy system can be smoothened by demonstration or pilot projects, still, in light of the tremendous change inherent in replacing and/or upgrading the established physical infrastructure by means of new sensors and controllers, and integrating more ICT infrastructure (and related software), the benefit–cost analysis of smart grid investments remains a big challenge (also in light of the fact that due to the need to decarbonize the energy system the timeline for accomplishment is less than 20 years). The uncertainty, aggravated by the time pressure, likely makes the smart grid much more costly than under more ideal planning and implimentation conditions.

With increasing shares of electricity from variable renewable energy sources (RES), system operators incur costs of integration. Hirth et al. (2015) classified these into three types: (1) grid-related ones; (2) balancing costs; (3) profile costs. The grid-related costs reflect the marginal costs (value) of electricity in space, referring to the opportunity costs of having to transport electricity from the place of generation to the place of consumption. Balancing costs arise due to forecast errors. Finally, profile costs, reflect the costs of matching demand and supply over time, and are the larger the more variable the intermittent output of RES is. Smart grids, enabling an efficient use of numerous flexibility potentials, can be expected to mitigate such RES integration costs.

3.3 Customers/End-users

Smart home and home automation technologies, containing a number of smart and integrated energy management components, also diffuse the market, enabling consumers to better optimize their energy use and match their needs, if they are prosumers, with their electricity generation and storage possibilities and preferences (Parag and Sovacool 2016). The increasing diffusion of smart grid technologies bears an unprecedented potential for interactions of prosumers with consumers, consumers with consumers, and both types of actors with a great many other actors. Energy consumer needs, behavior, and practices will be re-shaped and change, adding more risk and unknowns in the already complex transformation of the energy supply systems.

It can be expected that new customer classifications will emerge. Vulnerable customers in particular will have to be projected against rising energy bills, and may not be able to benefit from the incentives for prosumers, and may not be as responsive due to the inability to invest in smart home energy management systems and/or distributed energy resources. In order to avoid the ‘death spiral’ of grid operators suffering from rising numbers of self-sufficient (autarkic) end-users that defect from the grid, it might become necessary to use taxpayers’ money to sustain the (smart) grids. Also, revenues from grid use tariffs that are mainly based on volumetric components can be expected to decline the less grid electricity the end-users (esp. prosumers) need.

3.4 Prosumers

‘Prosumers’ (a neologism formed of the two words ‘pro-ducer’ and ‘con-sumer’; (cf. Toffler 1980) both self-produce and consume energy (sometimes also referred to as ‘prosumage’, cf. Green and Staffell 2017, to emphasize the storage unit as an integral component). The term is often used to describe a new role of private households in the politically pursued sustainable energy transition process. The role of energy prosumers is constantly evolving due to technological changes, leading to the market diffusion of new products and services, and creating new business opportunities as well as corresponding behavioral responses and demand changes, and maybe even lifestyles (Oberst and Madlener 2014). Prosumer households can be viewed either as individual, self-optimizing entities or, alternatively, as entities that are part of some energy sharing network enabled by smart grid technology, forming various kinds of communities (e.g., citizen energy communities, microgrids, and VPPs; see below) and allowing to address (environmental, economic, and social) sustainable development concerns. In the extreme, prosumers may have the ability to be autarkic/self-sustaining (provided the self-generation and storage systems are sufficiently large), and without interest to be part of an energy community, may even decide to opt for ‘grid defection’ (i.e., disconnecting from the grid), although this might be much more costly than ‘load defection’ (i.e., achieving net-zero balance between electricity taken from and fed into the grid) only (Sabadini and Madlener 2021).

Schill et al. (2017, p. 23) distinguish between the following four prosumage strategies:

  1. 1.

    Pure prosumage: Implies the complete avoidance of market transactions, restricting the optimization to the deferral of self-generated electricity to later periods via the storage unit;

  2. 2.

    Grid consumption smoothing: This implies that only prosumage storage loading from the market is enabled, allowing the storage unit to smoothen prosumers’ electricity sourcing over both the own DER and/or the market;

  3. 3.

    DER profiling: This strategy involves the activation of only discharge to the market, enabling prosumers to profile their available DER feed-in (when it is most system-friendly);

  4. 4.

    Full integration: This implies no restrictions in terms of linking the DER with the market, enabling to use the storage unit for consumption smoothing, PV profiling, and arbitrage on some markets.

There is emerging literature trying to find out how consumer and prosumer households differ from each other (e.g., Oberst and Madlener 2014; Oberst et al. 2019). Also, there is evidence that renewable energy policy measures geared toward private households, such as the reduction of feed-in tariffs—aimed at making self-consumption of self-generated electricity more attractive compared to feeding into the grid—may actually increase electricity consumption (Atasoy et al. 2021). This raises interesting questions also for the implications of pushing the transition toward smart grid technologies forward.

Markets for ‘prosumption’ services are different from others—e.g., demand–response programs or platforms—in that prosumers can also offer active services to other prosumers, electric utilities, TSOs, and others have to bid for. Parag and Sovacool (2016) argue that prosumer marketplaces will be more complex if envisaged as a multi-agent system with very different types of services, a wider variety of participant groups/types fulfilling diverse and changing roles, and a larger number of providers per prosumption service. They distinguish between peer-to-peer models: (1) organically evolving peer-to-peer models; (2) prosumer-to-interconnected or ‘island mode’ microgrids; (3) organized prosumer groups.

3.5 (New) Service Providers

Energy Service Companies

Energy Service Companies (ESCOs) Companies have been well-established for many years, typically helping to save energy (energy costs). Still, it makes sense to review and reflect on their typical services and future potentials unfolding in light of smart grid systems and component management. In Chap. “Demand Side Management”, their role is discussed regarding DR. energy service companys (ESCOs) comprise a wide range of market actors providing specific services which are typically not the task of DSOs. Through SGs, ESCOs are increasingly entering the household sphere, typically with services about energy monitoring data and better management of energy use (Verkade and Höffken 2018, p. 801). Hence, ESCOs can be seen as intermediaries between the household sphere and other energy system actors. An important subgroup among the ESCOs are the so-called ‘aggregators’. They employ ICT to make use, and bundle, distributed flexibility, offering this flexibility as a service to grid management. Alternatively, it may sell bundled energy from local supplies on the wholesale market or green energy market. In business and industry, this kind of activity has been already quite common; at the household or neighborhood/city quarter level, it is still rather novel, and often has been restricted to the aggregation of distributed battery capacities.

In the following, we discuss how utilities/energy suppliers can enrich their business models by providing aggregation services.

DER aggregators may or may not be part of the utility business. System operators will benefit from a widened access to auxiliary services offered in the form of balancing services, voltage control (Lu et al. 2020).

Creation of New Business Models

The ‘Energy Supplier 2.0’, introduced by Specht and Madlener (2019), is a conceptual business model for energy suppliers aggregating flexible distributed assets. The focus of such new business models compatible with the ES2.0 idea is on the existing and possible new customer needs and market potentials. The concept, which is based on the business model canvas approach introduced in chapter “Business Model Design”) embraces the notion of an aggregator of flexible capacities, e.g., on the household level, and unfolds how specific new energy business models can help to tap the potential of distributed flexible energy assets. Such assets can be seen as DR (see Chap. “Demand Side Management”, but can also go beyond as well (e.g., including flexible generation and storage units as well)).

Smart grids, by enabling and stimulating customer/end-user empowerment and market entry of new players (e.g., ICT firms, ‘big tech’), have disruptive potential also in the sense of ‘creative destruction’ in the energy business. An interesting question is whether electricity companies may either benefit or suffer from the deployment of smart grids—given that many of their established business models may no longer work (i.e., be no longer profitable). The main elements of any business model are value creation, value delivery, and value capturing (see also chapter “Business Model Design”). While many energy companies may eventually succeed to adapt and innovate their business model portfolios successfully, there is a lot of risk and uncertainty involved, e.g., related to customer engagement (needs and behavior), government support, and new market entrants/competitors. Several factors enable, whereas others inhibit, the transition of electricity firms toward doing business in the new, ‘smart’ grid environment (Shomali and Pinkse 2016). The main impacts of smart grids on electricity companies’ smart-grid-based value creation ability are summarized in Fig. 5, grouped by enabling and constraining factors and the three business model elements, value creation, value delivery, and value capture. It provides a balanced view on not only the chances but also the risks and uncertainties involved. The latter may incentivize firms to wait until major uncertainties have been resolved, or at least mitigated, and to scrutinize in detail whether complementary technologies, infrastructures, and institutions are developed in parallel in order to enable smart grids to actually come to fruition in the way it is hoped for.

Fig. 5
figure 5

Enabling and constraining impacts of smart grids on energy companies’ business model innovation activities (in terms of value creation, value delivery, and value capturing) (Shomali and Pinkse 2016, p. 3839)

Figure 6 depicts a set of companies active in business model innovation on the electricity retail market in Germany as well as a pattern grouping of different business activities (shown in a special type of ‘energy trilemma’ representation). It shows that most pattern groups investigated (top left corner) can directly link to a particular form of value creation. For example, the subscription and pay-per-use patterns are associated with mainly economic value creation, the open business, layer player and energy solution patterns mainly with environmental value creation, and cross-selling patterns mainly with socio-economic value creation. In contrast, the digitalization pattern is an integrative pattern group providing equal opportunities to all three forms of value creation. The same is true for the direct sale pattern group, although slightly less clear and somewhat biased toward economic value creation.

Fig. 6
figure 6

Business activity clusters and companies active in smart-grid-enabled retail services in Germany (Karami and Madlener 2021, p. 13)

Fig. 7
figure 7

Business model canvas for future electricity retailers (Karami and Madlener 2021, p. 13)

Figure 7 shows the proposed business model canvas (\(\rightarrow \) chapter “Business Model Design”) for future retail electricity suppliers as the perceived optimal business model canvas. The intuition behind this proposition, which is corroborated by an empirical study of Karami and Madlener (2021) among 11 major German players in the retail electricity market, is that subscription and pay-per-use patterns for renewables electricity are found to be the most popular business models applied. By generating predictable revenues with lower sale costs, these patterns provide direct impacts on companies’ revenue streams. Consumers presumably concentrate heavily on the high, and quickly rising, financial costs resulting from higher electricity end-use prices, along with the loss of time and effort needed to subscribe when deciding whether and what renewable electricity to buy. Such substantial transaction costs are seen as disincentives to subscribe, especially where tangible advantages are not straightforwardly apparent.

Citizen Energy Communities

Citizen energy communities (CEC), as the name implies, are geared toward local communities, with the aim that these are owned and/or (directly or indirectly) governed by the citizens themselves (the ‘democratization’ in the ‘4Ds’, so to speak). Gui and MacGill (2018) identify three CEC types based on how communities interact with the energy system: centralized, distributed, and decentralized CECs (Fig. 8) . In general, CECs are “social and organizational structures formed to achieve specific goals of [their] members primarily in the cleaner energy production, consumption, supply, and distribution, although this may also extend to water, waste, transportation, and other local resources” (Gui and MacGill 2018, p. 95). This paper focuses only on electrical energy in the context of citizen energy. To illustrate this, in the following we examine the concept of different types of energy communities as well as potential benefits and challenges of citizen energy.

Fig. 8
figure 8

Citizen energy community typology (centralized, decentralized, and distributed) (Gui and MacGill 2018, p. 100)

Centralized CEC are characterized by a relatively high level of cohesion, not necessarily in terms of spatial co-location but rather in terms of interaction. Its aim is to foster the achievement of common goals; members are normally directly connected with each other and conform to roles and social rules defined by the community. Rules and activities are typically managed by some governing body that is also controlling the communication and access to the members and with external parties (Gui and MacGill 2018, p. 100).

Distributed CEC are characterized as a “network of households and businesses that generate or own distributed generation individually, connected through a controlling entity either physically or virtually, and sharing the same rule in supplying and consuming electricity within the network” (Gui and MacGill 2018, p. 101). They are composed of a number of mostly homogeneous members who are not close to each other (in a spatial, normative, or cognitive sense). Most members are not directly connected with each other, and the boundary is transitory, partial, and permeable (linked by cross-cutting ties).

Decentralized CEC, finally, are “a community of households, businesses or a municipality that generates and consumes energy locally for self-sufficiency that may or may not connect to the main grid” (Gui and MacGill 2018, p. 102). The distinction from centralized and distributed CECs is due to its capacity for self-sufficiency and autonomy from the main grid. Decentralized CEC members typically belong to a spatially constrained area, such as a neighborhood, village/town, and municipality, and members may own DER individually, or collectively as a group, and sometimes even the distribution infrastructure as well (e.g., community-owned microgrid and integrated community energy system).

Figure 9 shows the citizen energy community setup in terms of actors involved and key relationships. A community-owned microgrid may involve different asset categories (generation, distribution, and microgrid), and be owned by single or multiple parties (e.g., the community, a utility, and other public/private enterprises). Generation assets include residential/commercial solar PV, storage units, and other DER (incl. demand-side ones). Distribution assets comprise all physical components that are part of the local distribution grid. Microgrid assets include the central controller of the microgrid, a central energy management system, smart meters, and a real-time communication and control unit.

Fig. 9
figure 9

Citizen energy community architecture (Gui et al. 2017, p. 1357)

Table 1 gives an overview of, and comparison between, the provider and customer relationships in a centralized electricity supply system versus one that is based on CEC (community microgrid). As can be seen, community microgrids are an emergent new customer service provider relationship which enables to reassess and redefine the role of customers. It allows for a new way of thinking about social inefficiencies and the disconnectedness of supply- and demand-side actors in the present-day centralized, hierarchical energy systems. Obviously, it also comes along with huge challenges in terms of structural, institutional, and regulatory changes to the centralized system required, and new business models (as discussed in the previous section, and in more detail in the chapter “Regulatory and Institutional Aspects of Smart Grids”).

Table 1 Customer and provider costs, benefits, and risks, centralized versus CEC power supply (Gui et al. 2017, p. 1361).

Economics of Microgrids, Virtual Power Plants, and Virtual Microgrids

Apart from the already discussed Citizen Energy Communities, many other ideas of how groups of consumers can be organized, and clustered, such that they benefit from grouping and following their joint interests and achieving their common targets, have been proposed for sharing the resources pooled, and potentially shared in a smart grid system.

Microgrids have been defined by the U.S. Department of Energy as “a group of interconnected loads and distributed energy resources within clearly defined electrical boundaries that acts as a single controllable entity with respect to the grid and that connects and disconnects from such grid to enable it to operate in both grid-connected or ‘island’ mode.” The ‘4 Ds’ point to an increase in the importance of microgrids in the future. Three sizes have been distinguished in the literature: small-sized microgrids (e.g., commercial buildings); medium-sized microgrids (e.g., communities); and large-sized microgrids (e.g., universities and military facilities) (Hanna et al. 2017, p. 47). Whether or not public benefits of microgrids are realized—e.g., in terms of improved power quality, grid reliability, improved black-start capability, lower cost of electricity or pollutant emissions—depends not only on the business model adopted but also the system boundary of the analysis. What might be beneficial for a specific local commercial building, a local community, or a university campus is not necessarily beneficial to society on a wider scale, as both in grid-connected or in island mode the microgrid might pose a burden on the public (i.e., the wider) grid. Likewise, there might be distributional effects resulting from the operation of the microgrid that are considered either as desirable or undesirable, depending on how the benefits and costs are shared between the energy end-users. Van Leeuwen et al. (2020) have investigated the social welfare implications of operating an integrated blockchain-based energy management platform for microgrid communities that also enables bilateral trading. For a real-world case study (a prosumer community in Amsterdam), they compare three scenarios (trade-only, grid-only, and hybrid) with a baseline scenario, finding that electricity import costs can be reduced by some 35% compared to the baseline, import volumes by 15%, and peak imports from the grid even by more than 50%. However, despite these impressive figures, total social welfare in the community was found to be highest without a trading mechanism, and the platform is only viable when sharing all costs equally between all households. Practical feasibility will also depend on social acceptance, and the social welfare might change over time, calling for some dynamic analysis or an evaluation over an extended period of time.

Prosumers can be clustered to virtual, orchestrated units, enabling the aggregator to participate in energy markets as one entity, thus greatly reducing total energy costs due to higher forecasting accuracy (especially if prosumers face penalties in the case of imbalances/load deviations) (cf. Vergados et al. 2016, p. 90).

The virtual power plant (VPP) concept has been widely investigated and used for managing geographically dispersed generation and/or storage units, typically managed by (or a utility or other energy company. Virtualization techniques and ICT are used for the optimized management (orchestration) of the DER (Vergados et al. 2016).

The VPP variant of Virtual Micro-Grids (VMGs), in contrast, has slightly different characteristics (cf. Vergados et al. 2016, p. 92):

  • Management of very small energy prosumers by an aggregator (in contrast to large generation/storage units owned by a large utility company);

  • Any type of small-scale facility able to produce or consume a small, or negligible, amount of energy (e.g., some public/municipal lamp-posts);

  • Very different monitoring and control functionalities (e.g., low-cost VINSEM gateways that are customizable, backward-compatible, and communication protocol-agnostic);

  • The VINSEM solution proposed is offered to the VINSEM prosumer who needs to install the VGW and required sensors and control equipment. In contrast to standard VPP cases, the VGW adopts some open-source software implementation, thus not requiring complex smart grid standards;

  • VMG system requirements are different (e.g., data acquisition, communication, decision-making, and active VMG control management);

  • Only RES prosumers are considered (thus aggravating the challenges due to RES intermittency);

  • In contrast to VPP concepts, where one actor (the DER aggregator) manages the assets, the VMG concept foresees a new actor (the VMGA) which cooperates with various SG stakeholders (DSO, TSO, balance responsible party (BRP)) to enable the VMGA to participate in the (liberalized) electricity markets, and to react to specific events (e.g., in the case of a local congestion problem declared by a DSO, the VMGA needs to be sure that only prosumers associated with a particular low-voltage substation are eligible to participate in the prosumer clustering process).

3.6 Local/Energy Markets

The deployment of DER enables to turn regular (passive) final consumers into active contributors to the local supply of electricity, both in terms of energy, capacity, and reserve/balancing energy. Digitalization of the power distribution grids (‘smart grids’) and innovative regulation enables peer-to-peer trading, (as, e.g., the last amendment of the German EEG contains; cf. EEG, 2021) but the design of local energy markets is still in its infancy and demonstration projects still prevail.

The increased use of variable renewable energy sources raises the need for flexibility that enables it to respond quickly to fluctuations in supply and demand. All the different flexibility options at large- and small-scale (e.g., flexible generation, demand response, and storage), and particularly those enabled by smart grid technologies, could be thought of competing in a dedicated new market for flexibility where flexibility providers (typically aggregators) and other parties in need for flexibility (typically grid operators) meet and trade with each other on a level playing field (Council for European Energy Regulators 2019). Such markets can be thought of operating locally, regionally, or nationwide, potentially enabling to reduce losses due to efficient, low-cost local load balancing. Multi-layer trading of flexibility on dedicated platforms, and increasingly automated decisions of smart DER enabled by ICT, will allow an increasingly effective and efficient orchestration of the manifold resources playing an active role in the energy system. Transparency and a clear regulatory framework for DER will be paramount for efficient, and ideally also social welfare-optimal, flexibility market outcomes. In the following, some more issues related to local energy market design and peer-to-peer trading are discussed.

Local Energy Markets and Peer-to-Peer Trading

In this section, it is described how an auction model for a local reserve energy market can be designed that enables to also accommodate the special needs of non-expert bidders such as private households. The model can be used to revolutionize the reserve energy market, as a glsbrp, in contrast to today’s standard practice, and is given the chance to self-supply reserve energy. Thereby, it serves at least two purposes: (1) it helps to further integrate DER; (2) it can help to lower the costs for reserve energy by mitigating the market power of the currently dominating, large-scale utility companies.

End-use energy consumers can benefit from this newly designed market twice, in that they are the ones providing the energy and getting paid for it, and in that their energy bill can be lower once the market offers reserve energy at a lower cost. At the same time, the mechanism supports the remuneration and subsidy schemes for DER that are already in place. In the longer term, when subsidy schemes are eventually phased out, it can serve as a long-lasting incentive scheme for investments in the designated technologies. It can be shown theoretically (Rosen and Madlener 2013), for a symmetric and an asymmetric setup, that the information policy in the market has a significant influence on the speed of convergence and also a small effect on the equilibrium market price that is finally reached. In the extreme case where no information is provided, the effect on the equilibrium price becomes substantial. Even more importantly, this effect is sustained indefinitely, which points out the importance of the market design choice.

Given the special characteristics of bidders in a local energy auction, a problem that needs to be solved is to find an adequate and at the same time reliable remuneration for each provider of reserve capacity and energy. Such an auction mechanism needs to be simple and easy to understand in order not to turn off potential participants. Furthermore, transaction costs in a market with such small quantities need to be low in order to leave room for at least a minimal profit, and opportunities for strategic behavior should be kept to a minimum. The design of an auction for such a purpose comprises many parts. Auctions for electricity are a specific type of auction, as the good is perfectly divisible and non-storable, which means transactions need to happen in real time, or at least at a predefined point of time in the future. The type of auction required can be compared to the treasury auction, which has received considerable scientific attention in the past. So far, game-theoretical analysis of reserve energy auctions with the properties needed in a local market is still very limited.

P2P trading enables direct interaction between local market participants without the involvement of third parties. As such, it is an alternative that also enables to switch energy suppliers on a high frequency (e.g., minute-by-minute) basis and to buy and sell electricity based on one’s individual preferences (prosumers as active ‘producer-consumers’). P2P might involve blockchain technologies in order to keep track of the transactions made and to offer a transparent and at the same time automated settlement of the market transactions that occurred (van Leeuwen et al. 2020; Morstyn et al. 2018).

Still, such P2P energy trade concepts are still at a very early stage, and there is a lack of consensus regarding what market design or business model is best to develop such advanced local energy markets, and how the interplay with the established electricity markets (intraday, day-ahead) is to be organized (Lüth et al. 2018). It leads to several market design questions, some of which are discussed in the chapter “Regulatory and Institutional Aspects of Smart Grids” of this textbook.

Crucial issues are the merit order of the various flexibility options over time, the fact that these may, or may not, all be offered in a single market for flexibility (which would bring them into direct competition and lead to some cost-efficient outcome), and the many interdependencies that not only lead to high uncertainties regarding investment decisions (potential ‘missing-money problems’, e.g., if a certain business model suddenly becomes obsolete and unprofitable) but also energy end-user costs (esp. small-scale energy consumers are often risk-averse and do neither have the expertise nor the capacity for hedging their risks, which will require aggregators and other players to offer such services).

Fig. 10
figure 10

Peer-to-peer trading platform architecture (Zepter et al. 2019, p. 165)

Fig. 11
figure 11

P2P community and diversification synergies (Zepter et al. 2019, p. 169)

Zepter et al. (2019) study how prosumers can be integrated into wholesale electricity markets, and what synergies can be reaped by P2P trade and the use of residential storage units. Figure 10 provides an overview of how the system architecture looks like. The prosumer community is depicted as a number of buildings/households; based on a renewable power generation forecast, these bid into a day-ahead market. At this stage, each household plans its grid consumption, P2P trade, battery storage utilization, and the grid feed-in. The community submits at this stage a commitment to the day-ahead market that takes into account the wholesale electricity price and local wind and solar power generation uncertainties. In a second stage, the community needs to balance any deviations from the day-ahead market commitment by adjusting P2P trade, battery usages, and grid electricity procurement from the intraday electricity market. The households are somewhat heterogeneous (enabling some diversification effect), and are assumed to be connected both to the main grid as well as interconnected with each other by a local grid. The objective of the optimization problem could then be to minimize the community’s expected costs of procuring electricity from the (intraday and day-ahead) wholesale markets.

The interplay and synergies of pooled heterogeneous households engaged in P2P (as well as wholesale market) trading are illustrated in Fig. 11. It visualizes the model community constructed, and shows the basic characteristics of the buildings/households considered in terms of technology portfolio and load.

In a study on the value of local P2P trade, Lüth et al. (2018) investigate two different market designs in the context of battery flexibility (decentralized, privately owned batteries in private households versus a centralized, commonly accessible, and thus shared battery). They find that P2P trade can save more than 30% of the electricity costs for a community, allowing for a significant increase in self-sufficiency, and utilization of local renewable energy resources. Note, however, that in such studies the assumption is made that the smart grid and related digitalization technologies are already installed (i.e., not accounting for those costs, considered as ‘sunk costs’, and financial risks).

4 Governance, Policy-Making, and Broader Regulatory Implications

Smart grid technologies enable the adaptation of the electricity supply system to the challenges ahead. The development path requires a coherent policy and regulatory framework that enables, and safeguards, a smooth innovation path. Apart from economic, social, and ecological sustainability, a major energy policy goal is the security of supply, especially when it comes to the power supply system which is particularly complex and sensitive to supply-demand shocks. The latter is increasingly investigated under the heading “resilience”.

A major challenge from a regulation perspective is to find out, and then to decide, how much business can safely be left to the market (with an adequate framework that safeguards a level playing field) and how much needs to be left in the hands of regulated entities such as DSOs. In any case, the regulator needs to be ready and running to deal with the challenges ahead.

Future business models and cases depend on the ability to provide multiple services—such as reserve energy and capacity, balancing energy, and arbitrage. The economics depend strongly on the regulatory framework. The regulatory framework in place does not match the rapidly increasing complexity of smart grid energy supply systems. Researchers, regulators, and policy-makers alike are asked to proactively reform markets, regulation, and policies toward smarter energy systems.

Proactive regulatory change and innovation can help not to slow down the evolution of new (utility) business models. Grid users and new businesses will arbitrage the widening gap between new technological and market realities and the established regulation. If regulatory change cannot keep up with the changes happening to the energy supply systems, and the electric power system in particular, then large inefficiencies might occur (MIT Energy Initiative 2013).

Smart energy systems, increasing energy system integration facilitated by ICT and ML, as well as the re-emergence of multi-business utilities call for multi-sector regulatory bodies that will have to closely overlook the activities, define adequate regulatory frameworks (incl. penalties for non-system-compatible behavior), and to steer the course toward efficiency and system resilience. Efficiency is ultimately determined by not only efficient markets but also efficient and effective institutions and proactive regulation, as well as the exploitation of economies of scale and economies of scope (Jamasb and Llorca 2019, Sect. 3.3).

4.1 DER Regulation

DER can provide multiple services, depending on the type of DER (storage being particularly versatile), including energy arbitrage, investment deferral of conventional (generation, transmission, and distribution) capacity, ancillary services, ramping, end-user applications, and curtailment of power generation from renewables (for a discussion specifically for energy storage as a specific type of DER, see Sioshansi et al. 2012, p. 48). Still, today there is a situation of incomplete markets and thus also incomplete quantification, capturing, and valuation of such services that often stem from multiple value streams. As a consequence, there are also no incentives in place yet for siting DER at locations where they provide the greatest benefit to the system. The often still lacking market diffusion of smart grid technologies precludes the real-time dispatch of DER and the provision of numerous services that will eventually be possible once the smart grid has become a widespread reality. The higher complexity of DER integration from the system planner’s perspective, and the lack of a more holistic simulation of networks with DER by the system operator/s can be expected to slow down, or even prevent, part of the DER installations and exploitation for many more years, even in cases where they would be expected to be more (cost-)effective than utility- or DSO-owned assets controlled by them primarily in terms of system support. Investors in DER can be thought of being given the choice to either sell their flexibility freely in some emergent flexibility market, or to benefit from some clearly pre-specified regulated service offered under some regulatory regime and rate base—as it has been common for conventional power generation, transmission, and distribution assets. Not only from a social welfare but also a business perspective, it is important to find out which regime is preferable (from a static and dynamic viewpoint, or in the shorter and longer terms, respectively). The lumpiness and irreversibility of investments, but also the presence of market power, can lead to inefficient and suboptimal investment in DER, although lumpiness and irreversibility can be assumed to be the smaller the more decentralized (and modular) the assets in question are. Modeling and understanding possible strategic behavior—of all actors involved (incl. DER operators) is important to understand the private and external (or systemic) value of DER. Both expected future costs and revenues from a DER investment are hard to quantify, partly by the continually changing environment, the need to re-optimize the DER operation, and uncertainty regarding flexibility market prices over multiple time scales (Sioshansi et al. 2012, pp. 52–53).

4.2 Smart Grid Regulation

Policy-makers are challenged to take the right governance decisions in terms of regulating the markets involved in the smart grid development—including, standardization and standards, DER, prevention of cyber-attacks and privacy infringements, system resiliency, and sustainable development. This enables them to adopt and adapt their support schemes in order to mitigate investors’ risks and to enable them to properly value and capture the benefits and incentivize system-friendly behavior allowing for social welfare maximization overall. In light of the enormous complexity of not just the co-evolution of the smart grid but also all the services enabled now and in the future, such a holistic social welfare cost–benefit analysis is a formidable task to accomplish (and to track over time in order to allow some timely ex-ante, or else short-run ex-post, corrections of any shortcomings that may arise).

Fig. 12
figure 12

Participants in the (smart) grid regulation process (Nolting et al. 2019, p. 756)

4.3 Governance and Policy-Making

New governance and regulatory policies are needed to shape the evolution of smart grid-based energy systems, and to enable the distributed and flexible assets to be orchestrated for a flexible and resilient supply system that is able to efficiently deal with all sorts of dynamics that might occur, and without jeopardizing security of supply. Ideally, it is also able to maintain, or even enhance, both a certain level of competition amongst suppliers of resources as well as of distributional justice (to be discussed amongst society what this means). Also, institutional inertia have to be tackled, new knowledge built up to be able to design new policies and regulation as the system/s evolve/s (e.g. on consumer engagement, cyber- and physical security, resilience in different dimensions). Ideally, governance, policy and regulation stays abreast of the changes, in order to enable ex-ante measures, rather than reacting ex-post (i.e. in retrospect), which likely creates higher costs than if measures are adopted on time and with some foresight about expected trends. Also, regulatory guidelines to promote the development of the smart grid should aim at improving cost efficiency of the power (or further integrated) system, but also try to optimize social welfare, embracing as many aspects as possible and manageable (e.g., distributed generation, network automation, demand response, reactive power management; cf. e.g., De Oliveira-De Jesus and Henggeler Antunes 2018).

An important barrier to smart grid investments is the lack of an appropriate regulatory framework (see also the chapter “Regulatory and Institutional Aspects of Smart Grids”). Besides, timely investments in electric grids can (and need to) be adequately incentivized.

Nolting et al. (2019) study the proper design of incentives for grid infrastructure investments based on social welfare considerations, and taking information asymmetry (principal-agent problems) into account. They further propose a so-called capital expenditure adjustment rule aimed at avoiding delayed cost recognition and leading to ill-incentivized (delayed) investment. It remains to be seen how useful the approach is under high uncertainty, as in the case of smart grid investments, where the value of waiting might severely delay the investment, which however might be perfectly rational (and even social welfare optimal, which needs to be assessed case by case). Principal–agent problems, (the lack of) property rights, and information asymmetries need to be taken care of, too. Figure 12 depicts the main participants engaged in the grid regulation process. As can be seen, three participating parties interact with, and mutually influence, each other in the grid regulation process:(1) the legislator; (2) the regulatory authority; (3) the regulated DSOs. Due to the information asymmetries between these participants, different principal–agent relationships may occur. For instance, conflicts between the goals of the regulatory authority (as the principal) and the DSOs (as the agents) may arise. While the regulator’s target is to raise/maximize social welfare (by reducing total energy system costs, etc.), the DSOs aim at profit maximization. The problem due to diverting goals can be mitigated, e.g., if the incentive regulation aims at inciting the DSOs to act in a more efficient and welfare-oriented way—by linking their profits to societal goals. Information may also be asymmetrically distributed between the parties concerned (e.g., in terms of an information deficit on the side of the regulatory agency with regard to specific operation and management data of the individual DSO). Still, the regulatory authority may have some information advantages, too, e.g., by having an overview of the whole regulated sector, whereas individual DSOs usually only know their own data.

5 Smart Grid Investments

Smart infrastructure technology and information system investments are needed to cope with changes in demand (patterns and levels), as well as increasing shares of variable power generation. New pricing schemes and mechanisms will be necessary for funding both the investment costs and operation & maintenance (O&M) costs, as will be re-regulation that takes the change of the technical standards and the evolution of the energy supply system into account (on regulatory issues, see also the chapter “Regulatory and Institutional Aspects of Smart Grids”, on the analysis of evolution strategies based on synthetic grid topologies see Pagani and Aiello 2015).

Values that accrue from smart grid investment can arise from different business models (and to different stakeholders involved). If only costs and benefits of individual smart grid investments are considered that accrue to the investing actor (e.g., a utility), then further value creation is not accounted for. This calls for an extension of the boundary for value (or cost–benefit) analysis arising from smart grid investment.

In principle, the cost for enhancing the current distribution grid toward a smart grid needs to be gauged against the benefits of higher connectivity. The main grid cost components are (1) losses (both in lines and at transformer substations); (2) security and capacity factors (robustness—e.g., evaluated by random removal strategies); (3) line redundancy (e.g., by analyzing a random sample of the nodes in the network and calculating the first n shortest paths of increasing length; the worst-case path between two nodes is considered); (4) power transfer limits (e.g., in terms of maximal and average current supported by a line) (cf. Pagani and Aiello 2015, p. 170)).

Investments in smart grids and smart grid applications need to pay off—both from an individual actor (private households, firms, and utilities) and social welfare point of view. From a distribution grid operator’s point of view, it could be analyzed how long it takes to amortize smart grid investments when a locational–marginal pricing scheme is applied (say, due to the regulatory framework in place). Such research has been done in the past, either using relatively simple economic models or applying models that take the multi-level structure of smart grids explicitly into account. Some of this research also deals with social welfare implications and social welfare optimization. An example of this kind of research is Jesus and Henggeler Antunes (2018), where the authors have explicitly analyzed the benefits of DG, network automation, demand response, and reactive power management, respectively, exploring progressive levels of implementation (in terms of expenditures in smart grid technologies). In their analysis, they find that in weak power systems, DR applications are suitable to be recovered in the short term through the marginal revenues gained by the network provider.

Fig. 13
figure 13

The smart grid investment problem (Hall and Foxon 2014, p. 606)

On the one hand, ideally, increased flexibility from the various flexibility options will reduce the need for infrastructure investments. On the other hand, major new investments are needed to make the system smarter.

While the potentials for efficiency gains through smart grids are largely undisputed, the issue of investment and related risks and uncertainties is more controversial, and needs some more attention. Investments in smart grid infrastructure are a key enabler for reaping the benefits of smart grid systems, as well as for exploiting the net gains of ‘smart investments’. Still, there can be a mismatch between where benefits and costs occur, leading to problems of value capture and redeployment (Hall and Foxon 2014). Also, some smart grid benefits are harder to quantify, and to price directly, such as security of energy supply and decarbonization (both can be considered as public goods). Value capturing and structural incentives need to be well understood, also in a (co-)evolving system—i.e., dynamically over time. Hall and Foxon (2014) discuss the political economy of smart distribution systems. Drawing on urban political economy they discuss in detail how smart grid investments can benefit municipal economic development, and are altered by changes in municipal-level value creation. In addition, they also characterize the co-evoluation of value capturing and structural incentives in the power distribution system. Obviously, the benefit of smart grid investments will also depend on consumer acceptance and willingness to pay for smart services (Lineweber 2011; Toft et al. 2014).

Apart from social acceptance, and ‘socio-economic acceptability’ (Bigerna et al. 2016), there can be numerous barriers, and interactions among them, hindering the adoption of smart grid technologies (Luthra et al. 2014). Figure 13 depicts the smart grid investment problem, illustrating the business models and institutional settings that create (or enable to exploit) three municipal economic values usually not accounted for (i.e., RE connection co-ordination, inward investment stimulus, and municipal supplier DR management).

Risk and uncertainties related to smart grid deployment can take many forms, and affect multiple categories: markets, users, data and information, supply mix, policy, investment conditions, and networks (cf. Connor et al. 2018, p. 1).

Cambini et al. (2016) reviewed 459 innovative smart grid projects in 30 European countries realized between 2002 and 2014. Their focus was especially on (1) distribution sector concentration (i.e., the level of market concentration in the power distribution sector); (2) regulatory mechanisms (i.e., the capacity of the regulatory scheme to provide incentives to DSOs for increasing either their productivity or cost efficiency); (3) innovation–stimulus measures (i.e., the mechanisms designed by regulatory authorities to stimulate the implementation of innovative SG pilot projects). The insights are useful for reflecting on the need for regulatory reform to stimulate innovation. These authors find that in countries with lower market concentration, DSOs invest on average much more (by a factor of 2) in SG than if the market concentration is high. Second, incentive-based regulatory schemes were found to also spur SG investment/innovation significantly more than cost-based regulation (by a factor of about 1.5). Finally, the adoption of innovation–stimulus mechanisms by regulation (e.g., adoption of an extra WACC or adjusted revenues) is rather successful in promoting SG investments (difference of a factor of 2.5).

DSO/DER regulation can be effected in different ways (Agrell et al. 2013):

  1. 1.

    Integrated regulation. An integrated DSO/DER entity considers the total cost of undertaking smart grid investments; the regulator, who does not know the costs but only forms expectations, makes a take-it-or-leave-it offer to the integrated entity.

  2. 2.

    Decentralized regulation. In this variant, the regulator makes an offer to the DSO. The latter decides what to offer to the DER investor. If the DSO investment costs exceed the budget alone, no investment can take place. The regulator’s desire to limit the DSO–DER information rents will generally lead to underinvestment. Moreover, the DER is rationed more harshly than the DSO, because the DSO must pass on information rents to the DER.

  3. 3.

    Centralized independent regulation. The regulator signs separate and independent contracts with the DSO and DER, and rations to lower information rents. It implies a unilateral commitment by the regulator to finance the investment irrespective of the coordination in the supply chain.

  4. 4.

    Centralized conditional regulation. In this variant, the regulator offers the investment possibility to the DER investor first; if that one accepts and undertakes the investment, she also offers the investment to the DSO. An obvious advantage of this variant is that a situation where the DSO invests but the DER investor does not is avoided. Overall, the outcome of this arrangement may often be that the regulator should refrain from any investment to begin with (i.e., much like in the variant with individual regulation). More specifically, it is assumed that the regulator can make a conditional regulation in the sense that a separate contract is offered to the DSO and the DER investor first; an accepted contract by one party becomes only valid if the other party accepts the contract it was offered as well, thus avoiding the losses inherent in the unconditional centralized variant that arise when only one party accepts.

The interplay between smart meters, smart grids, and demand response can be analyzed, e.g., for the case of automated load control. The provision of real-time information about grid usage and nodal prices in distribution grids without any load-side application is useless, whereas installing DR without smart metering and information provision devices will not enhance social welfare (Agrell et al. 2013, p. 662).

Figure 14 depicts a model to systematically analyze DER investment scenarios. Case A depicts the integrated case where the unbundling requirement on the DSO is relaxed. This allows the DSO to undertake direct investments in DER activities. Case B (separate decentralized incentives) depicts a decentralized DSO where the regulator contracts with the DSO and delegates the coordination of DER investments to the DSO. In Case C (separate and centralized incentives), the regulator coordinates the DER investments directly via centralized incentives.

Both DSOs and owners of DER can undertake the investments considered useful and profitable (technologies, measurement equipment, protective devices, etc.), and social welfare gains may materialize either when both parties invest (complementarity) or when it is sufficient if one of them invests (substitutability).

In a smart grid system, there are totally new relations between the economic actors—not just suppliers and consumers of energy (services) but also prosumers (producer–consumers (Fig. 15)). Overall, this transition will also create major shifts in consumer behavior, lifestyles, and culture (Bigerna et al. 2016).

Fig. 14
figure 14

DER and DSO investments and regulatory organization/delegation (Agrell et al. 2013, p. 661)

Fig. 15
figure 15

Conventional versus smart metering, unilateral versus bilateral information flows and actors involved (Ekanayake et al. 2012)

6 Different Time Scales of Smart Grid Optimization and Impacts

6.1 Short Term Versus Long Term

The time domain is particularly important, as smart grid technologies—combined with new management and optimization techniques—not only enable much higher resolution analyses and problem-solving but also impose higher requirements on energy system modeling (see also the chapter “Modeling Smart Grid Systems”), ICT infrastructure, and computational power.

In the very short term, smart solutions to demand response and dispatching—and the increased flexibility and smartness in the system—enable an unprecedented matching of supply and demand.

In the longer term, it remains to be seen whether that leads to lower risk for investment decision-making, which is likely also much affected by radical innovation and poor or lagged regulation, among other influencing factors on stability and resiliency of the system.

Well-designed electricity markets and regulatory schemes can help to enhance the flexibility of the power system, and provide a level playing field for flexibility providers offering services (e.g., DR, generation and storage capacity, energy, reactive power) that enable the efficient and smooth operation and evolutionary further development of the smart grid system. The design and implementation of faster markets (i.e. such with higher temporal resolution and response times) avoids unnecessarily high pricing (e.g. triggered by less flexible resources). It also enhances the possibilities for the integration or higher shares of VRES (Cruz et al. 2018, p. 348).

6.2 Machine Learning and Energy Economics

Artificial intelligence (AI), and especially its subdomain ML, is about to become a key enabler of a more complex and considerably more data-driven energy industry. The industry is changing rapidly and dramatically in the way energy supply, trading, and consumption are organized and managed. AI can outperform traditional approaches and models in terms of controllability (autonomously by smart software), handling of big data, smart grid system operation/management, predictive maintenance, cyber-security, energy efficiency optimization, and much more (Cruz et al., Ahmad et al. 2021).

Smart grids offer new fields of application for ML techniques also with regard to the economic analysis of actors’ behaviors and present and future behaviors of energy markets. Examples include demand forecasting, trading strategies for aggregated entities, analysis of (e.g., macro or energy) trends, energy price forecasting, and risk management (Ghoddusi et al. 2019). While economists are often concerned about the limitations of ML in terms of lacking theoretical foundation and causality inference, others are excited about the enormous potential for new ML applications in the energy domain, and the opportunities and benefits offered.

The most popular approaches used so far in the energy economics literature seem to be vector machines (SVMs), artificial neural networks (ANNs), and genetic algorithms (GAs).

A major difference between ML and traditional econometric modeling and analysis is the capability (and superiority) of ML algorithms in handling and analyzing large amounts of both structured and unstructured data, and to enable fast decisions or predictions. ML models neither require any structural modeling—i.e., assumptions of functional forms describing the interaction between variables—nor about stochastic distributions. ML has been shown to dominate statistical approaches in terms of learning more complex feature representations, so that, e.g., neural networks, support vector regression (SVR) and random forests typically outperform linear regression, lasso regression, or Box–Cox transformation regression, at least in terms of electric load forecasting. Still, there are also risks involved in using ML due to imperfect knowledge of the data or poor understanding of the algorithm/s used (many of which are ‘black box’ algorithms; cf. Kell et al. 2021).

In smart grids, where new data can in principle be provided in real time, online learning methods can be used to raise the accuracy of predictions. Online learning is particularly useful in the case of non-stationary data, and also for time series data where the recalculation of the algorithm would take a prohibitively long time. In contrast, offline learning methods need to be retrained each time new data has become available, leading to interruptions in the modeling.

Among the most frequent applications of ML in the energy economics literature—predicting energy prices (43%), predicting/modeling energy consumption/demand (39%), model calibration (2%), trading strategies (2%), structure of energy systems (7%), policy analysis (6%), and data management (1%) (cf. Ghoddusi et al. 2019, p. 712)—it can be expected that there will be some shifts in relative shares (and some new topics being added) when SG increasingly becomes a reality. Also, combining ML with econometric models, which has been a common approach aside from combining ML approaches (so-called ‘ensemble approaches’), many new fields of application can be thought of.

In the prosumer analysis domain, which rises in importance as the number and relevance of prosumers will increase, ML can be used in predicting the energy demand and for identifying consumption patterns in prosumer households. Moreover, it can be used in solving the energy trading problem among prosumers (peer-to-peer trading, P2P) of a future electricity distribution system. Wang et al. (2021), for instance, propose a fully distributed energy trading framework based on ML to optimize the load and price prediction accuracy and energy trading efficiency.

Chapter 7 of this textbook is dedicated to smart grid business analytics, distinguishing among descriptive (business intelligence tools and processes for ex-post analysis), predictive (revealing patterns in the data useful for the prediction of future events), and prescriptive analytics (comprising the former ones and providing optimization-based guidance for decision-makers).

7 Case Study: The UK Smart Grid Initiative

In the United Kingdom, substantial progress has been made over the last years in the deployment of smart grids (Jenkins et al. 2015). A special focus has been put on the distribution grids, where the perception is that early action is needed. Among the expected benefits are not only the reduction in grid losses but also to enable distribution grid operators to better manage their carbon footprint.

Among the main actors involved are the government (Department of Energy and Climate Change, DECC), the regulatory authority (Ofgem), various grid operators, equipment manufacturers, and academia. Considerable investment has been made in smart grid research, development, and demonstration (RD&D) projects. Projects have been realized through several initiatives, such as the Office of Gas and Electricity Markets price control model (of Ofgem), which puts a lot of emphasis on supporting grid innovation. Moreover, the (competitive) funds Low Carbon Networks Fund (LCNF) and its successor fund, the Electricity Network Innovation Competition (ENIC) fund, provide the funding for grid operators to realize innovation projects and to test new smart grid technologies and solutions (Jenkins et al. 2015, p. 413). Smaller funds that also support to some extent smart grid projects are the Innovation Funding Incentive (IFI) and its successor, the Network Innovation Allowance (NIA).

The UK has also started to roll out smart meters, with the aim to improve the grid management possibilities and to enable smarter energy demand response and more customer engagement than without, thus harnessing the demand side for the enhanced control and optimization of the power system.

Energy demand has been, and will be, much affected by the market diffusion of electric heat pumps, electric vehicles, private household and community microgeneration, penetration of small-scale wind power generation, and other developments, all having an impact on the load on the distribution networks. DSOs need to be empowered to be able to actively manage intermittent and 2-way power flows. The list of emerging new requirements for the grid/DSO is a long one, and timely action is paramount.

Some future directions envisaged for the development of the smart grid in Great Britain include the demonstration of medium voltage direct current technology to bring more controllable connections between sections of the power grid, which would raise the amount of renewable power generation that can be connected to the grid. It further comprises wider applications of advanced ICT in transmission grids, which involves fiber optic cables instead of copper hard wiring, which will enhance controllability, reduce environmental impact, and increase station safety.

Yet another element in the strategy is the national rollout of smart meters (for both electricity and gas) in all homes and most small businesses, involving the replacement of some 47 billion meters and costs of some £8.6 bn. Supplier benefits were estimated at more than £6 bn, of which avoided meter reading accounts for some £2.6 bn and reduced inquiries and customer overheads £1.13 bn (Jenkins et al. 2015). Consumer benefits were estimated at some £6.4 bn, composed of mainly energy cost savings (£4.2 bn) and load-shifting/TOU tariffs (£1.1 bn) (cost savings partly occurring upstream but or assumed to be passed on downstream to the consumers). For grid operators, smart metering is expected to provide benefits in terms of network planning, network operation, and demand management. Network planners can benefit from the analysis of load and voltage profiles obtained from smart meters (instead of standard/synthetic load profiles), better asset utilization on the distribution grid level (enabling, e.g., deferral of asset replacement), and providing a more accurate basis for predictions of future voltage and demand operating ranges in light of the expected rapid diffusion of heat pumps and electric vehicles. Network operators can benefit from better local outage detection and shorter system restoration times, also enabling them to communicate reinstatement of disconnected supplies using smart meter communication channels (greatly reducing reliance on customer calls both for local fault or outage detection). Finally, demand management will benefit greatly from smart meter installations in terms of enhancing demand response, e.g., by communicating dynamic pricing levels, or (incentive-driven) direct load control (see also the chapter “Demand Side Management”).

Obviously, such estimates of costs and benefits are subject to high uncertainty. Connor et al. (2018) systematically identified risks and uncertainty related to the UK smart grid deployment based on a detailed expert stakeholder analysis. They found that many of the risks identified arise from the increasing complexity of the system and of potential solutions that come along the evolutionary process of steadily increasing smartness. A major source of risk identified was continued inconsistency in the aims and objectives of different regulatory bodies (esp. Ofgem and DECC, now BEIS). As a remedy, there is a need for a virtuous co-evolution of the regulation (and the regulator) along with that of technology and market services (and thus also societal needs and lifestyles). Another important risk perceived as difficult to mitigate is that of ‘broken value chains’ (i.e., problems in ‘value-stacking’, i.e., if new technologies and services cannot be monetized because the value created is dispersed and difficult to aggregate and/or share cost-effectively among the stakeholders). Additional network costs, and the impact on consumer bills if passed on to them, were also seen as a significant risk. Finally, creating value from customer/consumer data was seen as a risk for acceptance of SG solutions on the consumer side. Overall, however, all these risks and uncertainties identified were not used as an argument to continue with the current solutions, as these will become less effective and more expensive as the sustainable energy transition progresses.

Figure 16 shows the shares of projects by technology among all tier-2 projects. It shows that ICT technologies played the most important role, as more and more sensors and remote terminal units are installed in the distribution grid for improving visibility and to enable grid control and automation. More than half of the projects were focusing on the active management of the power flows in the low- and medium-voltage grids, exploring concerns on voltage and thermal constraints. Fault-level management has so far seen limited attention. Automatic network reconfiguration can be realized by means of automatic switches and power electronics, and is considered a very effective means for demand balancing and voltage stabilization in smart grids.

Fig. 16
figure 16

Technical foci of the tier-2 smart grid projects realized in the UK (Jenkins et al. 2015, p. 418)

To date, the focus has been mainly on the distribution grids, including real-time information flows and interaction between suppliers and consumers based on improved ICT, active power flow management, and DSM. The national rollout of smart meters boosts the smart grid development, as it enables more efficient network planning and operations, and customer engagement through DR.

Review Question

  • What are the main potentials of smart grids to enhance economic efficiency?

  • Can you distinguish between the different flexibility options, their relative merits, and the competition between them?

  • Do you have a clear idea of types of actors, DER and concepts involved in a SG system, and more broadly, a sense of the economic implications of the ‘4Ds’?

  • Do you have a basic idea on local energy market design issues? Did you obtain a basic understanding of the investment needs under uncertainty?

  • Did you develop a sense for the many new fields of application for machine learning techniques arising with the realization of smart grids?