Local Energy Markets: From Concepts to Reality

Abstract

Local energy markets can create new societal value by offering a variety of energy services tailored to the local context and community desire. Based on real-world experience, this chapter describes how to enable active participation of a diverse set of local system stakeholders through a novel hybrid electricity market architecture. This market is the product of an agile co-design process carried out within Project LEO (Local Energy Oxfordshire), one of three large-scale energy community demonstrator initiatives in the United Kingdom. Firstly, we discuss the underlying concepts and desired services to be provided by the market. Next, we describe the agile approach taken to trial whole-system innovation in real-world contexts, describing the end-to-end procedures, data requirements and the digital platforms that enable local energy markets to function. Finally, we discuss some of the challenges faced and provide an outlook to the future of local energy markets.

Scot Wheeler, Filiberto Fele and Masaō Ashtine contributed equally to this work.

1 Introduction

As nations seek to rapidly decarbonise the energy sector to meet national decarbonisation targets, electricity networks are seeing a radical transformation in how they operate, particularly at the local level or ‘grid-edge’. Local Energy Markets (LEMs) that facilitate the transaction of power system services between network stakeholders are envisioned as a mechanism to encourage and coordinate active participation within a Smart Local Energy System (SLES) (Charbonnier et al. 2022; Council of European Energy Regulators 2017).

The widespread electrification of the heating and transport sectors, and the increase in decentralised generation technologies, is leading to a dramatic increase in the number of Distributed Energy Resources (DERs). Digitalisation and the growth of smart devices in homes and businesses allow traditionally inflexible primary energy demands to become controllable and therefore flexible.

Many local and global benefits can be achieved through the coordination of DERs. These include: revenue stacking for DERs, reduced bills, better utilisation of network (contracted) capacity, voltage and network constraints management, reduced energy transport, avoidance (or deferral) of infrastructure upgrades, and system balancing. Most importantly, coordination and cooperation of DERs can yield environmental gains by supporting the penetration of renewable generation, and facilitate social cohesion in the process (Abbas and Chowdhury 2021; Charbonnier et al. 2022; Eid et al. 2016; Pumphrey et al. 2020). Studies on the impact of flexibility and SLES deployment across the national GB system estimate between $\pounds$1.1bn/yr and $\pounds$5bn/yr, on the order of 5% of the total annualised system cost (Aunedi et al. 2022; Piclo et al. 2022).

Whilst many smart DER technologies and platforms already exist at high technology readiness level (TRL), whole system integration and demonstration within real-world markets remains limited. The latter require unique sets of stakeholders and communities willing and capable of working together within new and unknown market environments. LEMs require coordination of technology and people: this entails not just economic and technical interaction, but the fostering of legal, digital, data, regulatory and social relationships in the context of energy systems.

This chapter discusses the techno-social learning outputs gained from transforming LEM concepts into real-world implementation through the SLES demonstrator project, Project LEO (Local Energy Oxfordshire) henceforth referred to simply as LEO, based in Oxfordshire, United Kingdom (UK). While some of the activities within LEO are specific to the technical, social and regulatory energy landscape in the UK, many of the concepts are reflected in local energy transitions happening around the globe.

LEO is one of the most ambitious and holistic smart grid trials ever conducted in the UK. Part funded by the Industrial Strategy Challenge Fund’s ‘Prospering from the Energy Revolution’ (PFER) fund (UK Research and Innovation (UKRI) 2022), LEO seeks to understand how new LEMs and improved local engagement can unlock additional societal, financial and environmental value for households, businesses and communities provided by a smarter, more flexible, electricity system (Project LEO 2018). LEO brings together an exceptional set of system stakeholders from the energy industry, local government, community organisations, and academia. By mimicking the requirements of the future electricity system LEO is building an evidence base that will inform future market design.

LEO operates in parallel to the TRANSITION project, led by Scottish and Southern Electricity Networks (SSEN) and funded by the Government’s energy regulator in Great Britain, Ofgem (Scottish and Southern Energy Networks (SSEN) 2022). TRANSITION is designing and implementing trials that support the transition from the role of Distribution Network Operator (DNO) to that of Distribution System Operator (DSO). While the exact model of a DSO is still to be agreed by the industry, it reflects the move towards the ‘smart-grid’ which delivers additional data, monitoring and control systems at the local network level to enable bi-directional energy flow creating a more active and customer led flexible network while maintaining safety and reliability; this is in contrast to traditional DNO activities which largely focuses on infrastructure development and maintenance.

This chapter is organised into two main parts. Section 2 discusses core concepts of LEMs and how they relate to the market designed within Project LEO. Section 3 discusses how these LEM concepts were implemented in a real-world demonstrator project, the agile process developed to manage this, and a discussion around some of the key challenges faced. Finally, conclusions are drawn and a set of recommendations are made.

2 Concepts

LEMs are emerging over alternatives such as direct centralised control or network reinforcement as the favoured mechanism for the coordination of potentially millions of flexible DERs for better network utilisation and local socio-economic and environmental benefit (Council of European Energy Regulators 2017). This section describes some of the key concepts behind LEMs, particularly for the facilitation of local flexibility services within communities and with network operators. Firstly we describe the drivers behind LEMs and the challenges in enabling the ‘small and many’ to participate. Next we discuss the need for flexibility services, which are important and what opportunities exist to stack revenue streams through integrated markets. Then we introduce the concept of fair and equitable markets before finally discussing the data and digitalisation requirements that are needed to enable LEMs to operate successfully.

2.1 Local Energy Markets: The Main Drivers

Driven by cost reduction of small-scale renewable generation and energy storage, and the electrification of transport and heating/cooling demand impulsed by the net zero emissions goal, the electricity sector is becoming more decentralised and local in nature. As the aforementioned technologies populate the edge of the network, grid operators face growing concern with the reliability and resilience of the existing distribution infrastructure.

To ease the deployment of this new fleet of generation assets and loads, a consortium of European energy regulators recognised four mechanisms for enabling grid support from generation and/or demand flexibility in the distribution network (Council of European Energy Regulators 2017): rules based approach, network tariffs, connection agreements, and market based procurement. Within the latter, LEMs have been receiving particular attention as they allow for a competitive service provision from DERs and guarantee the necessary degree of decentralisation (Lezama et al. 2019; Schittekatte and Meeus 2020).

Flexibility can facilitate the penetration of low carbon generation, reduce the effective cost of electricity and its transportation, and accelerate the connection time for new demand and/or generation projects. Supported by the increasing desire and commitment from consumers and communities to cut energy expenditure and associated carbon emissions, a number of pilot projects are currently operating towards the incorporation of DERs in operational energy markets; besides Project LEO, see for example EcoGrid 2.0 (Denmark) (Heinrich et al. 2020), EnergyVille (Belgium),¹ and the various initiatives backed by the USEF foundation across Europe.² Important challenges transpire from these trials. In particular, limited local resources and expertise can prevent many potential DERs playing an active role in these markets. It is far from clear how to address the lack of detailed information on (let alone control of) each DER’s output. However, this detail is essential to distillate the value of their service, on which commercial agreements for the exchange of flexibility services need to be based.

To leverage the population of DERs in impulsing the transition to a smart and sustainable energy system, it is therefore important to steer the design of LEMs in the right direction in a timely manner. In the following section, we discuss in more detail the flexibility products that have emerged to support the aforementioned targets.

2.2 Flexibility—What Services are Important?

In a nutshell, flexibility services consist of (i) a temporary change in the demand or generation profile, or a storage charge/discharge operation, carried out upon request, or (ii) capacity trading, where grid users exchange their contracted import/export rights to temporarily exceed their power limits. Through LEMs, these become a commodity underpinning a more cost-efficient and sustainable network operation. Flexibility services are mainly utilised by:

• The Energy System Operator (ESO) in balancing the electricity system in real-time, e.g., frequency services to manage the imbalance between the level of national demand and the aggregate level of generation.

• The DSO in managing constraints and congestion in the distribution network. The need for this flexibility typically arises as part of the DNO tasks, due the fact that distribution constraints are not adequately taken into account in the existing wholesale and balancing markets (Kok et al. 2019; Schittekatte and Meeus 2020). These services can help prevent the development of a fault on the local network or assist in reinstating the network following a fault. In the long term, the DSO can also leverage flexibility services to avoid investment in new equipment, enable the connection of more renewables to connect to the network and thus support the decarbonisation of heat and transport.

• Two or more users establishing a bilateral contract, e.g., trading import and export capacity between sites to enable increased generation or demand in the local area, circumventing lack of infrastructural capacity that would delay (or increase the cost of) development.

• One or more users managing their own price risk behind the meter, e.g., reducing demand when electricity prices are high.

2.2.1 Flexibility Products

Notable products that are procured in LEMs by the ESO are balancing mechanism, dynamic containment and short term operating reserve, dedicated to settling imbalances; capacity market, for peak load management; Optional Downwards Flexibility Management, developed to cope with exceptionally low demand on the network (as experienced during the COVID-19 pandemic).

With regards to DSO-procured services, the industry has highlighted four relevant flexibility products to manage local congestion and network maintenance (Energy Networks Association 2020a):

• Sustain (season ahead, week ahead or day ahead notice): a change in demand or generation over a defined time period to prevent a network going beyond its firm capacity; while this service is mainly used to ease the stress on grid components in correspondence with predictable load peaks, it is also employed in the case of an excess of local generation, producing a too high reverse flow.

• Secure, whereby the DNO procures the ability to access a pre-agreed change in input or output based on network conditions; although this service can be procured with large advance, it is designed to respond to pre-fault conditions (e.g., to avoid a cascading failure); as such, the service provider must be capable of responding with short notice (${\sim }$4-h notice for planned maintenance).

• Dynamic (30-min notice), whereby the DNO procures the ability to access an agreed change in output following a network failure (post-fault); this service is designed to relieve the stress on the network while the system is being restored to normal operation.

• Restore: following a loss of supply, the DNO instructs a utility provider to either remain off supply, or to reconnect with lower demand, or to reconnect and supply generation to support increased and faster load restoration under depleted network conditions.

Peer-to-peer (P2P) services differ conceptually from the aforementioned ones in that they transcend the operational boundaries of DNOs into those of DSOs. In particular, while P2P services are typically defined as bilateral contracts between independent parties (network customers, albeit subject to the DSO’s approval), they do contribute to a virtual expansion of the local network capacity, and in the long term, to deferral of infrastructure maintenance and/or upgrade cost (Klyapovskiy et al. 2019; Spiliotis et al. 2016). Notable examples of P2P services are:

• Exceeding maximum import/export capacity: the maximum import (export) capacity is the upper limit on the total load (generation) from a given customer. These limits are established as part of contractual agreement with the network operator. One network customer (either generator or load) can temporarily restrict its power allowance and sell this nominal difference to another party so they can increase their power exchange by the same amount. Typically, it is required that the partners are connected at the same point of the grid (e.g., primary or secondary substation).

• Offsetting: This service consists in matching local increase in demand with local increase in generation at an agreed time (or a decrease in generation (load) with an increase in generation (load) elsewhere). This produces a ‘net zero’ change, cancelling out the impact on the network and allowing more energy to be generated and used.

Among the plethora of possible definitions—and variations thereof, tailored to specific system hierarchy—of flexibility products, there is still a debate over their standardisation. Indeed, a major argument in favour of it is the attainment of price transparency and a sufficient level of liquidity in the markets. Furthermore, a standard implementation of the requested services can potentially facilitate the coordination between ESO and DSOs, and among different DSOs that procure these services over the same platform (see, for example, the Piclo Flex market platform involving all DSOs operating in the UK (Stanley et al. 2019)).

However, product standardisation may conflict with the specific needs of DSOs, which may vary drastically from context to context. Moreover, a market based on a standardised definition of flexibility will make it more difficult to distinguish—and accordingly reward—the offers from the various providers. For example, the emissions associated with the delivery might not be considered among the nominal features of a standard flexibility service, therefore missing the opportunity of incentivising offers prioritising this factor (Schittekatte and Meeus 2020).

2.2.2 Energy Versus Power

A fundamental classification regards the way flexibility is quantified, where services are categorised as either baseline or capacity limitation ones (Heinrich et al. 2021).

In the first case, the flexibility provider is asked (e.g., by the DSO) to deviate from a typical power profile (the so called baseline) to achieve a desired modulation of the grid load. Therefore, baseline services are akin to energy trades in existing wholesale (and other ESO-procured) markets. However, important challenges hamper the quantification of a baseline: in fact, the typical population of customers in the distribution network does not consist of dispatchable assets, and as a consequence the definition of a baseline services has to rely necessarily on an estimated reference power profile.

In the second case, the flexibility service is directly defined as an import/export cap on the asset’s exchange of power with the grid. Clearly, the latter method can benefit from a much straightforward implementation, as the service definition relies on an absolute value (the capacity limit) versus a relative one (the baseline). Nonetheless, while in theory both concepts can yield equivalent outcomes, there is no clear-cut preference in the observed practice of LEMs so far, where the choice appears to be context-dependent. We will discuss this matter further in latter sections; for more details, we refer the reader to (Energy Networks Association (ENA) 2022; Rossetto 2018; Ziras et al. 2021).

We conclude with a perspective from a lower layer of the grid operation. The introduction of real-time metering has driven a transition from energy-oriented billing (e.g., based on energy generated or consumed in a half hour) to power-oriented (based on energy generated or consumed at intervals of minutes, or fractions of them). Settlement periods of the order of a few minutes—now technically possible—can transform how markets operate, as well as the landscape of exchanged products and services.

2.3 Enabling the Small and Many: Routes to Market

Delivering net zero carbon targets will require a transformation in the scale of active participation at the local level, using flexibility within the low voltage network from end users: the grid edge (Origami Energy, The Low Carbon Hub 2021). This will create opportunities to realise the currently dormant potential of millions of DERs in small and medium enterprises, public organisations and domestic premises to support the flexibility needs of the grid. These opportunities will, in turn, support the development of low carbon technologies (LCTs) and help the process of electrification of major infrastructures and the penetration of renewable generation. To enable the participation of DERs in flexibility markets and exploit their full potential in the delivery of the net zero targets, a number of challenges must be resolved. Among these are the attainment of societal participation and an attractive and fair commercial environment (Tushar et al. 2019). Underpinning both is the difficulty of estimating the true value of flexibility: in all markets currently in operation, the rewards to flexibility providers only partially reflect the benefits delivered to the whole system.

2.3.1 Technical Challenges

The technical challenges amount to the DER controllability (i.e., capability of regulating the power input/output and interfacing with market platform), and to the verification of the service delivery (by, e.g., dedicated physical metering or numerical data-based disaggregation methods).

The majority of households are expected to be equipped with smart appliances by the end of the next two decades, facilitating the manipulation and optimisation of the energy use. Moreover, manufacturers have embraced the Internet of Things (IoT) paradigm, whereby connectivity is embedded into appliances (Spanó et al. 2015). This shift in industry-standard solutions for automation aligns with the need of monitoring, controlling and coordinating the DERs energy use, and will underpin both the aggregated delivery of flexibility and its formal verification (Azizi et al. 2022). Nonetheless, to accommodate the number, geographical spread and heterogeneity of DERs, a distributed hierarchy will need to be in place, ranging from households to district, regional and national levels—thus spanning both the grid’s distribution and transmission levels. The coordination across layers can be supervised by the DSO in collaboration with the ESO; alternatively, this can be provided by a third-party coordinator or supplier of bundled services to the ESO/DSO in a secure and reliable way (Energy Networks Association 2018).

Another challenge is associated with minimum flexibility requirements, typically set in order to participate in the LEMs. While this varies by market, minimum levels considered in currently operating flexibility markets exclude the vast majority of potential providers from participating (unless they can combine their flexibility with others). Further specifications are typically set on response time (from receipt of a utilisation instruction) and on ramping time (once a utilisation instruction becomes active), minimum sustainable duration period for the flexibility service, and point of connection to the local grid.

2.3.2 Regulatory Challenges

These challenges relate to the regulations on the interoperability of small-scale DERs and the grid at distribution level. New standards can be imposed from governmental and industrial actions to attain attractive financial margins for each DER owner, by reducing the entry cost to flexibility markets and facilitating the participation to smaller scale DERs. In particular, communities should be enabled to provide their members a fair access to flexibility, and to interact with the network as a collective with low transaction costs. Universal standards also help ensure interoperability of hardware and software to maximise ease of integration.

In working in synergy with communities, local authorities will be key to ensuring that flexibility is fairly rewarded. It is important to create a framework that unlocks the contribution of the population of small-scale DERs towards a decarbonised and secure energy supply, where incentives are distributed in a way that encourages further investment in other LCTs. At the same time, the vital role that energy efficiency plays in delivering the net zero targets should be recognised and considered on parity with other LCTs. Facilitating upgrades on the existing housing stock has indeed a clear potential to contribute to the local and national energy system in terms of flexibility at the residential level (Jazaeri et al. 2020).

2.3.3 Commercial and Social Challenges

If reducing barriers to market entry and transaction costs will allow small-scale DERs direct market access, leveraging meaningful flexibility services from these will require a critical mass that can only be achieved by aggregation. Energy communities can support social aggregators who provide collective benefit and mutual sharing of the rewards, while ensuring a reliable and transparent delivery of flexibility. Importantly, communities can alleviate the contractual responsibility of individual owners of DERs, increasing the trust of system operators and promoting the creation of a broader set of flexibility services.

Standard solutions like direct load control (DLC) can be used to leverage flexible assets, by explicitly incorporating their scheduling in power flow calculations, whereby decisions about flexibility are driven by grid state (Correa-Florez et al. 2020). This approach relies on the assumption that the DSO and the DER come to an agreement. DLC agreements are commonly reflected in more mature flexibility and aggregation markets involving large industrial and commercial customers; however, they can often exclude real-world scenarios in the public and residential building sectors (Cardoso et al. 2020). Indeed, where public buildings are concerned, key aspects around demand-side response (DSR) for flexibility services raise very different constraints compared to those imposed to providers from the industrial and commercial sectors (Ofgem 2016). These issues include separate asset ownership (key stakeholders may not be decision-makers in an asset’s operation), misaligned interests in asset/building management, and privacy constraints that conflict with commercially-sensitive data.

2.4 Integrated Markets

This section considers the interaction of different flexibility services and highlights where revenues can be stacked across different time periods. In current flexibility markets, the value for flexibility paid to the DER does not typically reflect the full benefit delivered to the grid. Analyses carried out in the UK context have revealed that the value offered in some markets is insufficient to attract investment in new capacity as well as flexibility at the levels required, which will put the security of power systems under pressure in the medium term (Scottish and Southern Electricity Networks 2019).

2.4.1 Revenue Streams and Stacked Service Provision

As discussed in Sect. 2.2, the main income streams for flexibility services are made available through ESO- and DSO-procured services, and P2P-type bilateral agreements. Further revenue streams—generally reserved to large-scale DERs—are related with transmission and distribution charge minimisation (see, e.g., National Grid ESO 2022; Scottish and Southern Electricity Networks 2022a), and direct wholesale trading (possibly through third party aggregators).

In general, however, the estimation of the possible revenue for a given DER is not trivial. This is mainly due to the fact that the reward assigned to a flexibility service can be inherently volatile (this is the case, for example, for the ESO’s wholesale trading and balancing mechanism services). Indeed, many externalities—including market liquidity, local grid constraints, weather perturbation on flexibility levels—can affect the LEM’s settlement across different layers (Tao et al. 2020; Vespermann et al. 2021). Predicting a nominal price for these services is challenging as the bidding strategy is often determined at the day ahead stage (if not with larger advance, as common with many DSO-procured services, see Sect. 4). In the future, datasets relative to historical flexibility market operations can help in achieving a more precise prediction of the performance of a given DER class (Fele and Margellos 2021). However, the amount and quality of currently available data is insufficient to ensure adequate financial security for the DER manager; this is especially true for specific newly introduced flexibility services in the context of P2P trading. It is worth mentioning that this in turn has implications on the side of the flexibility procurer—hence on the general network performance and reliability—, as it may lack too the necessary information to carry out an optimal selection of providers for the required services.

A number of factors should be considered when determining the business case for determining the viability and enabling DERs to deliver flexibility services:

• the priority assigned to the provision of flexibility services versus the primary purpose of the DER;

• the ability of the DER to deliver stacked flexibility services to increase income;

• the operational impact and costs of delivering these services;

• the costs of enabling the DER to deliver flexibility services: this can include equipping the DER with remote monitoring and actuation technology.

The technical ability of a DER to deliver multiple flexibility services at the same time or during adjacent periods, allows the DER owner to bid on a stack of products (or combine them with wholesale trading) and realise a higher price (Energy Networks Association 2020b). The ability of flexibility providers to differentiate their bids depending on whether flexibility is traded locally or centrally is key to the development of sufficiently liquid markets. Indeed, selling at specific grid locations may be risky, since flexibility is typically needed locally only a few hundred hours a year.

For this reason, the availability of a common market platform can bring significant advantages, enabling the maximisation of the scope of flexibility services outside the local network boundaries (Lopes 2021). Establishing an efficient coordination between ESO and DSOs is critical to ensure ESO- and DSO-procured services do not have opposite impacts (e.g., a ESO service creates a constraint locally that triggers procurement of a DSO-service), and finally to enhance the value of flexibility across all layers of the grid (ENTSO-E et al. 2019; Le Cadre et al. 2019). Such a coordination is already possible (at least up to a certain degree) with some of the currently operating flexibility platforms (Schittekatte and Meeus 2020; Stanley et al. 2019).

2.4.2 Challenges Due to Uncertainty on Delivery and Coalitions of DERs

The flexibility output of some classes of DERs may be subject to uncertainty (e.g., some DERs can be highly sensitive to weather). This can have a significant impact on the financial performance of these DERs, unless the markets can offer specific mechanisms to deal with this challenge; see, e.g., Vespermann et al. (2021), Laur et al. (2020). In practice, the value flexibility is paid out on the market decreases as a conservative amount is commonly procured to offset the uncertainty of delivery. Similarly, on the providers’ side of the market, aggregators that utilise pools of DERs with low or unreliable flexibility have to factor this risk and contract with more flexibility than needed to avoid contractual penalties. Ultimately, such compensation on both sides can significantly decrease the value of flexibility and could reduce the attractiveness of such markets for new participants.

Establishing cooperative clusters of DERs can help address these issues (Fele et al. 2017; Han et al. 2021; Morstyn et al. 2018). A group of DERs can be used collectively to deliver a flexibility service where some could provide the initial speed of response and initial period of delivery (e.g., a battery which is energy limited and cannot deliver a long duration service) whilst others could either extend the duration or the capacity associated with the service (e.g., an industrial demand process which could not provide the service alone due to significant response delay, a plugged-in EV fleet could be used to fill in gaps in the delivery profile).

2.5 Fair and Equitable Markets

If mass participation within LEMs is a requirement, organisations, communities, and individuals need to engage with LEMs and become active participants (Tushar et al. 2019). Any system considered fair and ethical is more likely to be met with social and political approval and therefore succeed (Fleurbaey et al. 2014). System transitions inevitably lead to winners and losers based on an individual’s attributes and ability to participate in the new system (Savelli and Morstyn 2021).

The notion of a just transition concerns the overcoming risks of unfairness and leaving the vulnerable behind. It has been acknowledged by many key stakeholders at the centre of the energy transition that energy justice must be applied to the planning, implementation and assessment of any aspect of the transition (Carley and Konisky 2020). Energy justice has three core concepts: Distributional justice, ensuring any benefits and burdens are distributed across the population without some receiving an inordinate share or being denied access to participation; Procedural justice, ensuring decision-making process is fair, equitable and inclusive to all who may wish to participate; Recognition justice, an understanding of historic and ongoing inequalities.

How does this apply to LEMs and smart DERs at the grid edge? The Centre for Sustainable Energy produced a report on this, funded by some UK based DNOs. It concluded that for a market-led approach, interventions from regulators and policy makers will be required to secure inclusive participation and a ‘safety-net’ for anyone left behind. Whilst market commissioners are unlikely to be able to ensure fairly distributed participation alone, there are steps centred around transparency that will assist in delivering market equity (Centre for Sustainable Energy 2022).

Without starting entirely from a clean slate, energy networks and network users will have inherent non-uniformities, embedded both spatially and temporally, which determine the types of network services that are required to decarbonise in the most cost-effective way. Likewise, participants and DERs will differ in their capability to participate and compete to deliver energy and flexibility services within a market. For example, DERs with a high degree of uncertainty, perhaps due to being heavily weather dependent, or inexperienced market actors, are likely to find it much harder to produce firm capacity forecasts ahead of service periods (particularly season/month ahead) compared to more conventional storage technologies. This could translate to such DERs being underpaid or undervalued in their service offering. Therefore, the design and operation of competitive flexibility markets needs careful consideration and testing to ensure equitable principles are delivered.

It is necessary to make a clear distinction between energy equity and equality; it is not about offering the same thing, but what is right for the participants’ own circumstances to take part. To be an ethical system, it needs to engage participants through honest and transparent communication with the market rules, and clear and understandable benefits of participating. LEO has developed an ethical framework to support the delivery of SLES trials which includes the following principles: (1) Collaborative design, (2) Inclusive offering, (3) Fair distribution of benefits and costs, (4) Minimise risk, (5) Informed consent, (6) Respect and (7) Data fairness (Huggins 2022).

Translating these concepts to a fair and equitable implementation of a LEM has three main components. Firstly, through collaboration itself, the market should be designed with input from multiple stakeholders. This includes energy industry actors, academia, local authorities and representation of community energy groups. The LEM architecture can be adjusted through multiple trial period iterations which allows feedback from participants to be taken on board, and multiple market mechanisms and value propositions to be tested.

Secondly, the LEM should be an open marketplace that can offer multiple access opportunities to local network flexibility, peer-to-peer capacity and energy, and national balancing markets, tailored to specific market actor types or DERs (CGI et al. 2022; Schittekatte and Meeus 2020). Central to this is some form of Neutral Market Facilitator (NMF) platform through which coordination with the DSO is achieved. Crucially, the NMF integrates with third party market platforms, aggregators and other independent intermediaries so bespoke services and value stacking across multiple markets can be offered to meet participants needs (ENTSO-E et al. 2019); this approach is currently followed by Project LEO.

Finally, multiple routes to market should be tested for the DNO-procured flexibility services. These range according to the degree of competition—informed both by market liquidity but also participants’ appetite for risk—the length of the ahead-of-time procurement period and the ability to weight bids across availability ($\pounds$/MW/h paid whether called upon or not) and utilisation ($\pounds$/MWh paid for the energy delivered). For competitive and liquid market nodes, this includes a fully competitive auction or fixed price contract (fixed price contracts are based slightly below the average competitive market settlement price to reflect extra risk being taken by the DNO) while for non-competitive market nodes, price ceilings are in place. Market stimulation packages (MSPs) offer an alternative route to encourage new DERs and market actors into the market that might require some capital support to become ‘flex-ready’ (Scottish and Southern Electricity Networks 2022b). Initially, services will be procured in season ahead, week ahead and day ahead auctions but some studies believe continuous market clearing might be more suitable for early LEMs with low levels of liquidity (Kok et al. 2019; Prat et al. 2021).

While these market mechanisms are designed to reduce barriers for a range of participant types, a balance must be struck between multiple offerings versus both the market complexity (that may discourage smaller, non-traditional market participants) and cost of implementation.

2.6 Social Aggregation: The Community Side

One of the pillars supporting the long term sustainability of the future SLES will be the communities of active participants from the grid edge. To ensure social desirability, it is therefore important to align business plans for LEMs with the sustainable development objectives of local communities (Low Carbon Hub 2022; Tushar et al. 2019).

In particular, LEMs should foster the development of new products and services that can enable a smart and sustainable operation of the grid edge, for example, maximising the connection of embedded renewable energy and incentivising the electrification of heat and transport as well as efficiency improvement interventions. Functional LEMs are expected to deliver positive returns on the individuals and organisations that are the fabric of energy communities (Savelli and Morstyn 2021). Through the mediation of social aggregators, these can achieve the democratisation of mid-scale DERs ownership (e.g., solar parks, local hydro power plants) and access ad-hoc discounts and feed-in tariffs for community members (Low Carbon Hub and Origami Energy 2021).

Importantly, the social aggregator can provide small-scale DERs with low levels of flexibility a viable access route to LEMs. It can avoid owners needing to understand the market rules, and release them from the burden of participation. The social aggregator can provide market access on a not-for-profit basis using agreed shared-risk principles.

It is worth pointing out that this role requires a certain set of skills to interface with rapidly evolving LEMs. While this might currently lie out of the scope of most currently operating citizen communities, some larger co-operatives are taking action to reach this goal; an example of the latter is the Low Carbon Hub in Oxfordshire, UK, with the creation of the technical (behind the meter, P2P) and commercial (interfacing with external LEMs) aggregation platform People’s Power Station (Low Carbon Hub 2022).

2.7 Data and Digitalisation

Energy markets and the mere provision of energy services are inextricably tied to the data streams that enable them. Transmission services have always utilised data for grid balancing services, peak management, short-/long-term operating reserves, and for financial settlement within very liquid and competitive markets. However, the role of data at the grid edge is gradually morphing from one of network stability and monitoring, to unlocking greater levels of flexibility as least-regret investments become more attractive in the short-term. Smart systems, which buffer these short-term needs, can only be defined as such when the proper data layers and communication streams are in place for the provision of services that transcend high-level network operations. Data underpin the ability of LEMs and their agents to create local ecosystems for the mutual benefit of all stakeholders involved. The focus of this section will be on the rapidly evolving domain of distribution services, highlighting particular guidance for data management within LEMs and flexibility services.

Net-zero objectives at a national level are futile without structured digitisation at local scales that open networks to increased participation, services, and liquidity. Data innovation and investments play key connections across national-local boundaries and increased data ingestion will accelerate the transition of DNOs to DSOs, in particular, their ability to model and forecast network operations. This has been long recognised by agencies such as ElectraLink which stated that 40% of total renewable electricity generation falls under the regional DNOs, at the grid edge, rather than at the level of the transmission grid—highlighting the importance of increased data transparency at the distribution level (ElectraLink 2016). Flexibility has been a fixture of the energy landscape across the world during the last decade (Lopes 2021), but where local-scale provision from a diverse array of assets is concerned, the grid edge requires significant levels of data investment and governance. Increased connectivity and flexibility demand a level of dynamism whereby assets no longer act in silos, but the network aims for whole system optimisation versus data being used simply for monitoring and network stability.

There is little question on the importance of increased data visibility within networks and the added benefits to LEMs. The grid-edge has already moved beyond any doubt in this regard with homes, generation assets, and transport all becoming much ‘smarter’ in their collective operations and function (Vigurs et al. 2022). The question then remains around the barriers to connecting rapidly evolving assets within a historically inert grid to accelerate local energy transitions.

Data transparency across the grid however allows networks to better forecast local demand, and subsequently, provide more symbiotic advancements of energy provision.

There are three main bottlenecks which we posit are essential to unlocking the value of data for LEMs. These impingement points are centered around access, management, and validation as described below:

• Access: This layer is at the core of data value for network operations within LEMs. Simply put, if data cannot be accessed at levels that match the need of local network services, there will be little to no innovation at this scale. Asset owners and operators need to consider increased data monitoring and storage at the grid edge from where many other services can flow. It is not only the fundamental access to data that is needed, but the access (through both hardware and software) to real-time data streams with appropriate Application Programming Interfaces (APIs) for increased connectivity. Electric vehicle (EV) and vehicle-to-grid (V2G) services for instance will require instantaneous access to network data for decision making, scheduling, and operations. Access cannot be limited to just end-users, but must enable other smart systems that require interfacing with the network. Furthermore, access to long-term data is proving increasingly more important for forecasting and validation. DSR services are another example where real-time communications with assets are required. These services, historically functioning for ESO-procured balancing services in a number of markets in the UK, Europe and the US, must adapt for flexibility at the local level as long-term data are needed to validate services with appropriate levels of forecasting (Rossetto 2018).

• Management: Data need to be managed, not only through qualified/automated personnel/systems, but with increasingly standardised methods of privacy protection, aggregation, storage, and metadata. This particular data core is complex across a multi-stakeholder approach to LEMs as each agent within the overall system is aiming to optimise their own data and management needs. However, a critical point of open and standardised data protocols is key to unlocking many data barriers. In the UK, there has been considerable lack of a unified effort amongst DNOs where data management is concerned and standardisation is lacking (Energy Systems Catapult 2021). Many reports by the Energy Systems Catapult (Catapult 2021; Energy Systems Catapult 2021) and EnergyREV (Chitchyan 2021; Maidment et al. 2020; Morris and McArthur 2021; Verba et al. 2022; Vigurs et al. 2022) have touched on these issues to a large degree and it is clear that SLES need to become more open and transparent with data.

• Validation: Finally, with the inner two core data layers in place, the right systems are available for the effective validation of local services on the network. Data need to be open (where appropriate for network innovation as seen through projects like OpenLV³ and Western Power Distribution’s (WPD)⁴ stewardship in data access and portals), transparent with clear provenance (aggregation, pre-processing treatment, missing data and post-processing cleaning etc.), temporally sufficient with appropriate resolutions to invoke validation at differing scales (high-resolution data are not always needed depending on the application for instance), and of substantial quality to ensure solid footholds of analysis where validation is concerned.

Fig. 1

Three main data cores proposed for supporting flexibility services and local energy markets

From Fig. 1 we can see the main aspects of data access, management, and validation that are needed to remove barriers for LEMs. These high-level categories are certainly not the boundaries of data concerns within a SLES but highlight key areas of focus. Where access is concerned, the grid edge needs to be ‘armed’ (a term used to describe the process of readying assets or smart systems to participate in proposed services) to allow for effective bidirectional data streams. Data must shift from access with function of monitoring and case-specific assessment to access for real-time services to enable flexibility and dynamic responses to local markets.Arming of assets retrospectively will demand a high level of resources and DNOs and aggregators must play an increasing role in facilitating this to ensure a level playing field for smaller agents (Scottish and Southern Electricity Networks 2022b).

For management, system-wide standardisation at the utility-scale is needed for data management best practices to filter down at the local levels (Energy Systems Catapult 2021); one can also argue that top-level standardisation needs to be guided by local learnings. Translating these practices to grid edge can be guided by the 15 FAIR (Findable, Accessible, Interoperable, Reusable) data principles (Wilkinson et al. 2016) which are seen as a cross-disciplinary approach to data management. For local flexibility, data and their management systems will play a large role in the automation and scheduling of services (reducing marginal costs), with qualified personnel needed before full system maturation. Management also brings many questions around data custodians and proprietors, particularly for privacy and personal data. As SLESs are effectively ‘crowdsourcing’ energy data, there needs to be management around the protection of sensitive information.

Validation will require aspects of the data cores listed above but with a further level of specifications to ensure that services and markets benefit from trusted analysis. This data core does not have clearly defined roadblocks as those seen in data access, and issues centre more around the application of data processes in network validation. Thus, moving from access to validation, we see a shift in so-called blockers to more guiding measures for system enabling. For instance, baselining, prominent in the literature where domestic DSR application is concerned (Rossetto 2018; Ziras et al. 2021), presents a good use-case to highlight common data issues in validating network services. Where local systems are at play, baselining is an important step in grounding the day-to-day operations of a DER in order to confidently quantify benefits provided to the network during service provision (even before, to schedule said services). Commonly overlooked specifications of data such as the percentage of missing/corrupted values and the overall dataset length (for timeseries data) are often preliminary stumbling blocks for validating flexibility services as statistical and technical limitations are imposed. These issues fare better in low-capacity trading, but when capacities in orders of magnitude far greater are up for financial settlement, data quality cannot be trivialised. Validation is also key to ensure the fair scheduling and settlement of services, limiting data manipulation for commercial gain, and ensuring that reliable assets are rewarded by DSOs in a SLES.

Beyond the main data cores presented here, an increasingly digitised energy system must also have the necessary frameworks to ensure overall system resilience and security. SLES data transfers require communication streams that are not overly-linear in design as to avoid ‘weakest-link’ points of failure that can disrupt entire services, or in worst-case scenarios, expose the network and users to malicious attacks. Users, asset owners, aggregators, and DSOs all need confidence in the system when many complex business models inherently rely on their functioning. Moving to partially/fully digital systems shifts risk to one where many actors can be disrupted. Data resilience also means that in very liquidated flexibility markets, DSOs have a clear understanding on reliability at the grid edge, leading to better scheduling and frequently successful flexibility services based on critical market reliance indices.

Data need innovative yet standardised management with resilience weaved into all planning. Data, however, must not only consider function with a SLES, but wider system purpose that mutually benefits relevant stakeholders. In the road to robust data structures in SLESs and LEMs, data must also enable fair participation among energy actors to ensure that competitive advantages operate within equitable ecosystems. Though beyond the scope of this chapter, data communications must themselves be held up to net-zero targets where the embodied energy of increased data networks and traffic are assessed to truly achieve ‘smart’ design.

3 Real-World Experience: Translating Concepts Into Reality

For innovative change to take hold within a system, quickly and at scale, stakeholders and participants need confidence and engagement in the new concepts being proposed. Demonstrator projects test new concepts within real-world environments (or as near as possible to real-world while maintaining the safety and security of customers) to build an evidence base of what works well and uncover the barriers that may restrict acceptance.

In this section, we discuss the trial approach developed for asset participation in LEO market trials, alongside some of the insights (both successes and failures) that have arisen in the first three years of the four-year project. The first phase of trials, called the Minimum Viable System (MVS) trials, focused on initial market design and single asset activation, and ran approximately from month 6 to month 18. The second phase of trials, referred to as the MVS Programme and Smoke Tests, added more structure and complexity to asset trials and began market integration tests, running approximately from months 18 to 32. Finally, the full market trials, which are testing whole system operation across three trial periods, runs through to the end of the project at 48 months. At time of writing (38 months since the start of the project), LEO has completed the first of three full market trial periods.

Project LEO is far more than just a LEM demonstrator project, with a great deal of community engagement, policy and regulatory activity that is beyond the scope of this chapter. Readers are encouraged to visit the LEO website⁵ for continuous updates on activity across the whole project.

3.1 Learn by Doing: The Agile Approach

LEMs require the local interaction and operation of multiple stakeholders, energy resources, digital platforms, and business models, in new ways not previously demonstrated or necessarily designed for. Due to the nature of a fast-moving energy transition happening in an ever-increasing digital space, we took inspiration from agile approaches (Ghezzi and Cavallo 2020), particularly that of Lean Startup—a concept popular with entrepreneurs, allowing for agile innovation of business models through fast feedback loops that have led to ‘build-measure-learn’ approaches to growing startups (Ries 2011).

At the heart of early LEO trials was MVS, a concept akin to that of the Minimum Viable Product within lean theory; the difference being it is applied to whole-system integration rather than a single component. An MVS should represent the minimum set of participants and processes that are required in order to test a new process or asset use case. In doing so, new value can be identified and confirmed at a small, quick scale, before significant investment in time, money and user relations are committed.

The full Lean Ecosystem Transition approach used within LEO is summarised in Fig. 2 and described as follows: The societal need is the technical delivery of flexibility as per the proposed DSO flexibility market (versus current DNO operations). This need inspires a hypothesis, or Theory of Change, as to how best to deliver flexibility; this process could be a modification to an already established procedure or an entirely new one. Hypotheses are translated into trial objectives which lead to MVS trials being run. Through the collection of data during a trial, processes and Key Performance Indices (KPIs) are measured and analysed to enable new learnings to be generated. These can lead to a better understanding of the requirement—which in the extreme case could indicate a false need and therefore the opportunity to stop a particular avenue of exploration—or an adaption of the hypothesis proposed, both of which inform future iterations (Ashtine et al. 2021).

Fig. 2

The lean ecosystem transition approach developed and utilised in project LEO

This agile approach was extremely effective in the early stages of the project. It helped break-down the final goal of a very complex (and at that stage mostly unknown) technical system and market process, into more manageable problems and components. A series of MVS trials were run on LEO partners’ DERs of varying types (including battery storage, demand side response, micro-hydro generation and vehicle-to-grid). The majority of these DERs had not previously been used for flexibility services or were operated by non-traditional market actors (with little to no experience of flexibility or wider energy market participation). These trials started to identify the capabilities (or lack thereof) of the DERs, digital platforms for monitoring and communication, and market participants; this was crucial to direct the focus of development where it was most required for the market to become operational. The MVSs also provided the use case to begin developing the full end-to-end process for market operation. A set of reports summarising outputs of the early MVS trials are available through the Project LEO website (Origami Energy 2022; Wheeler 2022a, 2022b; Wheeler and Ashtine 2022a, 2022b; Wheeler et al. 2022).

As the project progressed, several challenges arose with an entirely agile project management approach:

• Firstly, more effort than was initially envisaged was required to enable existing DERs to deliver services in an easily controllable way, and for the project consortium to understand the flexibility services. See Sect. 3.3 for further discussion on this.

• Secondly, some core digital platforms such as the NMF were delayed. While manual proxy workarounds allowed other parts of the end-to-end process to be tested within the wider market context, progress was restricted for other components of the market, particularly external platform integration.

• Thirdly, limited personnel resource and the challenge of assigning responsibility across multi-organisational teams meant the iterative development of key inter-linkages between market components and organisations, that are critical to a whole-system approach to LEMs, was hampered. On occasion where partners did take a leading role in developing shared market processes, significant progress could be made in a short period. An example of this is the Commercial MVS run by Origami Energy which tested DER and participant preferences for different time-ahead auctions, the ability to baseline and the settlement processes (Origami Energy 2022).

• Finally, to be compatible with the external audit of deliverables and project monitoring as required by the funding body, time-based milestones were linked strongly with development of high-cost physical DERs. This removed emphasis from the agile development of the digital systems and market mechanisms. However, overall the slightly modified approach described below, led to more frequent trials and an enriched coordinated approach.

Table 1 Key stages within Project LEO’s MVS Programme to support enabling new assets and service providers technical participation in the LEM

What became known within the project as the ‘MVS Programme’, applies a more classic Waterfall Project Management structure to the MVS trials whereby particular steps must be completed (or at least started) as per a pre-planned timeframe. The MVS Programme was coordinated by Origami Energy and the University of Oxford. The steps themselves describe the objective of the trial and therefore simplify trial design and lead to similar development work across all assets (Wheeler 2022a). The steps broadly increase in service complexity and market integration, but do not strictly need to be executed in sequence; these are described in Table 1. For an experienced asset and market participant, multiple if not all could be achieved in one trial. The result was a much higher frequency of trials being run across all assets with a logical progression of asset capability. However, the programme became very asset centric with less attention on whole system integration or service design within the second year of the project. Following the introduction of the NMF in year three, full market trials are now focusing on whole-system operation of the LEM. The MVS Programme proved itself as a useful approach for enabling new DERs and supporting inexperienced market participants and remains a tool within the project for on-boarding within the market.

Fig. 3

Main steps in the DSO procured and DSO enabled market End-to-End (E2E) process from asset registration to verification and settlement; the spacing of nodes are for illustration only and do not reflect the relative timescale of events

3.2 Market Process

For a LEM to function successfully, the full end-to-end process and market rules for delivering market services must be well established and understood by all market participants. This extends beyond just the technical delivery of services and includes registration, legal agreements, market operation (including auctions), verification and settlement. Figure 3 shows a simplification of the combined DSO procured and DSO enabled full end-to-end process developed for the LEM being tested within LEO.

Relative timings for each step may differ depending on the particular market service being arranged. Registration of the service provider and asset with the DSO and market platforms needs to happen well in advance, between 2 and 6 months before participation. It requires the signing of the Flexibility Services Agreement (FSA), P2P term sheet and platform terms. Identify Need (P2P and DSO) can happen season ahead up to the week or day before the service window depending on service and must allow time for the DSO to run Power Flow Analysis (PSA) during the Trial Pre-approval stage to ensure no issues may arise on the network.

Requests for services are published on the market platforms alongside potential stimuli packages to encourage or support greater market participation in the Publish Stimuli and Requests step. Service providers are then able to Respond to Request which—depending on the service type, market liquidity and market competition—may proceed via an auction mechanism or fixed price at the set time-ahead period; currently only season ahead, week ahead or day ahead are being tested. Next there is the Assess and Qualify stage where the requester of the service (either DSO or peer) assesses offers before selecting services based on a particular criteria. In the early trials this is simply based on price, but other criteria such as historic reliability and carbon impact will be explored. The provider is notified of selection during the Select and Notify step and should be available to deliver the service if instructed.

Following a final Approval stage from the DSO and identification of actual flexibility needs, the service provider is Scheduled and Instructed to deliver the service. During the service window (step Deliver and Monitor), the asset and network are monitored; the DSO has the option to cancel the service delivery at any point if a network issue arises. Finally, the Verify and Settle step happens up to 1 month after the service: DER monitoring data must be submitted and compared to a baseline to verify the service was delivered, invoices are raised and the requester of the service pays the service provider.

3.3 Market Readiness

A common theme throughout the first three years of the project has been the higher than expected barriers to enable DERs and DER owners to participate in the market. While some of the challenges have been exasperated by the Covid-19 pandemic which has restricted site visits and forced organisations to operate in a more constrained way, we believe the insights gained will be important in the successful implementation of many future LEMs.

The vast majority of DERs that will participate in the future LEM are yet to be installed and, as such, the opportunity exists to ensure the capability is present at installation stage. However, for a LEM to be successful, it also needs to support existing passive DERs that have little or no experience of flexibility or active participation in an energy system (e.g., a building that can offer DSR); this support can be extended beyond physical energy assets to potential market participants and communities. This has led to the following definition of Non-traditional DERs used within the project:

Non-traditional DERs: Small but numerous assets of any type put forward for market participation by an organisation where energy (including flexibility services) is not already a component of their business activity. It is defined not just by technology but also by the combination of a market actor’s core business and previous experience. It should not be considered a clear binary label, rather a scale described by the capability of the asset-actor combination.

During the first period of the full market trials, two technology types (V2G and fixed lithium-ion battery storage) participated in 69 Sustain–Peak Management (SPM) events scheduled between 15:00 and 19:00, with a total of 540 kWh dispatched. These were operated by two service providers, both of whom had prior experience of the energy sector but only one had prior experience of flexibility markets. Meanwhile, around 30 other organisations were considering future participation or engaged in the early stages of market registration and asset activation process. Some of the barriers highlighted during this time that had greater impact for non-traditional participants are discussed below. Boxed content provides some examples from ongoing work within LEO to support DER participation in and understanding of the market.

DER potential: The first stage towards developing a case for flexibility provision is to establish the amount of capacity available for a single DER or DER portfolio. Without this understanding, service providers cannot gain access to market platforms or MSPs. Doing this accurately can be a complex, time-consuming process requiring expertise on energy generation or use, building control systems and machinery operation (in the case of DSR), historic data, and dynamic models of DER performance. For non-traditional DERs such as commercial/domestic DSR, those with a high degree of variability (e.g., due to weather), or public institutions that may operate over medium to large estates (tens or hundreds of potential DERs), this can be challenging without the knowledge or expertise in-house or the resource to procure external support.

Building Thermal Model

To estimate the potential flexibility provided by shedding the load of the chiller within a local authority’s library building over pre-specified time intervals (typically one hour in the afternoon), we trained a thermal dynamics model from dedicated tests conducted during the MVS phase of trials, an example of which is shown in Fig. 4. The thermal dynamics model takes as input the external temperature forecast for the day in which the flexibility delivery is requested, and outputs the estimated zone’s temperature trajectory and HVAC system energy expenditure.

Fig. 4

Flexibility event (chiller load shedding) on Sep 6 2021, 3.30–4.30 p.m. The top plot shows the metered active load of the chiller. The bottom plot shows the (averaged) temperatures relative to the four main library zones; the dashed trace indicates the outside temperature. Notice the slight rebound effect at the end of the service interval, where the HVAC load increases in order to restore the temperatures to within the desired setpoint as soon as possible

Cost of enablement: Once potential flexibility has been identified, there are likely to be costs related to activating a DER, the level of which depends on the DER and operator’s starting position, and requirements imposed by the market (e.g., metering and levels of automation). Our LEO experience with DSR in public institutions such as local authorities and universities has highlighted HVAC hardware and building management systems (BMS) software tend to be outdated or poorly implemented. Costs of enablement are likely in the range of $\pounds$1000–$\pounds$10,000 which can be hard to justify with limited understanding of the value that can be extracted from LEMs at this early stage. This is likely to be more prohibitive to non-traditional DERs and operators. The MSPs offered by SSEN are designed to help with this barrier by offering upfront capital which is paid back through guaranteed market utilisation over a set period. Initially, the MSP have not been popular, with potential market participants citing perceived complexity and the value of support being too low versus the high cost to activate a DER as reasons for not using them. Work is ongoing to better tailor these to meet participant needs.

Legal arrangements: Market participants, with and without prior experience of energy and flexibility markets, have found it challenging to sign some of the legal contracts required to participate in the market. In particular, for more traditional market participants and aggregators, minor disagreements over specific clauses are expected to be solved in the long term with standardised LEM contracts developed by recognised industry and regulatory bodies. For non-traditional participants with little to no experience as active participants in the energy sector, in-house legal teams may not have the capability to make informed decisions, requiring external legal advice and therefore a higher cost of participation.

Availability and Delivery: The more traditional DERs had the best availability (declaring themselves unavailable the least) and best record of successful delivery irrespective of the operator’s market experience. Uncertainty associated with an asset, perhaps because of weather dependence or responsibility to deliverer another primary service (away from flexibility—e.g., personal transport in the case of V2G) plays an important role here. Competitive bids must be made well in advance (weeks to months) with reduced remuneration or penalties if actual delivery is lower, risking loss of revenue on already marginal value. Accurate forecasting of load to allow for informed market bidding can be costly and complex to implement. It is unlikely that small scale DER and public institutions will have this capability, if not already part of their primary business. To reduce the risk to the DSO, future markets must be designed to accommodate asset uncertainty, ensuring these are not unfairly disadvantaged without the resultant LEM having large overcapacity at high cost or poor service security.

Forecasting

Following a data-driven approach, month ahead forecasts were synthesised for the local authority library DSR case study. Openly available historic temperatures from 2000 to 2019 (Pfenninger and Staffell 2016; Staffell and Pfenninger 2016) were filtered to within a range of ${\pm }5$ days around the corresponding day of delivery in past years. The interval was chosen to account for possible year-to-year variations, while also achieving a more statistically significant sample set. Figure 5 shows the empirical probability distribution of the historical temperature values measured from 2000 to 2019 at 4 p.m. on three days in June (these corresponded to the days selected for flexibility delivery in the first three commercial market trials within Project LEO).

Fig. 5

Empirical probability distribution of temperature in Oxford, for three selected days in June, relative to the 20-year interval 2000–2019

Based on the obtained temperature statistics, a forecast of flexibility delivery (in kW per hour) was derived by using the selected historical temperature data as input to the thermal dynamics model mentioned above. As a result, the empirical probability distribution of the provided flexibility could be synthesised, as shown in Fig. 6. This forecast can then be used to inform the capacity offered in response to any flexibility request, using the cumulative distribution as measure of risk of underdelivery.

Fig. 6

Probability mass function (left) and cumulative distribution (right) of the flexibility provision during 4–5 p.m. for three selected days in June, estimated by feeding the identified thermal dynamics model with historical temperature data. Note that the cumulative distribution is defined as $\mathbb {P}\{Z\ge x\}$, i.e., the probability that the delivered flexibility exceeds the value on the x-axis (in kW per hour)

Competition and liquidity: The lack of liquidity and competition within the market is a common problem with new markets, particularly at demonstrator stage. In the first trial period, many of the offers to provide flexibility were at or close to the price ceiling. While this could be an indication that the market is not offering enough value to participants, it is more likely just a sign that the trial is non-competitive. This could be reflective of a real market situation but also highlights the importance of understanding how to encourage greater market participation.

Bid Analysis

Figure 7 shows some of the outputs of a Bid Analysis Tool designed for LEO. The tool is intended to help inexperienced market participants better understand bidding strategies and how these affect the potential profitability of service provision. Market and asset parameters such as price ceilings, total availability hours, expected utilisation hours, asset round trip efficiency, levelised cost of storage, cost of energy and cost of personnel can be set before testing how different availability and utilisation bid strategies affect profit.

Fig. 7

Outputs from the LEO bid analysis tool: how the actual number of utilisation hours versus an expected utilisation of 3 h out of 10 available hours affects profit potential depending on bid strategy

As can be seen from Fig. 7, there is the possibility (for the set of parameters chosen in this example) of a monetary loss following service delivery, if the instructed utilisation differs significantly from the expected one (3 h in this example—note this value affects other parameters at bidding stage to comply with the total contract price ceiling). This is most prominent if an asset is over utilised having chosen a bidding strategy that is weighted towards availability. Anticipating an assets utilisation could be critical to ensuring the correct bid strategy is made. The green line presents a strategy that is independent of actual utilisation which might offer comfort in having a known profit, but does not guarantee the maximum one.

Baselining: Following the service window, all flexibility deliveries are verified by the DSO using metering data provided by the DER operator. At present, the position of metering, temporal resolution and accuracy of measurement is mostly the choice of the service provider so as not to force a set of regulations that restrict access to the market at such an early stage; half hourly metering from the site’s supply point meter (MPAN meter) is the minimum requirement.

The service delivery is determined by comparing the submitted metering data to a baseline. The service provider can choose between two methods for baselining. The first is one recommended by the Energy Networks Association as a future standard across all DSOs, the implementation of which is being tested through TRANSITION and LEO trials known as Historic Baseline with Same Day Adjustment (SDA), a modified X-in-Y method which is commonly used within the industry (Energy Networks Association (ENA) 2022). This is performed by the DSO using historic metering data up to 8 weeks prior to the service window with any previous event days removed. The second option is through a nominated baseline. This method uses a day ahead forecast provided by the service provider and submitted before 17:00 the day before delivery. While more lightweight in terms of data processing on the DSO side, it does require the ability to audit a DER to validate submitted baselines if unusual behaviour is identified.

The favoured baseline method is highly dependent on service provider forecasting and data processing capability, the market service, the DER type, how it is connected to the network and where the location of metering is relative to the DER. There are concerns from service providers as to how participation in other non-DSO services might impact the calculated baseline, while for the DSO, concerns regarding exposure of the market to gaming have been raised. Work is ongoing to explore other possible baseline methodologies which make use of more advanced data analytic tools.

Settlement Rule

In the event a DER under (or over) delivers on the instructed flexibility response, there is potential that network security is put at risk. To incentivise high delivery without being too strict (e.g., by applying penalties) such that inexperienced DERs with relatively high output uncertainty are not discouraged, a settlement rule is applied that reduces the utilisation payment by a greater amount for higher under delivery. This is shown in Fig. 8.

Fig. 8

The settlement rule used within project LEO to determine the fraction of utilisation payment rate as a function of the fraction of service delivered

The settlement rule was justified in a qualitative way with the key regions being: (right of point D) Any over delivery is limited to 100% of payment, there are no penalties for over delivery. (C–D) Under delivery down to 95% is still paid at 100% of the agreed utilisation rate—this reflects that there is likely some ability for the network to ‘run hot’. (B–C) Delivery lower than 95% sees the payment rate reduce gradually at 1.5% per 1% under-delivery down to 85% of delivery. (A–B) Below 85%, the payment rate reduces more strongly at 2.42% for every 1% of delivery fraction down to 50% of delivery. (left of point A) There is zero payment if delivery is lower than 50%. Further quantitative work is ongoing to see how such a settlement rule might influence bid strategy of market participants and the resulting risks that might impose on the DSO, and how a settlement rule could be designed that better reflects the requirements of the DSO.

Personnel resource: Ultimately, non-traditional market participants typically do not consider energy services, particularly flexibility, part of their business or life—energy is usually just seen as a cost incurred as part of delivering their primary function. In our experience with LEO, we have identified a personnel gap that needs to be be bridged to drive market innovation and LEM participation forward.

Organisations often have dedicated staff that deal with the procurement of energy (with little working knowledge of the estate), building managers whose role it is to maintain a building and ensure it is operating to the standard expected of its users (mostly irrespective of energy costs) and facilities management teams who upkeep hardware. It is indeed rare to encounter personnel that have the in-depth knowledge and responsibility across all three areas and with the ability—or desire—to participate in competitive flexibility markets on behalf of the organisation.

3.4 Digital Realities

The data cores of access, management, and validation (wrapped by short-term network resilience and mid-term net-zero objectives) are very different in practice where current energy ecosystems present many obstacles to implementation. The ideals of data best practices do not often translate into real-world operations and planning as systems inherently have internal inertia to overcome. This section will briefly touch on particular examples of the challenges in SLES data frameworks, using localised learnings from flexibility trials and markets to highlight generalised issues that are commonly faced in practice.

Within LEO, one of the chief causes of a primary data constraint, whereby flexibility services and participation were immediately limited, was the insufficient technical capability of assets to interface with the market. Our work uncovered many of the complexities that assets (buildings participating in DSR services in particular) can face in an immature flexibility market. Trials within both publicly and privately owned buildings showed that data infrastructure are the first impingement point for unlocking flexibility services. Buildings were often equipped with outdated and inadequate BMS that were unable to provide the level of automated connectivity needed to engage with the DSO for service provision. Considerable person-hours and stakeholder consultation was needed to be able to get necessary systems in place, and when in place, organisations often lacked the skilled personnel for continued monitoring and participation. Cases of these data roadblocks highlight some of the initial barriers in fostering LEMs and obvious data impediments in access can gravely limit the scale of services.

Although DSR is highlighted here to demonstrate issues with data access, complexities experienced in data management cannot be easily summarised with specific examples. The diverse stakeholders in a large consortium (such as that of LEO) are likely to carry out independent processes for internal data management and analysis: this brings certain challenges to attain unified and streamlined processes of data protocols and standards. To many energy agents, particularly those in emerging flexibility markets, these issues around data standards and management are very common. With so many actors in various aspects of energy procurement, operations, and research, data streams are complex and poorly regulated. The Energy Data Taskforce Report, “A Strategy for a Modern, Digitalised Energy System”, by the Energy Systems Catapult (Energy Systems Catapult 2021) has highlighted many of these concerns and the report shows that there is a clear divergence in data standards, sometimes within single organisations and projects, where the consequences of poor implementation of proper data practices can have long-term costs if action is not well-timed.

With the increased activity of LEO and associated connected assets and plugin projects, datasets will invariably possess an increasing amount of format diversity. Efforts to address these issues have led to the development of data templates that can be used for both high- and low-resolution (temporal) datasets before they are uploaded to the relevant platforms for further analysis and processing. This step allows for a standardisation of data from market trials, but more support is needed for partners beyond such ad hoc solutions. In LEO specifically, certain tools have been considered which allow partners the opportunity to run an automated ‘health scan’ of their datasets to better facilitate further processing, cleaning, and analysis. For flexibility trials, these tools are complemented by data pipelines where each ingested dataset (multimedia data) is automatically scrapped for metadata to produce data certificates, giving clear provenance of the dataset’s creation, sharing party, licencing, data description etc. The latter, though practiced with larger energy data catalogues such as the UKERC Energy Data Catalogue (UK Energy Research Centre (UKERC) 2022), often falls short as a widely adopted best practices in SLES management.

General challenges with baselining were anticipated in Sects. 2.2, 2.7 and 3.3. The research through market trials in LEO has highlighted some specific gaps for validation and data analysis. In essence, the process is simple in its framework but becomes quite complex for effective validation. Measured data brings its own issues of ownership, storage, formatting, and analysis If asset data are poor in resolution, quality, or access, validation becomes very difficult and local markets can be easily manipulated or disrupted. Validating services within LEMs is crucial to their evolution and longevity, but many data issues must be addressed in a SLES to facilitate required processes.

Across all the data cores, open-access tools are needed for wider energy system analysis and management. Web-based, transparent, open-access tools will not only demonstrate best data management practices through real-world projects and implementation but will open the opportunity for end-users to translate learnings at a local level and accelerate transitions through more coordinated efforts where data play a more central role in planning. The need for open-access tools internally will also support project needs, particularly centered around analysis and value extraction, helping local incubators of energy transitions to optimise data streams from SLESs. LEO’s core implementation of the MVS Programme and theory of change has incrementally flagged many data qualms from service providers, the DSO, and wider stakeholders. We believe that this build-measure-learn approach is crucial to rapid LEM evolution.

4 Conclusion and Outlook

This chapter has presented some of concepts behind LEMs and how through demonstrator projects such as Project LEO, these concepts can be translated into reality. Driven by the decentralisation and digitalisation of energy system, LEMs are seen as a mechanism to manage local active participation in the energy system underpinned by cost-efficient and sustainable network operation that exploits new digital infrastructure and data in innovative way.

Flexibility services are seen as a way to deliver on this cost-efficient operation allowing for constraint alleviation, connection of more variable renewable sources, the deferral or avoidance of costly network upgrades and peer-to-peer energy and capacity trading. To be successful, LEMs will need to engage with the small and many, potentially thousands of DERs per low-voltage substation at the grid edge. In order to do this, a number of technical, regulatory, commercial and social challenges need to be factored into the design. To get social acceptance and guaranteed participation, any LEM must be designed in a way that is perceived as fair and equitable to system users.

Demonstrating these concepts in real-world environments and communities is vital to uncover major barriers that could block the successful implementation of any LEM. Project LEO presented us with a unique opportunity to work in a broad collaboration that involves network operators, local authorities, academia, energy suppliers, aggregators, local DER owners and social enterprises with direct links into local communities. Implementing such a radical system change required an agile approach to be developed where DER enablement and system integration was progressively tested and adapted through MVS trials.

One of the main challenges faced was the difficulty in enabling as market participants non-traditional DERs that had limited or no prior experience as an active player within energy or flexibility markets. More specifically, potential market participants found it challenging to (i) make assessments of DER potential, (ii) justify the cost of activation in immature markets where the value of participation are still unclear, and (iii) overcome legal arrangements and understand the complexity of the market. Market trials in LEO continue to explore these challenges, trying to understand what is necessary and where support or simplification can be achieved to encourage greater participation.

In order to improve the likelihood of successful adoption of LEMs, we make the following recommendations for the basis for further work:

• Open and easily accessible tools are needed to provide quick assessment of a potential market participant’s capability of participating in LEMs. This should utilise open, shared datasets and established market outputs for technical and economic assessment, but should also include social and institutional assessments to ensure the full range of capabilities are developed. This should be a multi-stage approach, much like the MVS methodology presented in Sect. 3 of this chapter, that directs organisations or individuals to more detailed (and potentially costly) interventions only if it is sensible to do so.

• The value of ESO- and DSO-procured flexibility within LEMs is yet to be fully understood but likely to be marginal. Automation will be key to reducing operational costs associated with market participation from both the service procurement and delivery sides. Automation requires continued innovation in digital platforms and standards that ensure integration and interoperability. However, there will be a cost associated with the deployment of this technology: it is important the right balance is made that does not restrict market entry or the ability to compete for small inexperienced participants and maintains enough value in the market to be attractive.

• For non-traditional market actors that have little to no experience in the energy or flexibility markets, the in-house knowledge or personnel capable of making decisions to participate in LEMs are unlikely to be in place. This needs to be accounted for and such a role should be generally encouraged. It is expected that there will be a growth of trusted local organisation capable of offering support in this sense.

• A balance has to be struck between the required complexity in a market that ensures its successful operation and having it simple enough that it can be easily understood and engaged with by the prospective participants. While automated decision-making (e.g., service bidding performed by some AI algorithm) and post-event analysis and verification can help overcome some of these necessary complexities, simplifications should be considered wherever possible.

• Finally, intermediaries and aggregators are likely to play an important role in helping the majority of future participants engage with LEMs. They have the ability through building diversified portfolios to manage the risks associated with highly uncertain assets, helping to activate DERs and manage the legal arrangements between flexibility providers and local network operators. The concept of a social aggregator that takes a not-for-profit approach embedded within a community is likely to have good knowledge of local context and aspirations and therefore be welcomed and trusted by the community.