1 Introduction and the Relevance of Renewable Energy

Much economic and policy effort has concentrated on increasing the use of non-traditional clean and renewable sources of energy at all levels of society. Interest began muted, but by the turn of the century, countries in Northern Europe, plus most of European energy market participants, some specific countries such as Spain and Portugal, and even at the subnational level (such as Hawaii, in the US Pacific, or California and Texas), have been involved in changing the energy mix towards renewable and clean sources (EIA 2020).

From electricity generation, to grid management and expansion, to distribution, supply, and trading of energy products, all have moved from the traditional model of a monopolist utility subject to regulation in its monopoly segments in respect of its consumer base, to market-driven models of energy that incorporate renewable sources. Also, the demand part of the electricity market has changed much, from a captured consumer base by the monopoly, to facing market-driven aspects for the supply, and even to market models in distributed energy and smart grids (Nuttall et al. 2019).

A relevant question for the present chapter is; what justifies looking at market restructuring and renewable sources? The answer seems to arise from two different but clear origins. First, market restructuring has arisen from the clear acceptance that the energy industry has evolved from natural monopoly, to a complex one, with some parts maintaining their monopoly characteristics (ex. transmission, some distribution, some basic retail services), while other parts of the market have evolved away from huge economies of scale requirements towards competitive markets (an example is traditional generation of combined cycles, generation of renewables, trading, last mile transmission expansion and interconnection, among others).

In consequence, the above calls for a redefinition and restructuring, not only of the electricity market, but also of its regulatory agencies, and their attributes and instruments. Secondly, the market restructuring arises from the increasing need to commit to a greener and better-quality energy, and the integration of clean and renewable non-traditional sources of energy (one refers to clean energy as the addition of hydrogeneration to non-traditional sources such as solar, wind, biomass, tidal, geothermal, clean combined cycles). The latter fostering a more sustainable energy market.

Not only does the market-plus-energy source shift call for a thorough analysis of the new realities, but the new market definition is increasingly in need of a much more detailed, disaggregated, and multi-agent market analysis (sometimes called “high granular” information), or the move towards prosumers and “behind the meter” models of demand management. Hence the transition calls for much more detailed analyses than ever before in the history of the energy sector, including new roles and restructuring of existing policy-making ministries, agencies, and regulating bodies.

The Chapter addresses these dynamics for restructuring. In the next section, a description of market evolution and changes in nowadays mature markets is presented, from the role of multilateral agencies and worldly objectives, and how emerging and transition markets are following up, or leapfrogging the mature ones. Then a section is needed to address the institutions, policy making bodies, regulatory agencies, and the set of laws and regulations that govern new markets in the transition towards a shift of the energy source mix towards renewables. A fourth section is dedicated to technological evolution and the innovation aspects that increasingly shape the energy markets. A final part derives lessons learned, together with policy considerations and conclusions.

2 From the Multilateral Vision and Paris Agreement, Regional Efforts at the International Energy Agency, to Dynamics of Electricity Markets

2.1 The Multilateral United Nations Efforts and the Paris COP Agreement of 2015

The United Nations Climate Change initiative (UNCC), was begun in the late 1990s to multilaterally involve and encourage the main developed countries to commit to measurable efforts against climate change accelerators, such as carbon dioxide emissions by the world’s main sources, by type of sector and component effects (United National Framework Convention on Climate Change or UNFCCC 2019). Under the UNFCCC, the Kyoto Protocol promoted measurable commitments around green-house-gas emissions, separating developed countries from non-obligatory commitments by other countrie.Footnote 1 The UNFCCC was abandoned with the end of the so-called Kyoto Protocol vintage 2, around 2012 (withdrawal of Canada, echoed the non-commitment of the USA).

Then the UNCC Initiative had to revive involvement by as many governments as possible. It is part of the United Nations’ Sustainable Development Goals (SDGs), set in Monterrey, Mexico, in 2001. Since climate change is considered the most critical threat to sustainable development of the 17 SGDs, commitment is key not to allowing temperature to increase more than 2 °C by 2050. Three agendas arose after 2015 from the SDGs, as new targets after the demise of the Kyoto Protocol: (a) The Paris Agreement; (b) the 2030 c; and (c) the Sendai Framework for Disaster Risk Reduction (United Nations Climate Change 2019).

The Paris Agreement was formulated in 2015, with participation of 196 parties, but with commitments by the governments of only 30 developed countries (mainly the EU, plus Australia, and New Zealand), while increasingly involving single, medium-income emerging economies, such as China, India, Mexico, and Brazil. Its main objective is for all participants to lower all elements that could increase world temperature by 2 °C (and mostly closer to 1.5 °C) and reach the levels of the pre-industrial era (base year 1992, and target year 2050). The global temperature had already increased by around 1 °C by 2018, which is why the objectives of the Paris Agreement are now rather pressing. Note that the United States and Canada had still not signed the Agreement by the time of the present investigation (Fig. 1).

Fig. 1
figure 1

Doha Amendment to the Kyoto Protocol for Paris Accord (2019)

Distribution of signing parties of the Paris Accord 2015.

Full size image

The impacts of climate change can indeed be very devastating, mainly for poorer countries. For that matter, the UNCC seeks to facilitate financial flows to these economies, along with enhancing opportunities for technological advance and making it possible for countries to develop capacities to benefit and disseminate new technologies. Bureaucracy, red tape, and unclear obligations for institutions designated to host UNCC projects, could become impediments to the improvement of climate change conditions. Furthermore, UNCC aims for all project actions to be transparent and subject to good governance. Again, obscure funding rules could give rise to corruption, as disclosed in various documents, after commission memoranda of the UNCC (2019).

Nationally determined contributions (NDCs) are self-committed measurable obligations by each country/member, updated and communicated yearly. COP 15 in 2019 was held in Santiago, Chile (UNCC 2019). Some of the relevant areas of NDCs include country leaders in committees for overseeing the following topics (ibid.).

  1. 1.

    Mitigation

  2. 2.

    Social and Political Drivers

  3. 3.

    Youth and Public Mobilisation

  4. 4.

    Energy Transition

  5. 5.

    Industry Transition

  6. 6.

    Infrastructure, Cities, and Local Action

  7. 7.

    Nature-based Solutions

  8. 8.

    Resilience and Adaptation

  9. 9.

    Climate Finance and Carbon Markets.

In the past, projects were led, but then shared for multilateral yearly conferences, where governments offered measurable updated commitments, and declared a designated government agency, secretariat, or ministry to carry on the specific projects. In the case of Mexico, the Secretary of the Environment, called SEMARNAT, has been designated to lead the national inventories of pollutant agents, gases, and projects regarding the environmental agenda, with links to energy concerns. Now, given some developing countries’ elections and political changes, some have reduced their commitment positions, or even the financing and governing of their climate change obligations, and their links with the energy transition agenda (Ibarra-Yunez 2016; Ibarra-Yunez et al. 2017).

2.2 The International Energy Agency and Its Role in Bridging Gaps Between Climate and Energy

Take the example of Mexico, which has seen its GHG emissions increase rapidly until 2011, mainly as the result of strong economic growth rates. According to the International Energy Agency (IEA), a world forum to discuss climate change, obligations, role of various energy markets, capacity for expanding and sharing energy development efforts with private parties. From 1990, energy-related CO2 emissions grew by two-thirds to 431 million tonnes (Mt or Teragrams) in 2014, where fossil fuels account for 90% of Mexico’s primary energy supply. Emissions increased more markedly in transportation services and electricity generation. It was anticipated, until Mexico’s elections in 2018, that improvements in productivity, would motivate economic growth and energy demand.

According to the IEA, in its 2018 Report, power generation was expected to rise from around 300 Terawatt-hours (TWh) in 2017 to around 470 TWh in 2029, and over 500 TWh by 2040 (IEA 2018). Hence Mexico’s new government needs to internalise the challenges to the energy system to become more competitive and more sustainable. Mexico has now embraced the idea of integrating climate change objectives into energy policy making, as evidenced by its Climate Change Law to reduce GHG emissions by 30% from the business-as-usual level in 2020, and 50% from 2020 to 2050, as shown in the IEA Country Report (2018). Measuring cost effects of objectives by all projects was a recommendation set by OECD/IEA, 2017. One drawback in Mexico’s case is the fact that the new Energy Law, passed in 2014, was rather integral and detailed around swift changes to market and regulations (LIE 2014, published in the Official Gazette in August of that year: dof Aug 14, 2014). Many other Market Rules were set for all stakeholders in the electricity sector that promote private investment and participation. However, the so-called Energy Transition Law, the “green agenda”, was separated from the electricity law, and came into force afterwards, in December 2016.

As an example of the weak links between the energy and climate change agendas, Table 1 helps to understand such a weakness, for which renewed efforts should be adopted to increase coordination of the two agendas. Unfortunately, for this economy, the incoming administration of Pres. Lopez Obrador, in its National Development Plan, and voted on in Congress in May, 2019, little or no mention is made of climate change commitments, or that guarantees are clearly provided for market development.

Table 1 Scope and objectives of Mexico’s Energy Law (LIE) and Energy Transition Law (LTE)
Full size table
Table 2 Legislation in Mexico’s energy reform 2014–2018
Full size table

The IEA, in its declaration of intended nationally determined contribution (INDC), includes the goal to reduce the emissions of short-lived climate pollutants (SLCPs). Integrating climate policy and air pollution policy brings synergies, as reducing SLCPs has a direct impact on air quality and the health of the population. The IEA then urges the government to strengthen national air quality policy to provide a solid basis for limiting air pollution. The above requires that all signatories to the IEA vision, present clear measurable objectives for achieving; energy subsectors in the hydrocarbons industry, facility for gas transportation, warehousing, and commercialization, and that the electricity industry, including renewables, appoint responsible entities, and promote markets. At the time of this analysis, it is unclear if all requirements are, or will be, met by the incoming Mexican government.

Given a return to centralised government in the electricity and oil sectors, Mexico’s emissions may actually increase from today’s levels, with an apparent minimisation or disregard, by the new presidential administration, of Mexico’s compulsory position in multilateral forums. Additionally, from pure population growth, the projected increase in transport may have worrisome implications for emissions. This calls for urgent strengthening of sectoral policies, especially in transport, and for developing a comprehensive policy approach for all sectors. That Mexico’s current administration will take these concerns seriously is unclear, at the moment. Some clear efforts can be seen in specific urban projects abroad, such as those for London, Paris, Montreal, Berlin, Valencia, Austin, and Curitiba in Brazil, to name salient efforts by some smart communities.

Smart city initiatives are a clear and challenging part of IEA’s goals because they involve many actors in the energy and climate change panorama. Mexico, for example, a major energy producer, has experienced its energy policy as traditionally focused on the supply side rather than on the demand side, with some muted administrative decisions over distributed energy projects. The same has applied to climate-related policies. Noticeable is the challenge to coordinate government efforts by the Secretaries of Energy and the Environment, Economic Promotion by the Secretary of the Economy, including agencies such as the Energy Regulatory Commission, and all institutional settings to guarantee loci wholesaling of electricity and its derivative sub-markets, gas, and oil (named CENACE, CENAGAS, and CNH). As the IEA notes, “In general, energy efficiency and clean transport are important, as are cost-effective policy levers that could play a stronger role in a comprehensive clean energy and climate change plan”.

2.3 Electricity Sector Evolution Dynamics

Markets are stimulated by deregulation, re-regulation, and technological innovation. One can see the first market changes becoming widespread in Nordic countries, where, from centralised utilities, Finland, Sweden, Denmark, and later Norway, all moved to separating the competitive side of the markets from the monopoly parts, subject to regulations. International interconnections were part of the dynamic market alignments, for which the Nordel regional market set rules for electricity generation, transmission, and distribution, but they also progressed to align the prices of energy across the countries, and embrace transmission expansion to integrate renewable sources of generation and demand.

Then the so-called European market UCTE integrated many international Western European electricity interconnections into a market, where electricity in all its forms (including renewables) was transformed, from a utility towards a tradeable good and service. The market includes Austria, Belgium, France, Germany, Greece, Italy, Luxemburg, Netherlands, Portugal, Spain, and Switzerland. Since 2011, this interconnection market has adopted a new community name, the ENTSO-e, and it guarantees total convergence of trans-border transmission tariffs. Moreover, the European Commission on Energy has launched a trans-European plan for a 2050 carbon neutral energy market (European Commission 2018).Footnote 2

How do energy markets look in their transition, and how are the parts separated from being unified and centralised in their policy and administration, towards being decentralised, segmented, and competitive, and where policy and administration operate under shared ownership and collective decision-making, and not ‘the government’ only. Figure 2 could be useful to the understanding of such an evolution, which is where most countries have moved, with few exceptions, in the world.

Fig. 2
figure 2

Source Taken from Srivastava et al. (2011)

Alternative restructuring of electricity markets. ISO: Independent System Operator; AS: Ancillary Service Providers; PX: Power Exchanges; SC: Scheduling Coordinators; TO: Transmission Owners.

Full size image

As one will notice, the Independent System Operator (ISO), is the centrepiece of the wholesale market and locus for market clearance. Some markets such as NGC in the UK, Alberta in Canada, California in US, ERCOT in Texas, the case of Norway, and finally other comparative markets, such as VPX in Australia, PJM, and New York NYPP are also shown in the above figure.

The segments of the markets that are of interest are various components of the emerging electricity market configured as generators (G), power marketers (PM), power exchanges (PX), transmission owners (TO), the independent system operator (ISO), ancillary service providers (AS), scheduling coordinators (SC), retail service providers (R), and distribution service providers (D). Note the diversity of roles the ISO has in the various management models, such that in some markets like the UK’s NGC, the ISO integrates transmission and power exchange decisions, while in Norway, the ISO integrates with transmission services and infrastructure, while in most of the relevant mature markets, power exchange decisions and management are an integral part of the ISO. ERCOT, Alberta, and notably California are examples of totally disaggregated or market-oriented sectors, with multiple independent players in the electricity ecosystem. All integrate renewables (Srivastava et al. 2011).

An additional critical aspect of market segmentation, that the energy sector has experienced, is cost allocation, hence pricing across all segments of the industry to ascertain sustainability. Moreover, some segments have given rise to new players (aggregators, traders, service engineering, financing, among others, as well as products additional to only-energy, but that include submarkets for capacity, balance, financial transmission rights, clean energy certificates or credits, and other submarkets), for which one has to accept that new markets are, nowadays, deep and complex.

The present chapter gives a brief but complete idea of the dynamics of the electricity sector; the role of multilateral agendas under UNFCCC-then UNCC, for example, the Paris Accords, role of the IEA in promoting renewables and governance of the energy sector under new market conditions for many stakeholders and interest groups in the industry, and then briefly and visually present alternative market structures across the main mature markets. The next section will address the institutional and regulatory settings across some energy markets, with emphasis on Mexico’s recent changes in commitments, agencies, and the legal impacts of the deepest and most integral energy reform.

3 Institutions, Policy Making Bodies, Regulatory Agencies, and Their Updated Roles

In Mexico, arising out of the Constitutional changes that occurred in December, 2013, so-called secondary laws were enacted, starting in mid-2014, by the Mexican Congress, These included at least 22 new laws whose objective was a long-awaited de-regulatory stance, that also promoted private investors as needed complements of the three main energy submarkets: hydrocarbons, natural gas, its infrastructure, storage, and commercialization, and electricity, in both its basic services and wholesale markets. In the latter, a specific law established the breath of the ISO called Centro Nacional de Control de Energía, or CENACE, as the central pillar of the new market.Footnote 3

Before describing and analysing the crucial role of CENACE, suffice it to list all relevant legislation, executive orders, administrative rulings, and regulatory decisions to end this section.

The Law of the Electric Industry (Ley de la Industria Eléctrica), Chapter II, Arts. 107–112, established that CENACE would become an independent public organisation, with juridical independence and its own funding. Stated objectives included that it: (a) be the operative control of the national electrical system (SEN); (b) be the locus of all operations of the electricity wholesale market; (c) guarantee open access, with no undue discrimination, to the national transmission and distribution networks, among the ones spelled out in the Law (Art. 107); and (d) guarantee the operation of the whole system, within the parameters of efficiency, quality, reliability, continuity, security, and sustainability (Art. 109). Later, CENACE was administratively established by a decree published on August 28, 2014, in the National Gazette (Dof Aug 28, 2014).

Two critical mandates were placed on CENACE. First, in order to align the objectives of increasing generation and the supply of renewable sources of energy, auctions were to be scheduled to occur at least once a year, beginning in 2015. Three such renewable energy auctions were successful in opening up a market for long term commitment of renewable markets for energy, power, and clean energy certificates, until 2018. In the first auction, 11 projects were adjudicated (6 were in operation by 2018, mainly in the Yucatán peninsula in southern Mexico); in the second, 20 projects received permits (4 were operating by the end of 2018); in the third, 10 projects were adjudicated (by the end of 2018, none of them was operating).

It is important to note that the auction mechanisms were successful in attracting competitive players to the market. At the same time as investment was committed to renewable wind and solar project capacity, hydroelectric projects were set up in the States of Chihuahua, Hidalgo, Jalisco, México, Oaxaca, Puebla, Sinaloa, and Veracruz; together with geo-thermal contracts in Jalisco, Michoacán, and Nayarit, plus one biomass project in the State of Chiapas (García Alcocer 2018; Ramirez-Cabrera 2019). According to the Energy Regulatory Commission, USD$2.5 billion (109USD) were committed in the first auction, while USD$4 billion were committed in the second, and USD$ 2.4 billion in the third (García-Alcocer 2018).

45 countries have used renewable energy auctions as a transparent vehicle to promote new players and new sources. As the main results, one can posit the following:

  1. (1)

    Generation costs have declined, which creates substantial savings to the incumbent state enterprise CFE-Basic services.

  2. (2)

    Auctions create a diversification of the energy matrix, as renewable energy projects are incentivised.

  3. (3)

    Auctions promote investment away from the incumbent CFE which, during the transition, had not enough resources to embark on creating a new market itself. Committed investments, as shown above, reached almost USD$ 9 billion, with no risk to the state.

  4. (4)

    Success at the auctions creates positive externalities, as more investors are drawn to subsequent auctions.

A second critical mandate was for CENACE to establish clear rules for: the short term electricity market (day-ahead, and real time), medium term, and long term (latter pending); technical interconnection (network codes, market rules); a capacity submarket; a market for renewable energy certificates and allocation (somewhat pending); ancillary services; financial transmission rights; and, other submarkets. At the time of the present analysis, some of the latter parts have been put on hold by the new administration of Pres. Lopez Obrador, which could hinder the wholesale market and its participants (SENER 2019).

How does CENACE compare with the United States’ markets, after three years of operation, with around ten years of review and the boost of a host of modern regulations, and taking into account the brief report of the Federal Regulatory Energy Commission (FERC) on ISO’s performance, and, to begin with, FERC order 2000, which established a similar mandate for the operation of the crucial Independent System Operator, with its so-called “minimum functions”?

By the end of 1999, FERC defined the “minimum functions” of an RTO/ISO in Docket No. RM99-2-000, Order No. 2000. FERC based these minimum functions on lessons learned by the regulator since FERC Order No. 888, which provided a basis for the first U.S. RTOs in 1996. In Table 3, FERC’s minimum functions are applied in the context of Mexico’s CENACE as a basis for evaluating the system operator’s development to date. These RTO/ISO minimum functions are listed and defined in the first two columns, and their strengths and weaknesses are evaluated in the following columns (Table 3).

Table 3 Minimum functions and CENACE’s strengths and weaknesses afster its third year in operation 2018
Full size table

While CENACE is on track to fulfil most of the minimum functions outlined in this table (Table 3) in the medium term, the table shows lack of progress in some submarkets, such as in ancillary services development. Other areas that will be especially important for CENACE to address in terms of encouraging renewables uptake, include encouraging congestion management and parallel path flow improvements to provide flexibility that an increasingly renewables-driven grid demands (Mohler and Sowder 2017, p. 285). As it plans grid expansion and interregional coordination of control areas, CENACE can develop strategies to harness renewable resources in areas with high wind, solar, and/or geothermal generation potential, whilst also considering critical transmission points across regions that may arise due to high-renewable generation scenarios, although these might change due to the stalling of renewable and wholesale market promotion.

CENACE’s capacity as a market monitor and tariff designer also positions it well to continue to open the wholesale market to new participants and provide a fair platform for their participation, reducing costs through greater competition in generation, and transparency in pricing. CENACE’s continued role in serving as the sole provider of transmission services is also a key feature of ensuring non-discriminatory access to new generators, renewable or otherwise (FERC 1999, p. 330).

For the transitional market in Mexico, an additional dimension is the speed of moving from a centralised governance CENACE model, towards a true market with minimal central intervention. However, the new administration of Pres. López Obrador, and overall policy by the Secretary of Energy in Mexico, seem to be at odds with the concept of markets. Additionally, CENACE, in an increasing mature market, would be moving towards a neutral-passive role in the market clearing processes. Speed and a strategy of increased implementation of market functions is critical for success, especially as it pertains to rapidly developing approaches that support renewable uptake and utilization.

Given the rapidly changing nature of Mexico’s electricity system, and CENACE’s identified need for performance assessment (and metrics for its performance in the transition), the application of minimum market functions to CENACE’s third and following years would be an important step towards building a continually evolving assessment framework. Given that the functions of wholesale markets continually develop, market operators and monitors can establish new minimum functions as needed, which can be used additional with the minimum functions presented above. From the established minimum market functions, market operators can go into further detail to create measurable indicators of success, or performance metrics. However, the path is additionally complicated by technology changes in the sector.

4 Technology Drivers of Renewable Transitions

The present section provides a literature review to help assess technological developments and advances in renewable energy technologies for electricity markets. Technological developments are traditionally classified with respect to; generation, transmission, and final consumption. This section assesses aspects, such as installed capacity, and the costs of key technologies in each of the subsections (generation, transmission, and final consumption) that enable a more reliable and secure grid. Highlighted U.S. trends and policy drivers for each technology are also summarised.

In 2016, the contribution of US renewable energy sources to the energy mix experienced its largest growth, ever, especially in the global electricity sector, according to IRENA (2017a), when generation from renewables increased by 6% (IEA 2017). Generation capacity increased from 1,845 to 2,006 GW between 2015 and 2016, an addition of 161 GW (IRENA 2017c). Wind and solar photovoltaics (PV) have been the technologies making the greatest contribution, and accounting for 51 GW and 71 GW, respectively, again, according to IRENA (2017b).

Growth in the United States markets apparently has grown more than has the total worldwide. Thus the 6% yearly US growth in the past decade, compares to an average worldwide growth of 3.6% (IEA 2017). For renewable power capacity, growth rates were around 4% in the early 2000’s. By 2015, however, growth rates had increased, almost to 10% (IRENA 2017d).

Now, regarding installed capacity for generation, per technology, hydropower is still the main technology (International Energy Agency 2016a). However, it does not compare to what are now the dominant “modern renewables”, since, in addition, hydropower generating capacity has been decreasing (International Energy Agency 2016b). Modern renewables, such as solar PV (photovoltaic) and wind-generated energy have experienced relevant growth in 2016. Solar PV capacity increased to around 70–75 GW. Onshore wind capacity’s growth was 15% less compared to 2015, with 50 GW of new installed capacity compared to 70 GW installed in 2015. As for offshore wind, new installed capacity accounted for 2 GW, compared to 3 GW of additional capacity installed in 2015 (IEA 2017; International Energy Agency 2016b; Ibarra-Yunez et al. 2017).

The renewable agenda in the United States arises from incentives at federal level, but mostly at state and local levels. They could be described in terms of the following levels: de-regulation and re-regulation incentives; fiscal incentives; and, financial incentives. DSIREusa.org, presents the three levels of incentives at the zip code level, for; individual projects at residential generation, self-supply, distributed generation, projects for self-sufficiency, including, among others, construction standards. Additionally, information and incentives are readily available for non-residential efficiency, renewable energy uptake, and dissemination of new technologies. Such as the Renewable Portfolio Standards in many states (RPS), and at regional transmission organisation levels, arising mostly from the state’s Public Utility Commissions (PUCs), as well as by regional market agencies, such as CAISO in California, the Southwestern Power Pool (SPP), ERCOT in the majority of Texas, the Midwestern interconnection regional market (MISO), the PJM regional market, NYISO, New England interconnection (NE ISO), and others.

For Mexico, renewable energy uptake has been motivated by two main drivers: on the one hand, the Energy Reform, its legal compact, and its institutions both emanated from the reform itself; but also from the Law of Energy Transition (meaning the green agenda and the agenda for the setting of standards and norms, similar to the quest of the Renewable Portfolio Standards mentioned above). These have been the key boost for renewable energy projects, along with a commitment to Mexico’s objectives of 35% renewable generation, and load, by 2024. Agencies in charge of incentivising these are the Commission for the Efficient Use of Energy (CONUEE), the ASEA with special overseeing competences on hydrocarbons, together with some residual competences by the national energy regulator CRE, similar to the US FERC, and the North American Electricity Reliability Corporation (NERC), dedicated to overseeing reliability standards, and interconnections (NERC.com).

Secondly, and in a different but related issue for renewables, the drive also comes from technological developments, to which we can now turn. One can visualise the different technological changes, both from pure innovation, but mainly from intent applied to the different parts of the electricity markets, as is shown in the next sections.

4.1 Solar-Photovoltaic Generation Technology

Solar-Photovoltaic power capacity grew from 39 GW in 2010 to 295 GW in 2016, a dramatic change in the United States. Between 2015 and 2016 solar power capacity reached a new record with the addition of approximately 70 GW. It is the first time that solar-photovoltaic additions surpass wind additions (IEA, 2017). This represents an approximate growth rate of 40% (IRENA 2016b, d). Also in 2016, solar-photovoltaic represented about 15% of total renewable generating capacity.

Now, the averaged capacity weighted cost of solar generation (LCOE, meaning the net present value of the unit-cost of an installation over its lifetime), fell around 60%, between 2015 and 2017, according to IRENA (IRENA 2017e). This means that costs have become rather competitive in relation to fossil fuel, even gas combined cycle. Part of the explanation for such an expansion of solar-photovoltaic generation investment, has been tax credits of 30%, along with the so-called state Renewable Portfolio Standards RPS (IRENA 2017b, Delmans and Montes-Sancho 2011).

For a solar-photovoltaic installation to succeed as a renewable alternative, two elements should be present in the market: (a) smart net metering; and (b) third party resale of electricity at retail prices back to the utility, taking care to keep the latter economically viable. In many markets, the solar-photovoltaic uptake has been less than surprising, since badly designed mechanisms have been ousted in favour of “prosumers”. Rate design is critical. In the case of Mexico, some of the above critical aspects are yet to be settled. Moreover, a slight but critical issue is that of designing individual project net metering/net billing, and that at community net metering for cases of distributed energy, and even urban “smart” developments, these aspects extend well beyond energy prosumer capacities (Durkay 2016).

4.2 Wind Generation

By the end of 2016, power generating capacity derived from renewables amounted to over 2,000 GW. Of this, 23% was attributable to wind technologies (IRENA 2017c). Global installed wind-generating capacity increased from 7.5 GW in 1997 to more than 465 GW in 2016, for both onshore and offshore wind (IRENA 2017c; IRENA and IEA-ETSAP 2016). One of the most promising technologies is offshore wind production, and by the end of 2014 installed capacity worldwide was 8.8 GW (Global Wind Energy Council 2015). Most wind offshore farms in developed economies, could learn from experiences in the UK (Scotland, England, and Wales), The Netherlands, Germany, China, and Denmark (farms with around 400–600 MW capacity, according to the Siemens and Siemens Gamesa company webpages, the largest project developers in 2015–2018).

The average cost of energy, for both onshore and offshore wind has experienced a large decline. In 2010, the average LCOE for onshore wind was 0.071 USD/kWh, while in 2016 it decreased to 0.056 USD/kWh. As for offshore wind, LCOE in 2010 corresponded to 0.133 USD/kWh but has decreased to 0.123 USD/kWh in 2016. Between these technologies, onshore wind is still the most competitive against fossil fuels, which costs approximately 0.045 USD/kWh (IRENA 2017b).

On a US federal level, wind development has been stimulated by the $0.024/kwh subsidy known as the Production Tax Credit. Accelerated depreciation, which allows owners to write off capital costs, is another federal policy that has contributed to the growth of the industry (DOE 2015). In the US case, Texas leads wind generation across the country, with 18 GW capacity and growing steadily. One critical aspect for the success of this technology, is the siting facilitation and fast community consultation for rights of way. The other critical aspect is to coordinate wind projects and uptake in zones far away from load and consumer nodes and zones, with transmission expansion to evacuate energy towards consumption centres. One notable success story is the Texas-ERCOT programme for establishing “Competitive Renewable Energy Zones” (CREZ). (see https://www.oncor.com/en/Pages/CREZ.aspx).

4.3 Distributed Generation

On the side of generation developments, distributed energy has been recently present as an innovating way of solving generation in rural and distant areas, but increasingly so in urban developments, with a recent emphasis on how spaces can be opened for isolated supply projects, or in Spanish, projects in “Abasto Aislado”. The Mexican regulator has produced documents to incentivise distributed generation along with planning of a back-up basic electricity supply by the utility CFE. However, projects are moving slowly.

Currently, over 1 billion people in the world lack electricity access, and another billion have an unreliable supply (IRENA 2017d). Distributed generation, or off-grid systems, allow access to electricity for distant communities. Despite the population still lacking electricity access, off-grid systems have grown significantly in recent years. Off-grid systems can be classified into two types, depending on how they produce electricity: conventional systems; and, renewable off-grid systems.

Renewable off-grid systems are the most economical option for off-grid electrification (IRENA 2017d). Conventional systems use fuel and gas generators to produce electricity. Currently, installed capacity of diesel generators is 400 GW. However, from 50 to 250 GW of the total installed diesel capacity could be hybridised, according to Kempener et al. (2015). Renewable off-grid systems use small renewable sources to produce electricity. Almost 26 million households benefit from off-grid renewable systems worldwide. Approximately 20 million households have solar home systems, and another 5 million households use renewable-based micro-grids. Even more, 0.8 (% of) worldwide households use small wind turbines (Kempener et al. 2015).

Distributed generation also includes technologies such as fuel cells, and energy storage systems. Small-scale solar-photovoltaic is the most common distributed technology in the U.S., while the net capacity from the residential, industrial, and commercial photo-voltaic sector was over 13 GW as of February 2017 (EIA 2020) 0.102 Property Assessed Clean Energy (PACE), which finances residential and commercial energy efficiency and renewable installations through a property lien or retention right, has financed $1.37 billion in residential renewable energy installations from 2010 to 2016. In addition to access to end-user financing, distributed generation requirements, streamlined permitting processes, and clearly defined interconnection procedures are all policies needed for the growth of distributed energy technologies. They are still a lesson for Mexico to put in place. Let us now turn to the area of transmission innovation, and technological developments.

4.4 Transmission Areas of Opportunity for Development

A pressing challenge is how to move from evaluating demand, balancing, and congestion in a grid with high demand growth and integrating renewables in a neutral way, meaning a system that does not affect decisions for generating locations, or consumption centres, or what is called solving the problem of ‘generation drives, transmission’, as opposed to ‘transmission drives generation’. The above implies that planning and incentive investments must be aligned for transmission with new sources of generation and large consumption nodes.

Moreover, one needs to separate transmission maintenance needs, from transmission expansion, with the existence of new technologies. Moreover, in mature markets, there is a way of looking at energy, not only as a public service in many cases, but also as a tradeable good that incentivises interconnection by regions, but also connects across borders. Mexico’s CFE Transmisión is a subsidiary of the CFE headquarters, but where financing of maintenance, last-mile expansion projects with private investment participants are allowed by regulations, but where little or no thought has been developed for long transmission (even the High Voltage DC Transmission), or for transmission across the US and Mexico borders (such as with California. See Rosellón et al. 2017, 2018).

Another source of innovation in transmission is the area of smart grids. Smart grid technology—or digital technology that enables two-way communication between a utility, it is the transmission and distribution system, and the end-user—includes technologies such as advanced metering infrastructure, voltage regulation equipment, power flow controllers, and equipment health sensors to balance the demand and supply of markets. IT technology is a critical component of smart grid deployment.

While early smart grid solutions were targeted toward distribution, given that they were applied close to consumers at low voltages, there has been an increasing incorporation of smart technologies in switches, substations and transformers, allowing for two-way flow and increased transmission flexibility. The increase in electricity generation from variable renewable energies (VRE) poses several challenges for the grid, including the need to rapidly synchronise supply and demand. This has become more of an issue in grid management, since the peaks of VRE are more pronounced. Therefore, grids need to become smarter and more flexible (IRENA 2016c).

Moreover, demand response, consumer efficiency, and energy management have become areas of high expectations for energy intensive industries, but also for small and medium enterprises that, given that they cannot reach the minimal required load of, say 1 MW of capacity demand, have begun to promote the role of aggregators, which become another vertical link in the energy market (at wholesale levels). Such have been studied recently for Mexico (Ibarra-Yunez et al. 2019).

Advanced metering infrastructure (AMI) developments intend to capitalise on “big data” (International Energy Agency 2016b). This has increased the relevance of joint information technology and operation technology (IT/OT) solutions. The IT/OT solutions market grew approximately 65% per year and is expected to grow by 600% by 2023 (Navigant 2014). IT/OT solutions will allow utilities to have control over all connected devices, and therefore have a “real time” management of supply and demand (IRENA 2016c). Investment in smart grid technology, which incorporates IT/OT systems, increased by 12% in 2015. Moreover, new business models, such as virtual power plants (VPPs) are also expanding (International Energy Agency 2016b). Over the next twenty years, experts predict that $17 to $24 billion, on average, will be invested yearly in smart grid upgrades.

One last point about IT/OY solutions is their application in urban areas, mainly in the case of public transportation and traffic congestion management, but with clear expansions aimed at reducing pollution in metropolitan areas, such as London, Paris, Valencia, Curitiba-Brazil, or Los Angeles. Innovation can be integrated into new business models.

4.5 Storage

Electricity storage plays a critical role in the integration of VRE to the grid. The extent to which the power generated during the peak production hours can be stored is crucial for full exploitation of renewable energies. Storage is also crucial for increasing operating system flexibility (IRENA 2017d).

The most developed storage technology is pumped hydropower. This technology currently has over 145 GW in operation, accounting for the majority of energy storage worldwide. In the case of Mexico, the new vision is for traditional hydropower to gain importance over the next 5–10 years, although little is known of new business models that integrate storage with balancing markets at intraday, seasonal, or emergency management in respect of virtual versus physical capacity.

According to IRENA, pumped storage hydropower will increase from 150 to 352 GW by 2030. On its part, battery storage is the subsector with the most important recent growth. In 2014, 400 additional MW were added, thereby doubling 2013 installations (International Energy Agency 2016b). Battery storage is expected to increase from 0.8 to 250 GW by 2030 (IRENA 2015). Currently, lithium-ion batteries dominate the electricity storage market, which is a significant shift from prior decades when sodium sulphur batteries were the dominant technology (IRENA 2017d). The market for this type of battery grew by 50% per year, helped by the commercialization of the Tesla Powerwall, a 10 kWh lithium-ion battery. (International Energy Agency 2016b).

Energy storage technologies are becoming increasingly cost-efficient. In 2005, costs for lithium-ion batteries were around US$1,500/kWh. In 2016 costs were around US$350/kWh (New York Battery and Energy Storage Technology Consortium, 2016). Moreover, between 2015 and 2016, the cost of batteries at grid scale, fell 12% for Peaker plant replacement, and by 24% in the case of transmission (Lazard 2016). Finally, it is expected that battery prices will continue to fall; forecasts suggest a 50% decline by 2019 (Wilkinson 2015).

4.6 Electric Vehicles and Smart Buildings

Let us finish this section with a note on electric vehicles and smart building technology developments. Electric vehicles today can be classified into two main technologies, battery electric vehicles (BEV), and plug-in hybrid electric vehicles (PHEV). BEVs only use batteries to store energy, and therefore, they must be plugged-in in order to recharge (IRENA 2017a). New discussions at the utility level, have tried to calculate the stress on the electricity distribution grid of these new developments. Moreover, a related issue concerns consumer habits in charging BEVs during the day in commercial or office outlets during semi-base or peak tariffs, or at home, at night, at basic electrical tariff, with full recognition of electricity consumption in electricity bills; a case of free riding, or not free riding.

For their part, PHVEs use batteries as well as liquid-fuel storage. Moreover, liquid fuel can be replenished, allowing for greater ranges of travel (IRENA 2017a). A new discussion has arisen about public transport vehicles, whether they should be BEV or PHEV. Worldwide, the number of electric vehicles reached one million in 2015, and surpassed 2 million by 2016 (ibid. IRENA, 2017a). In 2015, 477,000 electric passenger cars were sold worldwide, according to the International Energy Agency(2016b). Typically, BEVs have dominated sales. However, PHEVs have been gaining strength, mainly as a more levelled option, mainly for buses, but also for electric tramways and more massive transportation.

Beyond light electric vehicles, a market exists for the two-wheeled variety. These types of vehicles are sold mainly in Asia, but one can see electric mopeds (and the Segway, now banned in many cities) in many US and European cities, or even some highly congested traffic cities in Mexico. Over the last ten years or so, over 200 million units have been sold (Cherry 2016). Only in 2015, almost 40 million vehicles were sold mainly in China and Japan (IRENA 2017a).

Now, regarding smart buildings and smart building materials, in order to reduce energy consumption in installations where demand for heating and cooling forms almost one third of energy demand, several countries have developed building energy codes and energy efficiency policies. As a result, global building energy performance has improved 1.5% (International Energy Agency 2016b).

Global markets for smart buildings in urban areas are growing. The estimated value of the market was USD 4.8 billion in 2012, however; value is expected to rise to more than USD 35 billion by 2020. Moreover, sales for smart appliances are also rising. Sales in 2015 accounted for USD 5 billion, and they are expected to reach the USD 34 billion by 2020. According to energy efficiency codes, such as Mexico’s CONUEE’s norms, government buildings are first to adopt energy efficient materials for construction, but many government facilities use old buildings, which sets a big challenge for enacting the CONUEE’s building materials codes. In any case, the process of using technologies to integrate smart buildings in all their electricity, gas, and water use is ongoing. In the United States, per FERC Order 745, energy efficiency and demand response are marketable, and must be compensated at the same locational marginal price as an energy generator. However, it is state level policy that drives energy efficiency, and therefore smart building technology.

5 Conclusions and Recommendations for the Future of Energy Markets, Regulations, and Best Practice

The present analysis of the status of both the energy and climate change agendas, first, has shown that both are fundamental ingredients towards reaching an increasingly needed energy sector in its overall economic importance, at the same time that countries adhere and commit to their environmental concerns and include them in all government and private sector planning.

However, as the chapter emphasises, the institutional and regulatory agendas for the case of Mexico, have departed from different bases, for which the links between the two objectives seem weak at the least. One example is the link between the legal promotion of renewable and clean (including hydrogeneration) sources of energy via the clean energy certificates (credits), or CELs as they are called in Mexico, their assignment through long term auctions between 2016 when they started, and 2018, after which they have been put on hold by the new presidential administration.

Moreover, market regulatory change during 2013 and 2018 was rather profound and included 21 changes in laws and regulations, additional to more than one hundred administrative orders that opened the electricity sector to private participants in all but natural monopoly parts of the sector, and a wholesale market was launched in 2014: the National Center for Energy Control as the Independent System Operator ISO, or Centro Nacional de Control de Energia, CENACE. As it is argued in this chapter, the role of ISO’s in most mature markets in the world, is central to promote wholesale trade of energy, capacity, clean energy certificates, ancillary services, financial transmission rights in short, medium term, and long-term trades.

For a new restructured energy sector such as Mexico’s, some parts of these markets are still in the planning process, and not all sub-markets are operating at present. Moreover, the new administration has found itself undefined if a more profound market-driven electricity sector, including renewable energy and a quantitative-measurable target of lowering the carbon footprint for this economy, or more centralised, top-down government intervention is a preferred model (going back to the old model). In such a decision-making uncertainty, the key CENACE locus of wholesale market clearing, planning, and leadership, has seen its role somewhat diminished, at the same time that lack of performance measures has characterized this independent system operator. Moreover, the chapter presents the reader with the methodology of “minimum functions,” that are derived from FERC Order 2000 of 1999, and other market experiences. Indeed, the minimum required functions are spelled out in their critical parts towards a mature, well-behaved wholesale market, with many participants (except electricity basic services, subject to asymmetric regulation).

Then the chapter compared the experiences from international institutions, such as the United Nations and its evolution of the climate change objectives and institutions and agencies. Also, in the analysis, other institutions are reviewed in their aims and representativeness, such as the Paris COP Agreement and its targets, and the International Energy Agency, from which Mexico has been welcomed as a new member. All of the clean energy objectives and commitments by governments, and their agencies pertain to all signatories, including Mexico, additional to the country’s Secretariats and agencies in charge of overseeing norms and standards towards reaching the clean energy and climate change targets: Secretary of the Environment SEMARNAT, and agencies such as CONUEE for norms and standards of clean energy, and ASEA for hydrocarbons.

Finally, one has to bear in mind, that public policy, sectoral regulation, ex post competition policy regulation all come later than technology advances. The final part of the present analysis summarises main technology concerns and advances in the energy sector, as we divide it for technological analysis into generation: solar-photovoltaic, wind, and distributed generation; then transmission: AC, DC-HDC, last mile private complementarities, interconnections; then storage and batteries; and finally electric and hybrid vehicles and smart buildings. With the above, the reader is able to clearly locate areas of opportunity for the electricity and renewable energy industry in Mexico, in comparison with other experiences in the world.

  • As a series of policy and industry recommendations for the future, first, Mexico is member and has committed to international obligations related with the multilateral climate change agenda that sets the clear path to deepen this country’s international commitments. Moreover, even the recent launch of the USMCA (United States- Mexico-Canada trade agreement) contains in its articles, jointly agreed upon, trilateral and regional binding obligations related to energy, energy markets, energy regulations, and energy security that complement the multilateral obligations. Those set the international framework of maintained obligations and commitments.

  • At the national level, the 21 secondary laws set the market commitments at various levels and sub-markets that need to be maintained. Recent decisions at the Secretary of Energy, CENACE, the Energy Regulatory Commission (CRE), regarding stability of the grid, prefer the incumbent utility CFE to increase its monopoly powers against other market participants from the generation, supply, intermediation, trading, load centers, and innovative prosumers, has set a bad precedent to attract investment efforts in an energy market and the environment agenda, and needs to be recaptured.

  • In case efforts are continued by the present administration, to dismantle the present operative energy reforms, it will create costly litigious environment and could move Mexico to a welfare reducing position. The recommendation here is that being the social and economic cost much larger than the social and economic benefits of true sub-markets, all stakeholders in the short term and large term sub-markets (energy, capacity, renewable energy certificates (CELs), ancillary services, and qualified users of energy, load centers, and even prosumers) should push for deepening commitments, one way or another.

  • On the regulatory front, the main recommendation is that CRE, as an independent and autonomous entity, clearly and forcefully separate regulatory quality with asymmetric regulations between the monopoly parts and the competitive parts of the market, observe best practice elsewhere in the world, and promote parts of the market that so far have not been implemented as shown throughout this chapter.